An eye localization, tracking and blink pattern recognition system: Algorithm and evaluation

Abstract

This study is to investigate the fundamental problems of, (1) facial feature detection and localization, especially eye features; and (2) eye dynamics, including tracking and blink detection. We first describe our contribution to eye localization. Following that, we discuss a simultaneous eye tracking and blink detection system. Facial feature detection is solved in a general object detection framework and its performance for eye localization is presented. A binary tree representation based on feature dependency partitions the object feature space in a coarse to fine manner. In each compact feature subspace, independent component analysis (ICA) is used to get the independent sources, whose probability density functions (PDFs) are modeled by Gaussian mixtures. When applying this representation for the task of eye detection, a subwindow is used to scan the entire image and each obtained image patch is examined using Bayesian criteria to determine the presence of an eye subject. After the eyes are automatically located with binary tree-based probability learning, interactive particle filters are used for simultaneously tracking the eyes and detecting the blinks. The particle filters use classification-based observation models, in which the posterior probabilities are evaluated by logistic regressions in tensor subspaces. Extensive experiments are used to evaluate the performance from two aspects, (1) blink detection rate and the accuracy of blink duration in terms of the frame numbers; (2) eye tracking accuracy. We also present an experimental setup for obtaining the benchmark data in tracking accuracy evaluation. The experimental evaluation demonstrates the capability of this approach.

References

1. Alex, M., Vasilescu, O., and Terzopoulos, D. 2002. Multilinear analysis of image ensembles: Tensorfaces. In Proceedings of the European Conference on Computer Vision.
2. Bell, A. and Sejnowski, T. 1995. An information-maximization approach to blind separation and blind deconvolution. Neural Computation 7, 1129--1159.
3. Bellman, R. E. 1961. Adaptive Control Processes. Princeton University Press.
4. Chang, C. and Ansari, R. 2005. Kernel particle filter for visual tracking. IEEE Signal Process. Lett. 12, 3, 242--245.
5. Chau, M. and Betke, M. 2005. Real time eye tracking and blink detection with USB cameras. Tech. Rep. Boston University Computer Science, 2005-12.
6. Cohn, J. F., Xiao, J., Moriyama, T., Ambadar, Z., and Kanade, T. 2003. Automatic recognition of eye blinking in spontaneously occurring behavior. Behav. Research Methods, Instruments Computers 35, 2, 420--428.
7. Duda, R., Hart, P., and Stork, D. 2000. Pattern Classification (2nd ed.). Wiley Interscience.
8. Fletcher, L., Petersson, L., Apostoloff, N., and Zelinsky, A. 2003. Vision in and out of vehicles. IEEE Intell. Syst.
9. Ghinea, G., Djeraba, C., Gulliver, S. R., and Coyne, K. P. 2007. Introduction to special issue on eye-tracking applications in multimedia systems. ACM Trans. Multimedia Comput. Comm. Appl. 3, 4.
10. Gorodnichy, D. 2003. Second order change detection, and its application to blink-controlled perceptual interfaces. In Proceedings of the International Association of Science and Technology for Development (IASTED) Conference on Visualization, Imaging and Image Processing (VIIP).
11. Grauman, K., Betke, M., Lombardi, J., Gips, J., and Bradski, G. 2003. Communication via eye blinks and eyebrow raises: Video-based human-computer interfaces. Universal Access Inform. Society 2, 4, 359--373.
12. Guan, Y., Fleissner, R., Joyce, P., and Krone, S. M. 2006. Markov chain Monte Carlo in small worlds. Statist. Comput. 16, 2, 193--202.
13. Gustafsson, F., Gunnarsson, F., Bergman, N., Forssell, U., Jansson, J., Karlsson, R., and Nordlund, P. 2002. Particle filters for positioning, navigation, and tracking. IEEE Trans. Signal Proc. 50, 425--435.
14. He, X., Cai, D., and Niyogi, P. 2005a. Tensor subspace analysis. In Proceedings of the Conference on Neural Information Processing Systems (NIPS).
15. He, X., Yan S., Hu, Y., Niyogi, P., and Zhang, H.-J. 2005b. Face recognition using Laplacian faces. IEEE Trans. Patt. Anal. Mach. Intell. 27, 3 (Mar.).
16. Hsu, R. L., Abdel-Mottaleb, M., and Jain, A. K. 2002. Face detection in color images. IEEE Trans. Patt. Anal. Mach. Intell. 24, 5, 696--706.
17. Huang, K. and Trivedi, M. M. 2001. Networked omnivision arrays for intelligent environment. In Proceedings of the 4th IEEE International Workshop on Applications and Science of Soft Computing.
18. Huang, K. and Trivedi, M. M. 2003. Driver head pose and view estimation with single omnidirectional video stream. In Proceedings of the 1st International Workshop on In-Vehicle Cognitive Computer Vision Systems, in Conjunction with the 3rd International Conference on Computer Vision Systems.
19. Isard, M. and Blake, A. 1998. Condensation—conditional density propagation for visual tracking. Inter. J. Comput. Vision 29, 1.
20. Ji, Q. and Yang, X. 2002. Real-time eye, gaze, and face pose tracking for monitoring driver vigilance. Real-Time Imag. 8, 5, 357--377.
21. Kojima, N., Kozuka, K., Nakano, T., and Yamamoto, S. 2001. Detection of consciousness degradation and concentration of a driver for friendly information service. In Proceedings of the IEEE International Vehicle Electronics Conference.
22. Lee, M., Cohen, I., and Jung, S. 2002. Particle filter with analytical inference for human body tracking. In Proceedings of the IEEE Workshop on Motion and Video Computing.
23. Moriyama, T., Kanade, T., Cohn, J. F., Xiao, J., Ambadar, Z., Gao, J., and Imamura, H. 2002. Automatic recognition of eye blinking in spontaneously occurring behavior. In Proceedings of the International Conference on Pattern Recognition (ICPR).
24. Morris, T., Zaidi, F., and Blenkhorn, P. 2002. Blink detection for real-time eye tracking. J. Netw. Comput. Appl. 25, 2, 129--143.
25. Nguyen, K. 2002. Differences in the infrared bright pupil response of human eyes. In Proceedings of the Eye Tracking Research and Applications Symposium.
26. Nishiyama, K. 2005. Fast and effective generation of the proposal distribution for particle filters. Signal Process 85, 12, 2412--2417.
27. Okuma, K., Taleghani, A., de Freitas, N., Little, J. J., and Lowe, D. G. 2004. A boosted particle filter: Multitarget detection and tracking. In Proceedings of the European Conference on Computer Vision.
28. Onoe, Y., Yokoya, N., Yamazawa, K., and Takemura, H. 1998. Visual surveillance and monitoring system using an omnidirectional video camera. In Proceedings of the 14th International Conference on Pattern Recognition.
29. Pantic, M., Pentland, A., Nijholt, A., and Huang, T. S. 2006. Human computing and machine understanding of human behavior: a survey. In Proceedings of the 8th International Conference on Multimodal Interfaces (ICMI). 239--248.
30. Phillips, P. J., Moon, H., Rizvi, S. A., and Rauss, P. J. 2000. The feret evaluation methodology for face recognition algorithms. IEEE Trans. Patt. Anal. Mach. Intell. 22.
31. Roweis, S. T. and Saul, L. K. 2000. Nonlinear dimensionality reduction by locally linear embedding. Science, 290.
32. Rui, Y. and Chen, Y. 2001. Better proposal distributions: Object tracking using unscented particle filter. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR).
33. Rurainsky, and Eisert, P. 2003. Template-based eye and mouth detection for 3D video conferencing. In Proceedings of the International Workshop on Very Low Bitrate Video.
34. Schneiderman, H. 2004. Learning a restricted Bayesian network for object detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.
35. Shen, C., Brooks, M. J., and van den Hengel, A. 2005. Augmented particle filtering for efficient visual tracking. In Proceedings of the IEEE International Conference on Image Processing.
36. Smith, P., Shah, M., and da V. Lobo, N. 2000. Monitoring head/eye motion for driver alertness with one camera. In Proceedings of the 15th IEEE International Conference on Pattern Recognition.
37. Trivedi, M. M., Huang, K. S., and Mikic, I. 2005. Dynamic context capture and distributed video arrays for intelligent spaces. IEEE Trans. Syst. Man, Cybern. A Syst. Humans 35, 1 (Jan.), 145--163.
38. Veeraraghavan, H. and Papanikolopoulos, N. P. 2001. Detecting driver fatique through the use of advanced face monitoring techniques. Tech. rep. University of Minnesota, Center for Transportation Studies.
39. Wang, P., Matthew, B. G., Qiang, J., and Wayman, W. J. 2005. Automatic eye detection and its validation. In Proceedings of the IEEE Workshop on Face Recognition Grand Challenge Experiments (in Conjunction With CVPR).
40. Welch, G. and Bishop, G. 1995. An introduction to the Kalman filter. Tech. rep. University of North Carolina at Chapel Hill, Chapel Hill, NC.
41. Zhu, S. C. 2003. Statistical modeling and conceptualization of visual patterns. IEEE Trans. Patt. Anal. Mach. Intell. 25, 6, 691--712.

Index Terms

• Computing methodologies

  • Artificial intelligence

    • Computer vision

      • Computer vision problems

        • Reconstruction

      • Image and video acquisition

        • Motion capture

  • Computer graphics

    • Animation

      • Motion capture

      • Motion processing