Inferring object relevance from gaze in dynamic scenes

ABSTRACT

As prototypes of data glasses having both data augmentation and gaze tracking capabilities are becoming available, it is now possible to develop proactive gaze-controlled user interfaces to display information about objects, people, and other entities in real-world setups. In order to decide which objects the augmented information should be about, and how saliently to augment, the system needs an estimate of the importance or relevance of the objects of the scene for the user at a given time. The estimates will be used to minimize distraction of the user, and for providing efficient spatial management of the augmented items. This work is a feasibility study on inferring the relevance of objects in dynamic scenes from gaze. We collected gaze data from subjects watching a video for a pre-defined task. The results show that a simple ordinal logistic regression model gives relevance rankings of scene objects with a promising accuracy.

References

1. Hardoon, D., Shawe-Taylor, J., Ajanki, A., Puolamäki, K., and Kaski, S. 2007. Information retrieval by inferring implicit queries from eye movements. In International Conference on Artificial Intelligence and Statistics (AISTATS '07).
2. Henderson, J. M. 2003. Human gaze control during real-world scene perception. Trends in Cognitive Sciences 7, 11, 498--504.
3. Hyrskykari, A., Majaranta, P., Aaltonen, A., and Räihä, K.-J. 2000. Design issues of 'idict': A gaze-assisted translation aid. In Proceedings of ETRA 2000, Eye Tracking Research and Applications Symposium, ACM Press, ACM Press, 9--14.
4. Itti, L., Koch, C., and Niebur, E. 1998. A model of saliency-based visual attention for rapid scene analysis. IEEE Transactions on Pattern Analysis and Machine Intelligence 20, 11, 1254--1259.
5. Kandemir, M., Saarinen, V.-M., and Kaski, S. 2010. Inferring object relevance from gaze in dynamic scenes. In To Appear in Short Paper Proceedings of ETRA 2000, Eye Tracking Research and Applications Symposium.
6. Klami, A., Saunders, C., de Campos, T. E., and Kaski, S. 2008. Can relevance of images be inferred from eye movements? In MIR '08: Proceeding of the 1st ACM international conference on Multimedia information retrieval, ACM, New York, NY, USA, 134--140.
7. Kozma, L., Klami, A., and Kaski, S. 2009. GaZIR: Gaze-based zooming interface for image retrieval. In Proc. ICMIMLMI 2009, The Eleventh International Conference on Multimodal Interfaces and The Sixth Workshop on Machine Learning for Multimodal Interaction, ACM, New York, NY, USA, 305--312.
8. McCullagh, P., and Nelder, J. 1989. Generalized Linear Models. Chapman & Hall/CRC.
9. Qvarfordt, P., and Zhai, S. 2005. Conversing with the user based on eye-gaze patterns. In CHI '05: Proceedings of the SIGCHI conference on Human factors in computing systems, ACM, New York, NY, USA, 221--230.
10. Torralba, A., Oliva, A., Castelhano, M. S., and Henderson, J. M. 2006. Contextual guidance of eye movements and attention in real-world scenes: the role of global features in object search. Psychological Review 113, 4, 766--786.
11. Ward, D. J., and MacKay, D. J. C. 2002. Fast hands-free writing by gaze direction. Nature 418, 6900, 838.
12. Zhang, L., Tong, M. H., Marks, T. K., Shan, H., and Cottrell, G. W. 2008. Sun: A bayesian framework for saliency using natural statistics. Journal of Vision 8, 7 (12), 1--20.

Index Terms

• Human-centered computing

  • Human computer interaction (HCI)

    • Interactive systems and tools

      • User interface management systems