Eye tracking support for visual analytics systems: foundations, current applications, and research challenges

ABSTRACT

Visual analytics (VA) research provides helpful solutions for interactive visual data analysis when exploring large and complex datasets. Due to recent advances in eye tracking technology, promising opportunities arise to extend these traditional VA approaches. Therefore, we discuss foundations for eye tracking support in VA systems. We first review and discuss the structure and range of typical VA systems. Based on a widely used VA model, we present five comprehensive examples that cover a wide range of usage scenarios. Then, we demonstrate that the VA model can be used to systematically explore how concrete VA systems could be extended with eye tracking, to create supportive and adaptive analytics systems. This allows us to identify general research and application opportunities, and classify them into research themes. In a call for action, we map the road for future research to broaden the use of eye tracking and advance visual analytics.

References

1. Sayeed Safayet Alam and Radu Jianu. 2017. Analyzing eye-tracking information in visualization and data space: from where on the screen to what on the screen. IEEE Transactions on Visualization and Computer Graphics 23, 5 (2017), 1492--1505.
2. Robert Amar and John Stasko. 2004. A knowledge task-based framework for design and evaluation of information visualizations. In IEEE Symposium on Information Visualization. IEEE Computer Society Press, 143--150.
3. Gennady Andrienko, Natalia Andrienko, Peter Bak, Daniel Keim, and Stefan Wrobel. 2013. Visual analytics of movement. Springer. 73--101 pages.
4. Gennady Andrienko, Natalia Andrienko, Guido Budziak, Jason Dykes, Georg Fuchs, Tatiana von Landesberger, and Hendrik Weber. 2017. Visual analysis of pressure in football. Journal Data Mining and Knowledge Discovery 31, 6 (2017), 1793--1839.
5. Gennady Andrienko, Natalia Andrienko, Michael Burch, and Daniel Weiskopf. 2012. Visual analytics methodology for eye movement studies. IEEE Transactions on Visualization and Computer Graphics 18, 12 (2012), 2889--2898.
6. Gennady Andrienko, Natalia Andrienko, Georg Fuchs, and Jose Manuel Cordero Garcia. 2018. Clustering trajectories by relevant parts for air traffic analysis. IEEE Transactions on Visualization and Computer Graphics 24, 1 (2018), 34--44.
7. Natalia Andrienko and Gennady Andrienko. 2019. V-analytics. http://geoanalytics.net/V-Analytics/. {Online; accessed 08-Jan-2019}.
8. Natalia Andrienko, Gennady Andrienko, Jose Manuel Cordero Garcia, and David Scarlatti. 2019. analysis of flight variability: a systematic approach. IEEE Transactions on Visualization and Computer Graphics 25, 1 (2019), 54--64.
9. Roman Bednarik. 2005. Potentials of eye-movement tracking in adaptive systems. In Proceedings of the 4th Workshop on Empirical Evaluation of Adaptive Systems. 1--8.
10. Jacques Bertin. 1967. Sémiologie graphique (1st ed.). Paris, Mouton et Gauthier-Villars.
11. Tanja Blascheck, Markus John, Kuno Kurzhals, Steffen Koch, and Thomas Ertl. 2016. VA²: a visual analytics approach for evaluating visual analytics applications. IEEE Transactions on Visualization and Computer Graphics 22, 1 (2016), 61--70.
12. Tanja Blascheck, Kuno Kurzhals, Michael Raschke, Michael Burch, Daniel Weiskopf, and Thomas Ertl. 2017. Visualization of eye tracking data: a taxonomy and survey. Computer Graphics Forum 36, 8 (2017), 260--284.
13. Jonas Blattgerste, Patrick Renner, and Thies Pfeiffer. 2018. Advantages of eye-gaze over head-gaze-based selection in virtual and augmented reality under varying field of views. In COGAIN '18 Proceedings of the Workshop on Communication by Gaze Interaction. 1:1--1:9.
14. SensorBis Blog. 2017. Eye tracking technology - user pro-filing and privacy concerns. http://blog.sensorbis.com/eye-tracking-technology-user-profiling-and-privacy-concerns/ {Online; accessed 18-Nov-2018}.
15. Magdalena Borys and Malgorzata Plechawska-Wójcik. 2017. Eye-tracking metrics in perception and visual attention research. European Journal of Medical Technologies (2017), 11--23.
16. Matthew Brehmer and Tamara Munzner. 2013. A multi-level typology of abstract visualization tasks. IEEE Transactions on Visualization and Computer Graphics 19, 12 (2013), 2376--2385.
17. Sylvain Castagnos, Nicolas Jones, and Pearl Pu. 2010. Eye-tracking product recommenders' usage. In Proceedings of the ACM Conference on Recommender systems. ACM, 29--36.
18. Sylvain Castagnos and Pearl Pu. 2010. Consumer decision patterns through eye gaze analysis. In Proceedings of the Workshop on Eye Gaze in Intelligent Human Machine Interaction. ACM, 78--85.
19. Davide Ceneda, Theresia Gschwandtner, Thorsten May, Silvia Miksch, Hans-Jörg Schulz, Marc Streit, and Christian Tominski. 2017. Characterizing guidance in visual analytics. IEEE Transactions on Visualization and Computer Graphics 23, 1 (2017), 111--120.
20. Li Chen and Pearl Pu. 2011. Users' eye gaze pattern in organization-based recommender interfaces. In Proceedings of the 16th International Conference on Intelligent User Interfaces. ACM, 311--314.
21. Lewis Chuang, Andrew Duchowski, Pernilla Qvarfordt, and Daniel Weiskopf. 2019. Ubiquitous gaze sensing and interaction (Dagstuhl seminar 18252). Dagstuhl Reports 8, 6 (2019), 77--148.
22. Christopher Collins, Natalia Andrienko, Tobias Schreck, Jing Yang, Jaegul Choo, Ulrich Engelke, Amit Jena, and Tim Dwyer. 2018. Guidance in the human-machine analytics process. Visual Informatics 2, 3 (2018), 166--180.
23. Neven A. ElSayed, Bruce H. Thomas, Kim Marriott, Julia Piantadosi, and Ross T. Smith. 2016. Situated analytics: demonstrating immersive analytical tools with augmented reality. Journal of Visual Languages & Computing 36 (2016), 13--23.
24. A. Endert, W. Ribarsky, Cagatay Turkay, B. L. William Wong, Ian T. Nabney, Ignacio Díaz Blanco, and Fabrice Rossi. 2017. The state of the art in integrating machine learning into visual analytics. Computer Graphics Forum 36, 8 (2017), 458--486.
25. Paolo Federico, Markus Wagner, Alexander Rind, Albert Amor-Amorós, Silvia Miksch, and Wolfgang Aigner. 2017. The role of explicit knowledge: a conceptual model of knowledge-assisted visual analytics. In Proc. IEEE Conference on Visual Analytics Science and Technology (VAST).
26. Alexander Felfernig, Michael Jeran, Gerald Ninaus, Florian Reinfrank, and Stefan Reiterer. 2013. Toward the next generation of recommender systems: applications and research challenges. In Multimedia Services in Intelligent Environments. Springer, 81--98.
27. Maite Frutos-Pascual and Begonya Garcia-Zapirain. 2015. Assessing visual attention using eye tracking sensors in intelligent cognitive therapies based on serious games. Sensors (Basel, Switzerland) 15, 5 (May 2015), 11092--11117.
28. Fabian Göbel, Peter Kiefer, Ioannis Giannopoulos, Andrew Duchowski, and Martin Raubal. 2018. Improving map reading with gaze-adaptive legends. In Proceedings of the ACM Symposium on Eye Tracking Research & Applications. ACM, 29:1--29:9.
29. Florian Haag, Tanja Blascheck, Bernhard Schmitz, and Michael Raschke. 2013. Berührpunkte mit der visualisierung. In Multi-Touch - Interaktion Durch Beruhrung, Thomas Schlegel (Ed.). Springer, 339--367.
30. Jiawei Han, Micheline Kamber, and Jian Pei. 2011. Data mining: concepts and techniques (3rd ed.). Morgan Kaufmann Publishers Inc., San Francisco, CA, USA.
31. Christophe Hurter, Nathalie Henry Riche, Steven M. Drucker, Maxime Cordeil, Richard Alligier, and Romain Vuillemot. 2019. Fiberclay: sculpting three dimensional trajectories to reveal structural insights. IEEE Transactions on Visualization and Computer Graphics 25, 1 (2019), 704--714.
32. Radu Jianu and Sayeed Safayet Alam. 2018. A data model and task space for data of interest (DOI) eye-tracking analyses. IEEE Transactions on Visualization and Computer Graphics 24, 3 (2018), 1232--1245.
33. Jehan Jung, Yuka Matsuba, Rammohan Mallipeddi, Hiroyuki Funaya, Kazushi Ikeda, and Minho Lee. 2013. Evolutionary programming based recommendation system for online shopping. In Signal and Information Processing Association Annual Summit and Conference (APSIPA), 2013 Asia-Pacific. IEEE, 1--4.
34. Marcel Just and Patricia Carpenter. 1980. A theory of reading: from eye fixations to comprehension. Psychological Review 87, 4 (1980), 329--354.
35. Daniel Keim, Florian Mansmann, Jörn Schneidewind, Jim Thomas, and Hartmut Ziegler. 2008. Visual analytics: scope and challenges. In Visual Data Mining, Simeon Simoff, Michael Böhlen, and Arturas Mazeika (Eds.). Springer, 76--90.
36. Daniel A. Keim, Jörn Kohlhammer, Geoffrey Ellis, and Florian Mansmann. 2010. Mastering the information age - solving problems with visual analytics. Eurographics.
37. Peter Kiefer, Ioannis Giannopoulos, and Martin Raubal. 2013. Using eye movements to recognize activities on cartographic maps. In Proceedings of the 21st ACM SIGSPATIAL 2013. ACM, 488--491.
38. Kuno Kurzhals, Brian Fisher, Michael Burch, and Daniel Weiskopf. 2014. Evaluating visual analytics with eye tracking. In Proceedings of the BELIV Workshop: Beyond Time and Errors - Novel Evaluation Methods for Visualization. ACM, 61--69.
39. Dmitry Lagun, Cecelia Manzanares, Stuart M. Zola, Elizabeth A. Buffalo, and Eugene Agichtein. 2011. Detecting cognitive impairment by eye movement analysis using automatic classification algorithms. Journal of Neuroscience Methods 201, 1 (30 Sep 2011), 196--203. 21801750{pmid}.
40. Bloomberg Law. 2017. Privacy issues in virtual reality: eye tracking technology. https://www.alston.com/en/insights/publications/2017/07/privacy-issues-in-virtual-reality {Online; accessed 18-Nov-2018}.
41. Daniel Liebling and Sören Preibusch. 2014. Privacy considerations for a pervasive eye tracking world. In Proceedings of the ACM International Joint Conference on Pervasive and Ubiquitous Computing. ACM, 1169--1177.
42. Nikolaos Marianos. 2018. Foveated rendering algorithms using eye-tracking technology in virtual reality. Technical University of Crete.
43. Tamara Munzner. 2014. Visualization analysis and design (1st ed.). CRC Press Taylor and Fracis Group.
44. Mershack Okoe, Sayeed Safayet Alam, and Radu Jianu. 2014. A gaze-enabled graph visualization to improve graph reading tasks. Computer Graphics Forum 33 (2014).
45. David O'sullivan and David Unwin. 2014. Geographic information analysis. John Wiley & Sons, Ltd.
46. William Pike, John Stasko, Remco Chang, and Theresa O'Connell. 2009. The science of interaction. Information Visualization 8, 4 (2009), 263--274.
47. Bernhard Preim and Raimund Dachselt. 2010. Interaktive systeme: band 1: grundlagen, graphical user interfaces, informations visualisierung (2nd ed.). Springer.
48. Ahmad M. Razip, Abish Malik, Shehzad Afzal, Matthew Potrawski, Ross Maciejewski, Yun Jang, Niklas Elmqvist, and David S. Ebert. 2014. A mobile visual analytics approach for law enforcement situation awareness. In Proceedings of the IEEE Pacific Visualization Symposium. IEEE Computer Society Press, 169--176.
49. Patrick Renner and Thies Pfeiffer. 2017. Attention guiding techniques using peripheral vision and eye tracking for feedback in augmented-reality-based assistance systems. In Proceedings of the Symposium on 3D User Interfaces (3DUI). 186--194.
50. Dominik Sacha, Andreas Stoffel, Florian Stoffel, Bum Chul Kwon, Geoffrey Ellis, and Daniel Keim. 2014. Knowledge generation model for visual analytics. IEEE Transactions on Visualization and Computer Graphics 20, 12 (2014), 1604--1613.
51. Lin Shao, Nelson Silva, Eva Eggeling, and Tobias Schreck. 2017. Visual exploration of large scatter plot matrices by pattern recommendation based on eye tracking. In Proceedings of the 2017 ACM Workshop on Exploratory Search and Interactive Data Analytics. ACM, 9--16.
52. Ben Shneiderman. 1996. The eyes have it: a task by data type taxonomy for information visualizations. In Proceedings of the IEEE Symposium on Visual Languages. IEEE Computer Society Press, 336--343.
53. Nelson Silva, Tobias Schreck, Eduardo Veas, Vedran Sabol, Eva Eggeling, and Dieter W Fellner. 2018. Leveraging eye-gaze and time-series features to predict user interests and build a recommendation model for visual analysis. In Proceedings of the 2018 ACM Symposium on Eye Tracking Research & Applications. ACM, 13.
54. Nelson Silva, Volker Settgast, Eva Eggeling, Torsten Ullrich, Tobias Schreck, and Dieter Fellner. 2015. Increasing fault tolerance in operational centres using human sensing technologies: approach and initial results. 25.
55. Nelson Silva, Lin Shao, Tobias Schreck, Eva Eggeling, and Dieter W. Fellner. 2016a. Sense.me - open source framework for the exploration and visualization of eye tracking data. In Proceedings of the 2016 IEEE Conference on Information Visualization.
56. Nelson Silva, Lin Shao, Tobias Schreck, Eva Eggeling, and Dieter W Fellner. 2016b. Visual exploration of hierarchical data using degree-of-interest controlled by eye-tracking. Proceedings of the 9th Forum Media Technology 2016 and 2nd All Around Audio Symposium 2016 1734 (2016), 82--89.
57. Heyjin Song and Nammee Moon. 2017. A preference based recommendation system design through eye-tracking and social behavior analysis. In Advances in Computer Science and Ubiquitous Computing. Springer, 1014--1019.
58. Jay Stanley. 2013. The privacy-invading potential of eye tracking technology. https://www.aclu.org/blog/national-security/privacy-and-surveillance/privacy-invading-potential-eye-tracking-technology {Online; accessed 18-Nov-2018}.
59. Ben Steichen, Giuseppe Carenini, and Cristina Conati. 2013. User-adaptive information visualization: using eye gaze data to infer visualization tasks and user cognitive abilities. In Proceedings of the International Conference on IUIs. ACM, 317--328.
60. Tableau. 2019. Tableau software. https://www.tableau.com/. {Online; accessed 08-Jan-2019}.
61. Yasuo Terao, Hideki Fukuda, and Okihide Hikosaka. 2017. What do eye movements tell us about patients with neurological disorders?: - an introduction to saccade recording in the clinical setting. Proceedings of the Japan Academy, Series B, Physical and Biological Sciences 93, 10 (11 Dec 2017), 772--801. 29225306{pmid}.
62. James J. Thomas and Kristin A. Cook. 2005. Illuminating the path: the research and development agenda for visual analytics. IEEE Computer Society.
63. Windsor Kwan-Chun Ting, Jose Luis Perez Velazquez, and Michael D. Cusimano. 2014. Eye movement measurement in diagnostic assessment of disorders of consciousness. Frontiers in Neurology 5 (29 Jul 2014), 137--137. 25120529{pmid}.
64. Tatiana von Landesberger, Sebastian Fiebig, Sebastian Bremm, Arjan Kuijper, and Dieter Fellner. 2014. Interaction taxonomy for tracking of user actions in visual analytics applications. In Handbook of Human Centric Visualization, Weidong Huang (Ed.). Springer, 653--670.
65. Songhua Xu, Hao Jiang, and Francis Lau. 2008. Personalized online document, image and video recommendation via commodity eye-tracking. In Proceedings of the 2008 ACM conference on Recommender systems. ACM, 83--90.
66. Ji Soo Yi, Younah Kang, John Stasko, and Julie Jacko. 2007. Toward a deeper understanding of the role of interaction in information visualization. IEEE Transactions on Visualization and Computer Graphics 13, 6 (2007), 1224--1231.
67. Qian Zhao, Shuo Chang, F Maxwell Harper, and Joseph A Konstan. 2016. Gaze prediction for recommender systems. In Proceedings of the 10th ACM Conference on Recommender Systems. ACM, 131--138.

Index Terms

• Human-centered computing

  • Human computer interaction (HCI)

    • Interactive systems and tools

  • Visualization

    • Visualization application domains

      • Visual analytics

    • Visualization theory, concepts and paradigms