Pupillometry and Head Distance to the Screen to Predict Skill Acquisition During Information Visualization Tasks

ABSTRACT

In this paper we investigate using a variety of behavioral measures collectible with an eye tracker to predict a user's skill acquisition phase while performing various information visualization tasks with bar graphs. Our long term goal is to use this information in real-time to create user-adaptive visualizations that can provide personalized support to facilitate visualization processing based on the user's predicted skill level. We show that leveraging two additional content-independent data sources, namely information on a user's pupil dilation and head distance to the screen, yields a significant improvement for predictive accuracies of skill acquisition compared to predictions made using content-dependent information related to user eye gaze attention patterns, as was done in previous work. We show that including features from both pupil dilation and head distance to the screen improve the ability to predict users' skill acquisition state, beating both the baseline and a model using only content-dependent gaze information.

References

1. Akaike, H. Factor analysis and AIC. Psychometrika 52, 3 (1987), 317--332.
2. Amar, R., Eagan, J., and Stasko, J. Low-Level Components of Analytic Activity in Information Visualization. Proceedings of the Proceedings of the 2005 IEEE Symposium on Information Visualization, IEEE Computer Society (2005), 15--21.
3. Baker, R., Corbett, A., and Aleven, V. More accurate student modeling through contextual estimation of slip and guess probabilities in bayesian knowledge tracing. Proceedings of the 9th International Conference Intelligent Tutoring Systems, (2008), 406--415.
4. Baker, R.Sj., D'Mello, S.K., Rodrigo, M.M.T., and Graesser, A.C. Better to be frustrated than bored: The incidence, persistence, and impact of learners' cognitive--affective states during interactions with three different computer-based learning environments. International Journal of Human-Computer Studies 68, 4 (2010), 223--241.
5. Barnes, T. and Stamper, J. Toward automatic hint generation for logic proof tutoring using historical student data. Proceedings of the 9th International Conference on Intelligent Tutoring Systems, Springer (2008), 373--382.
6. Beatty, J. Task-evoked pupillary responses, processing load, and the structure of processing resources. Psychological Bulletin 91, 2 (1982), 276--292.
7. Beck, J.E. and Sison, J. Using knowledge tracing in a noisy environment to measure student reading proficiencies. International Journal of Artificial Intelligence in Education 16, 2 (2006), 129--143.
8. Bednarik, R., Eivazi, S., and Vrzakova, H. A Computational Approach for Prediction of ProblemSolving Behavior Using Support Vector Machines and Eye-Tracking Data. In Y.I. Nakano, C. Conati and T. Bader, eds., Eye Gaze in Intelligent User Interfaces. Springer London, London, 2013, 111--134.
9. Biswas, P., Dutt, V., and Langdon, P. Comparing Ocular Parameters for Cognitive Load Measurement in EyeGaze-Controlled Interfaces for Automotive and Desktop Computing Environments. International Journal of Human-Computer Interaction 32, 1 (2016), 23--38.
10. Bixler, R. and D'Mello, S. Automatic Gaze-Based Detection of Mind Wandering with Metacognitive Awareness. In F. Ricci, K. Bontcheva, O. Conlan and S. Lawless, eds., User Modeling, Adaptation and Personalization. Springer, Cham, 2015, 31--43.
11. Bixler, R., Kopp, K., and D'Mello, S. Evaluation of a Personalized Method for Proactive Mind Wandering Reduction. Proceedings of the 4th Workshop on Personalization Approaches for Learning Environments, 22nd conference on User Modeling, Adaptation, and Personalization, Springer (2014), 33--41.
12. Bondareva, D., Conati, C., Feyzi-Behnagh, R., Harley, J.M., Azevedo, R., and Bouchet, F. Inferring Learning from Gaze Data during Interaction with an Environment to Support Self-Regulated Learning. In H.C. Lane, K. Yacef, J. Mostow and P. Pavlik, eds., Artificial Intelligence in Education. Springer Berlin Heidelberg, Berlin, Heidelberg, 2013, 229--238.
13. Boy, J., Rensink, R.A., Bertini, E., and Fekete, J.-D. A Principled Way of Assessing Visualization Literacy. IEEE Transactions on Visualization and Computer Graphics 20, 12 (2014), 1963--1972.
14. Brusilovsky, P., Ahn, J., Dumitriu, T., and Yudelson, M. Adaptive Knowledge-Based Visualization for Accessing Educational Examples. Proc. 10th International Conf. on Information Visualization, IEEE (2006), 142--150.
15. Bunt, A., Conati, C., and McGrenere, J. Supporting interface customization using a mixed-initiative approach. ACM Press (2007), 92.
16. Carenini, G., Conati, C., Hoque, E., Steichen, B., Toker, D., and Enns, J.T. Highlighting Interventions and User Differences: Informing Adaptive Information Visualization Support. Proc. of the SIGCHI Conference on Human Factors in Computing Systems, (2014).
17. Carenini, G., Conati, C., Hoque, E., Steichen, B., Toker, D., and Enns, J.T. Highlighting Interventions and User Differences: Informing Adaptive Information Visualization Support. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, ACM (2014), 1835--1844.
18. Çöltekin, A., Fabrikant, S.I., and Lacayo, M. Exploring the efficiency of users' visual analytics strategies based on sequence analysis of eye movement recordings. International Journal of Geographical Information Science 24, 10 (2010), 1559--1575.
19. Conati, C., Carenini, G., Hoque, E., Steichen, B., and Toker, D. Evaluating the impact of user characteristics and different layouts on an interactive visualization for decision making. Computer Graphics Forum, Wiley Online Library (2014), 371--380.
20. Conati, C., Carenini, G., Toker, D., and Lallé, S. Towards User-Adaptive Information Visualization. AAAI Conf. on Artificial Intelligence, (2015).
21. Conati, C., Hoque, E., Toker, Dereck, and Steichen, Ben. When to Adapt: Detecting User's Confusion During Visualization Processing. Proc. of the 1st International Workshop on User-Adaptive Information Visualization (WUAV), in conjunction with the 21st Conference on User Modeling, Adaptation and Personalization (UMAP), (2013).
22. D'Mello, S., Olney, A., Williams, C., and Hays, P. Gaze tutor: A gaze-reactive intelligent tutoring system. International Journal of Human-Computer Studies 70, 5 (2012), 377--398.
23. D'Mello, S.K., Craig, S.D., Sullins, J., and Graesser, A.C. Predicting affective states expressed through an emote-aloud procedure from AutoTutor's mixed initiative dialogue. International Journal of Artificial Intelligence in Education 16, 1 (2006), 3--28.
24. Field, A. Discovering Statistics Using IBM SPSS Statistics. Sage Publications Ltd, London, 2012.
25. Gingerich, M.J. and Conati, C. Constructing Models of User and Task Characteristics from Eye Gaze Data for User-Adaptive Information Highlighting. Twenty-Ninth AAAI Conference on Artificial Intelligence, (2015).
26. Gotz, D. and Wen, Z. Behavior-driven visualization recommendation. Proc. of the 14th international conf. on Intelligent user interfaces, ACM (2009), 315--324.
27. Green, T.M. and Fisher, B. Impact of personality factors on interface interaction and the development of user profiles: Next steps in the personal equation of interaction. Information Visualization 11, 3 (2012), 205-- 221.
28. Haas Lyons, S., Walsh, M., Aleman, E., and Robinson, J. Exploring regional futures: Lessons from Metropolitan Chicago's online MetroQuest. Technological Forecasting and Social Change 82, (2014), 23--33.
29. Hess, E.H. and Polt, J.M. Pupil Size in Relation to Mental Activity during Simple Problem-Solving. Science 143, 3611 (1964), 1190--1192.
30. Hurst, A., Hudson, S.E., and Mankoff, J. Dynamic Detection of Novice vs. Skilled Use Without a Task Model. Proc. of the SIGCHI Conference on Human Factors in Computing Systems, ACM (2007), 271--280.
31. Hyönä, J., Tommola, J., and Alaja, A.-M. Pupil dilation as a measure of processing load in simultaneous interpretation and other language tasks. The Quarterly Journal of Experimental Psychology 48, 3 (1995), 598-- 612.
32. Iqbal, S.T., Adamczyk, P.D., Zheng, X.S., and Bailey, B.P. Towards an index of opportunity: understanding changes in mental workload during task execution. ACM Press (2005), 311.
33. Jacobs, A.M., Fransen, B., McCurry, J.M., Heckel, F.W.P., Wagner, A.R., and Trafton, J.G. A Preliminary System for Recognizing Boredom. Proceedings of the 4th ACM/IEEE International Conference on Human Robot Interaction, ACM (2009), 299--300.
34. Jaques, N., Conati, C., Harley, J.M., and Azevedo, R. Predicting Affect from Gaze Data during Interaction with an Intelligent Tutoring System. Proceedings of the 12th International Conference on Intelligent Tutoring Systems, Springer (2014), 29--38.
35. Kalyuga, S. Enhancing Instructional Efficiency of Interactive E-learning Environments: A Cognitive Load Perspective. Educational Psychology Review 19, 3 (2007), 387--399.
36. Kardan, S. and Conati, C. Comparing and Combining Eye Gaze and Interface Actions for Determining User Learning with an Interactive Simulation. In: Proc. of UMAP, 21st Int. Conf. on User Modeling, Adaptation and Personalization, (2013).
37. Kong, N., Hearst, M.A., and Agrawala, M. Extracting references between text and charts via crowdsourcing. Proceedings of the SIGCHI conference on Human Factors in Computing Systems, ACM (2014), 31--40.
38. Kuhn, M. Building predictive models in R using the caret package. Journal of Statistical Software 28, 5 (2008), 1--26.
39. Lallé, S., Conati, C., and Carenini, G. Predicting confusion in information visualization from eye tracking and interaction data. Proceedings on the 25th International Joint Conference on Artificial Intelligence, (2016), 2529--2535.
40. Lallé, S., Mostow, J., Luengo, V., and Guin, N. Comparing Student Models in Different Formalisms by Predicting their Impact on Help Success. Proceedings of the 16th International Conference on Artificial Intelligence in Education, (2013), 161--170.
41. Lallé, S., Toker, D., Conati, C., and Carenini, G. Prediction of Users' Learning Curves for Adaptation while Using an Information Visualization. Proceedings of the 20th International Conference on Intelligent User Interfaces, ACM (2015), 357--368.
42. Marshall, S.P. The index of cognitive activity: Measuring cognitive workload. Proceedings of the 7th IEEE Human Factors Meeting, IEEE (2002), 5--9.
43. Marshall, S.P. Identifying cognitive state from eye metrics. Aviation, space, and environmental medicine 78, Supplement 1 (2007), B165--B175.
44. Martínez-Gómez, P. and Aizawa, A. Recognition of understanding level and language skill using measurements of reading behavior. ACM Press (2014), 95--104.
45. Mueller, F. and Lockerd, A. Cheese: tracking mouse movement activity on websites, a tool for user modeling. CHI'01 Extended Abstracts on Human Factors in Computing Systems, ACM Press (2001), 279--280.
46. Ooms, K., De Maeyer, P., and Fack, V. Study of the attentive behavior of novice and expert map users using eye tracking. Cartography and Geographic Information Science 41, 1 (2014), 37--54.
47. Pascual-Cid, V., Vigentini, L., and Quixal, M. Visualising Virtual Learning Environments: Case Studies of the Website Exploration Tool. IEEE (2010), 149--155.
48. Pavlik, P.I., Cen, H., and Koedinger, K.R. Performance Factors Analysis--A New Alternative to Knowledge Tracing. Proceedings of the International Conference on AI in Education, IOS Press (2009), 531--538.
49. Prendinger, H., Hyrskykari, A., Nakayama, M., Istance, H., Bee, N., and Takahasi, Y. Attentive interfaces for users with disabilities: eye gaze for intention and uncertainty estimation. Universal Access in the Information Society 8, 4 (2009), 339--354.
50. Saraiya, P., North, C., and Duca, K. An Insight-Based Methodology for Evaluating Bioinformatics Visualizations. IEEE Transactions on Visualization and Computer Graphics 11, 4 (2005), 443--456.
51. Speelman, C. and Kirsner, K. Beyond the Learning Curve. Oxford University Press, 2005.
52. Steichen, B., Conati, C., and Carenini, G. Inferring Visualization Task Properties, User Performance, and User Cognitive Abilities from Eye Gaze Data. ACM Transactions on Interactive Intelligent Systems, (2014).
53. Toker, D., Conati, C., Steichen, B., and Carenini, G. Individual user characteristics and information visualization: connecting the dots through eye tracking. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, ACM (2013), 295--304.
54. Toker, D. and Conati, C. Eye Tracking to Understand User Differences in Visualization Processing with Highlighting Interventions. In V. Dimitrova, T. Kuflik, D. Chin, F. Ricci, P. Dolog and G.-J. Houben, eds., User Modeling, Adaptation, and Personalization. Springer International Publishing, 2014, 219--230.
55. Toker, D., Steichen, B., Gingerich, M., Conati, C., and Carenini, G. Towards Facilitating User Skill Acquisition - Identifying Untrained Visualization Users through Eye Tracking. Proceedings of the 2014 international conference on Intelligent user interfaces, ACM (2014).
56. Vrochidis, S., Patras, I., and Kompatsiaris, I. An Eye tracking-based Approach to Facilitate Interactive Video Search. Proc. of the 1st ACM International Conference on Multimedia Retrieval, ACM (2011), 43:1--43:8.
57. Zhu, Y. Measuring Effective Data Visualization. Proc. of the 3rd International Symposium on Advances in Visual Computing, Springer (2007), 652--661.
58. Ziemkiewicz, C. and Kosara, R. The Shaping of Information by Visual Metaphors. IEEE Transactions on Visualization and Computer Graphics 14, 6 (2008), 1269--1276.

Index Terms

• Human-centered computing