Classification of strategies for solving programming problems using AoI sequence analysis

ABSTRACT

This eye tracking study examines participants' visual attention when solving algorithmic problems in the form of programming problems. The stimuli consisted of a problem statement, example output, and a set of multiple-choice questions regarding variables, data types, and operations needed to solve the programming problems. We recorded eye movements of students and performed an Area of Interest (Aol) sequence analysis to identify reading strategies in terms of participants' performance and visual effort. Using classical eye tracking metrics and a visual Aol sequence analysis we identified two main groups of participants---effective and ineffective problem solvers. This indicates that diversity of participants' mental schemas leads to a difference in their performance. Therefore, identifying how participants' reading behavior varies at a finer level of granularity warrants further investigation.

References

1. Tanja Blascheck, Kuno Kurzhals, Michael Raschke, Michael Burch, Daniel Weiskopf, and Thomas Ertl. 2017. Visualization of Eye Tracking Data: A Taxonomy and Survey. Computer Graphics Forum 36, 8 (2017), 260--284.
2. Martha Crosby and Jan Stelovsky. 1990. How do we Read Algorithms? A Case Study. Computer 23, 1 (1990), 25--35.
3. Françoise Détienne and Elliot Soloway. 1990. An Empirically-derived Control Structure for the Process of Program Understanding. International Journal of Man-Machine Studies 33, 3 (1990), 323--342.
4. Rodrigo Duran, Juha Sorva, and Sofia Leite. 2018. Towards an Analysis of Program Complexity From a Cognitive Perspective. In Proceedings of the ACM Conference on International Computing Education Research. ACM, 21--30.
5. Joseph Goldberg and Xerxes Kotval. 1999. Computer Interface Evaluation using Eye Movements: Methods and Constructs. International Journal of Industrial Ergonomics 24, 6 (1999), 631--645.
6. Anabela Gomes and António José Mendes. 2007. Learning to Program - Difficulties and Solutions. In International Conference on Engineering Education. ACM, 1--5.
7. Kenneth Holmqvist, Jana Holsanova, Mari Barthelson, and Daniel Lundqvist. 2003. Reading or Scanning? A Study of Newspaper and Net Paper Reading. In The Mind's Eye. Elsevier, 657--670.
8. Lai Meng Lung, Tsai Meng Jung, Ying Yang Fang, Yuan Hsu Chung, Chien Liu Tzu, Wen Yu Lee Silvia, Hsien Lee Min, Li Chiou Guo, Chong Liang Jyh, and Chung Tsai Chin. 2013. A Review of Using Eye-tracking Technology in Exploring Learning from 2000 to 2012. Educational Research Review 10 (2013), 90--115.
9. Unaizah Obaidellah, Mohammed Al Haek, and Peter Cheng. 2018. A Survey on the Usage of Eye-tracking in Computer Programming. Comput. Surveys 51, 1 (2018), 5.
10. Daniel Richardson and Rick Dale. 2005. Looking To Understand: The Coupling Between Speakers' and Listeners' Eye Movements and Its Relationship to Discourse Comprehension. Cognitive Science 29, 6 (2005), 1045--1060.
11. Robert Rist. 1989. Schema Creation in Programming. Cognitive Science 13, 3 (1989), 389--414.
12. Dario Salvucci and Joseph Goldberg. 2000. Identifying Fixations and Saccades in Eye-tracking Protocols. In Proceedings of the Symposium on Eye Tracking Research & Applications. ACM, 71--78.
13. Carsten Schulte, Tony Clear, Ahmad Taherkhani, Teresa Busjahn, and James Paterson. 2010. An Introduction to Program Comprehension for Computer Science Educators. In Proceedings of the ITiCSE Working Group Reports. ACM, 65--86.
14. Zohreh Sharafi, Timothy Shaffer, Bonita Sharif, and Yann-Gaël Guéhéneuc. 2015. Eye-Tracking Metrics in Software Engineering. In Asia-Pacific Software Engineering Conference. IEEE Computer Society Press, 96--103.
15. Kshitij Sharma, Katerina Mangaroska, Halvard Trætteberg, Serena Lee-Cultura, and Michail Giannakos. 2018. Evidence for Programming Strategies in University Coding Exercises. In European Conference on Technology Enhanced Learning. Springer, 326--339.
16. Elliot Soloway. 1986. Learning to Program = Learning to Construct Mechanisms and Explanations. Commun. ACM 29, 9 (1986), 850--858.
17. Elliot Soloway, Jeffrey Bonar, and Kate Ehrlich. 1983. Cognitive Strategies and Looping Constructs: An Empirical Study. Commun. ACM 26, 11 (1983), 853--860.
18. Elliot Soloway and Kate Ehrlich. 1984. Empirical Studies of Programming Knowledge. IEEE Transactions on Software Engineering SE-10, 5 (1984), 595--609.
19. James Spohrer, Elliot Soloway, and Edgar Pope. 1985. A Goal/Plan Analysis of Buggy Pascal Programs. Human-Computer Interaction 1, 2 (1985), 163--207.
20. Anna Stolińska and Magdalena Andrzejewska. 2017. Analysis of the Strategy Used to Solve Algorithmic Problem: A Case Study Based on Eye Tracking Research. In New Trends in Analysis and Inter disciplinary Applications. Springer, 77--86.

Index Terms

• Applied computing

  • Education

• Human-centered computing

  • Visualization

• Theory of computation

  • Semantics and reasoning

    • Program constructs

      • Program schemes