Paper vs. Computer-based Exams: A Study of Errors in Recursive Binary Tree Algorithms

ABSTRACT

This paper reports on a study of goal-plans and errors produced by students who wrote recursive solutions for a binary tree operation. This work extends a previous study of difficulties CS2 students experienced while writing solutions on paper-based exams. In this study, participants solved the same recursive binary tree problem as part of a hands-on computer-based exam where students had access to an IDE and Java API documentation. Not surprisingly, students who took the computer-based exams were more successful than those who took the paper-based exams (58% vs. 17% correct solutions). However, even with the advantage of access to an IDE, documentation, and test cases, 42% of students taking the computer-based exam still made errors, indicating that students exhibit persistent errors even with support. The most common errors observed included incorrect calculations, missing method calls and missing and incorrect base cases.

References

1. Benander, A. C., Benander, B. A., and Pu, H. 1996. Recursion vs. iteration: An empirical study of comprehension. J. of Systems and Software 32, 1, 73--82.
2. Benander, A. C., Benander, B. A., and Sang, J. 2000. An empirical analysis of debugging performance - differences between iterative and recursive constructs. J. of Systems and Software 54, 1, 17--28.
3. Biggs, J. and Collis, K. 1982. Evaluating the Quality of Learning: The SOLO Taxonomy (Structure of the Observed Learning Outcome). Academic Press, New York, NY.
4. Corney, M., Fitzgerald, S., Hanks, B., Lister, R., McCauley, R., and Murphy, L. 2014. 'Explain in plain English' questions revisited: Data structures problems. In Proceedings of the 45th ACM Technical Symposium on Computer Science Education (Atlanta, GA, USA, March 5-8, 2014). SIGCSE '14. ACM, New York, NY, USA, 591--596.
5. Danielsiek, H., Paul, W., and Vahrenhold, J. 2012. Detecting and understanding students' misconceptions related to algorithms and data structures. In Proceedings of the 43rd ACM Technical Symposium on Computer Science Education (Raleigh, NC, USA, Feb. 29-March 3, 2102). SIGCSE '12. ACM, New York, NY, USA, 21--26.
6. Karpierz, K. and Wolfman, S. A. 2014. Misconceptions and concept inventory questions for binary search trees and hash tables. In Proceedings of the 45th ACM Technical Symposium on Computer Science Education (Atlanta, GA, USA, March 5-8, 2014). SIGCSE '14. ACM, New York, NY, USA, 109--114.
7. Kopec, D., Yarmish, G., and Cheung, P. 2007. A description and study of intermediate student programmer errors. SIGCSE Bull. 39, 2, 146--156.
8. Lister, R., Clear, T., Simon, Bouvier, J., Carter, P., Eckerdal, A., Jacková, J., Lopez, M., McCartney, R., Robbins, P., Seppälä, O. and Thompson, E. 2010. Naturally occurring data as research instrument: analyzing examination responses to study the novice programmer. SIGCSE Bull. 41, 4 (January 2010), 156--173.
9. McCauley, R., Hanks, B., Fitzgerald, S., and Murphy,L. Recursion vs. Iteration: An Empirical Study of Comprehension Revisited, in Proceedings of the 49th Technical Symposium on Computer Science Education, (Kansas City, MO, USA, March 4-7, 2015). SIGCSE '15. ACM, New York, NY, USA 350--355.
10. Murphy, L., Fitzgerald, S., Grissom, S. and McCauley, R. 2015. Recursion vs. iteration: An empirical study of comprehension. In Proceedings of the 46th ACM Technical Symposium on Computer Science Education (Kansas City, MO, USA, March 4-7, 2015). SIGCSE '15. ACM, New York, NY, USA, 482--487.
11. Rinderknecht, C. 2014. A survey on teaching and learning recursive programming. Informatics in Educ 13, 1, 87--119.
12. Spohrer, J. C., Soloway, E., and Pope, E. 1985. A goal/plan analysis of buggy Pascal programs. Human-Computer Interaction 1, 2, 163--207.
13. Tenenberg, J. and Murphy, L. 2005. Knowing what I know: An investigation of undergraduate knowledge and self-knowledge of data structures. Computer Science Education 15, 4, 297--315.
14. Yarmish, G. and Kopec, D. 2007. Revisiting novice programmer errors. SIGCSE Bull. 39, 2, 131--137.

Index Terms

• Applied computing

  • Education