Michael Kölling
Fraser McKay
Skip Abstract Section
Abstract
The past few years has seen a proliferation of novice programming tools. The availability of a large number of systems has made it difficult for many users to choose among them. Even for education researchers, comparing the relative quality of these tools, or judging their respective suitability for a given context, is hard in many instances. For designers of such systems, assessing the respective quality of competing design decisions can be equally difficult.
Heuristic evaluation provides a practical method of assessing the quality of alternatives in these situations and of identifying potential problems with existing systems for a given target group or context. Existing sets of heuristics, however, are not specific to the domain of novice programming and thus do not evaluate all aspects of interest to us in this specialised application domain.
In this article, we propose a set of heuristics to be used in heuristic evaluations of novice programming systems. These heuristics have the potential to allow a useful assessment of the quality of a given system with lower cost than full formal user studies and greater precision than the use of existing sets of heuristics. The heuristics are described and discussed in detail. We present an evaluation of the effectiveness of the heuristics that suggests that the new set of heuristics provides additional useful information to designers not obtained with existing heuristics sets.
- E. Allen, R. Cartwright, and B. Stoler. 2002. DrJava: A lightweight pedagogic environment for Java. SIGCSE Bull. 34, 1 (2002), 137--141. DOI:http://dx.doi.org/10.1145/563517.563395 Google Scholar
Digital Library
- J. M. C. Bastien and D. L. Scapin. 1995. Evaluating a user interface with ergonomic criteria. Int. J. Hum.-Comput. Interact. 7, 2 (1995), 105--121. DOI:http://dx.doi.org/10.1080/10447319509526114 Google ScholarDigital Library
- A. Begel and E. Klopfer. 2005. StarLogo: A programmable complex systems modeling environment for students and teachers. Artificial Life Models in Software. Springer London, 187--209.Google Scholar
- M. M. Bekker, E. Baauw, and W. Barendregt. 2008. A comparison of two analytical evaluation methods for educational computer games for young children. Cogn. Technol. Work 10, 2 (2008), 129--140. DOI:10.1007/s10111-007-0068-x Google ScholarDigital Library
- M. Ben-Ari. 1998. Constructivism in computer science education. In Proceedings of the 29th SIGCSE Technical Symposium on Computer Science Education. 257--261. DOI:http://dx.doi.org/10.1145/274790.274308 Google ScholarDigital Library
- M. Berland and T. Martin. 2011. Clusters and patterns of novice programmers. In Proceedings of the Meeting of the American Educational Research Association.Google Scholar
- B. E. Birnbaum and K. J. Goldman. 2005. Achieving flexibility in direct-manipulation programming environments by relaxing the edit-time grammar. In Proceedings of the IEEE Symposium on Visual Languages and Human-Centric Computing. 259--266. Google ScholarDigital Library
- A. F. Blackwell. 1996. Metacognitive theories of visual programming: What do we think we are doing? In Proceedings of the IEEE Symposium on Visual Languages. 240--246. Google ScholarDigital Library
- K. B. Bruce, A. P. Danyluk, and T. P. Murtagh. 2001. Event-driven programming is simple enough for CS1. ACM SIGCSE Bull. 33, 3 (2001), 4. DOI:http://dx.doi.org/10.1145/507758.377440 Google ScholarDigital Library
- Q. Burke and Y. B. Kafai. 2010. Programming & storytelling: Opportunities for learning about coding & composition. In Proceedings of the 9th International Conference on Interaction Design and Children. 348--351. DOI:http://dx.doi.org/10.1145/1810543.1810611 Google ScholarDigital Library
- M. E. Caspersen and H. B. Christensen. 2000. Here, there and everywhere-on the recurring use of turtle graphics in CS1. In Proceedings of the 4th Australasian Computing Education Conference (ACE’00). 34--40. Google ScholarDigital Library
- S. Cooper, W. Dann, and R. Pausch. 2003. Teaching objects-first in introductory computer science. In Proceedings of the 34th SIGCSE Technical Symposium on Computer Science Education (SIGCSE’03). ACM, 191--195. DOI:http://dx.doi.org/10.1145/792548.611966 Google ScholarDigital Library
- A. Cooper, R. Reimann, and D. Cronin (Eds.). 2007. Errors, alerts and confirmation. In About Face 3: The Essentials of Interaction Design. Wiley, Indianapolis, 529--550.Google Scholar
- M. de Raadt, R. Watson, and M. Toleman. 2002. Language trends in introductory programming courses. In Informing Science: Where Parallels Intersect. 329--337.Google Scholar
- P. Denny, A. Luxton-Reilly, E. Tempero, and J. Hendrickx. 2011. Understanding the syntax barrier for novices. In Proceedings of the 16th Annual Joint Conference on Innovation and Technology in Computer Science Education. 208--212. DOI:http://dx.doi.org/10.1145/1999747.1999807 Google ScholarDigital Library
- T. R. G. Green. 1989. Cognitive dimensions of notations. In People and Computers V: Proceedings of the 5th Conference of the British Computer Society Human-Computer Interaction Specialist Group. 443--460. Google ScholarDigital Library
- T. R. G. Green and M. Petre. 1996. Usability analysis of visual programming environments: A ‘cognitive dimensions’ framework. J. Vis. Lang. Comput. 7, 2 (1996), 131--174.Google ScholarCross Ref
- H. R. Hartson, T. S. Andre, and R. C. Williges. 2001. Criteria for evaluating usability evaluation methods. Int. J. Hum.-Comput. Interact. 13, 4 (2001), 373--410.Google ScholarCross Ref
- C. D. Hundhausen and J. L. Brown. 2007. An experimental study of the impact of visual semantic feedback on novice programming. J. Vis. Lang. Comput. 18, 6 (2007), 537--559. DOI:http://dx.doi.org/ 10.1016/j.jvlc.2006.09.0012 Google ScholarDigital Library
- M. C. Jadud. 2006. Methods and tools for exploring novice compilation behaviour. In Proceedings of the 2nd International Workshop on Computing Education Research. 73--84. DOI:http://dx.doi.org/ 10.1145/1151588.1151600 Google ScholarDigital Library
- M. R. Jakobsen and K. Hornbæk. 2006. Evaluating a fisheye view of source code. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. 377--386. DOI:http://dx.doi.org/ 10.1145/1124772.1124830 Google ScholarDigital Library
- C. Kelleher and R. Pausch. 2007. Using storytelling to motivate programming. Commun. ACM, 50, 7 (2007), 58--64. DOI:http://dx.doi.org/10.1145/1272516.1272540 Google ScholarDigital Library
- A. J. Ko, H. H. Aung, and B. A. Myers. 2005. Design requirements for more flexible structured editors from a study of programmers’ text editing. In CHI’05 Extended Abstracts on Human Factors in Computing Systems. 1557--1560. Google ScholarDigital Library
- A. J. Ko and B. A. Myers. 2006. Barista: An implementation framework for enabling new tools, interaction techniques and views in code editors. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. 387--396. DOI:http://dx.doi.org/10.1145/1124772.1124831 Google ScholarDigital Library
- A. J. Ko, B. A. Myers, and H. H. Aung. 2004. Six learning barriers in end-user programming systems. In Proceedings of the IEEE Symposium on Visual Languages and Human Centric Computing. 199--206. DOI:http://dx.doi.org/10.1109/VLHCC.2004.47 Google ScholarDigital Library
- M. Kölling. 1999. The Design of an Object-Oriented Environment and Language for Teaching. Ph.D. Dissertation. University of Sydney, Basser Department of Computer Science.Google Scholar
- M. Kölling. 2010. The greenfoot programming environment. ACM Trans. Comput. Educ. 10, 4 (2010), 14. DOI:http://dx.doi.org/10.1145/1868358.1868361 Google ScholarDigital Library
- M. Kölling, B. Quig, A. Patterson, and J. Rosenberg. 2003. The BlueJ system and its pedagogy. J. Comput. Sci. Educ. 13, 4 (2003), 249--268.Google ScholarCross Ref
- E. Konidari and P. Louridas. 2010. When students are not programmers. ACM Inroads 1, 1 (2010), 55--60. DOI:http://dx.doi.org/10.1145/1721933.1721952 Google ScholarDigital Library
- M. MacLaurin. 2009. Kodu: End-user programming and design for games. In Proceedings of the 4th International Conference on Foundations of Digital Games. 2. DOI:http://dx.doi.org/10.1145/1536513.1536516 Google ScholarDigital Library
- T. W. Malone. 1980. What makes things fun to learn? heuristics for designing instructional computer games. In Proceedings of the 3rd SIGSMALL Symposium and the 1st SIGPC Symposium on Small Systems. 162--169. DOI:http://dx.doi.org/10.1145/800088.802839 Google ScholarDigital Library
- J. Maloney, M. Resnick, N. Rusk, B. Silverman, and E. Eastmond. 2010. The scratch programming language and environment. ACM Trans. Comput. Educ., 10, 4 (2010), 16. DOI:http://dx.doi.org/ 10.1145/1868358.1868363 Google ScholarDigital Library
- J. H. Maloney, Y. B. Kafai, M. Resnick, and N. Rusk. 2008. Programming by choice: Urban youth learning programming with scratch. In Proceedings of the 39th SIGCSE Technical Symposium on Computer Science Education. 367--371. DOI:http://dx.doi.org/10.1145/1352135.1352260 Google ScholarDigital Library
- G. Marceau, K. Fisler, and S. Krishnamurthi. 2011. Measuring the effectiveness of error messages designed for novice programmers. In Proceedings of the 42nd ACM Technical Symposium on Computer Science Education. 499--504. DOI:http://dx.doi.org/10.1145/1953163.1953308 Google ScholarDigital Library
- F. McKay. 2012. A prototype structured but low-viscosity editor for novice programmers. BCS-HCI’12. In Proceedings of the 26th Annual BCS Interaction Specialist Group Conference on People and Computers. 363--368. Google ScholarDigital Library
- R. J. Miara, J. A. Musselman, J. A. Navarro, and B. Shneiderman. 1983. Program indentation and comprehensibility. Commun. ACM, 26, 11 (1983), 861--867. DOI:http://dx.doi.org/10.1145/182.358437 Google ScholarDigital Library
- W. W. Milner. 2010. A broken metaphor in Java. ACM SIGCSE Bullet. 41, 4 (2010), 76--77. DOI:http://dx.doi.org/10.1145/1709424.1709450 Google ScholarDigital Library
- J. Nielsen. 2005. Ten Usability Heuristics. Retrieved May 08, 2015 from http://www.useit.com/papers/heuristic/heuristic_list.html.Google Scholar
- J. Nielsen. 2005b. How to Conduct a Heuristic Valuation. Retrieved May 8, 2015, from http://www.useit.com/ papers/heuristic/heuristic_evaluation.html.Google Scholar
- J. Nielsen and R. Molich. 1990. Heuristic evaluation of user interfaces. CHI’90. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems: Empowering People. 249--256. DOI:http://dx.doi.org/ 10.1145/97243.97281 Google ScholarDigital Library
- M. Overmars. 2004. Teaching computer science through game design. Computer 37, 4, 81--83. DOI:http://dx.doi.org/10.1109/MC.2004.1297314 Google ScholarDigital Library
- J. F. Pane. 2002. A programming system for children that is designed for usability. Ph.D. Dissertation. Carnegie Mellon University, Pittsburgh, Pennsylvania. Google ScholarDigital Library
- J. F. Pane and B. A. Myers. 1996. Usability Issues in the Design of Novice Programming Systems. Carnegie Mellon University, School of Computer Science Technical Report CMU-CS-96-132, Pittsburgh, PA, August 1996, 85.Google Scholar
- S. Papert. 1980. Mindstorms: Children, Computers, and Powerful Ideas. Basic Books New York, NY. Google ScholarDigital Library
- R. E. Pattis. 1981. Karel the Robot: A Gentle Introduction to the Art of Programming (1st ed.). John Wiley & Sons, New York, NY. Google ScholarDigital Library
- Python Software Foundation. 2012. Python standard library: Turtle graphics for tk. Retrieved May 08, 2015 from http://docs.python.org/library/turtle.html.Google Scholar
- G. K. Rambally. 1986. The influence of color on program readability and comprehensibility. In Proceedings of the 17th SIGCSE Symposium on Computer Science Education. 18, 1, 173--181. DOI:http://dx.doi.org/10.1145/953055.5702 Google ScholarDigital Library
- A. Robins, P. Haden, and S. Garner. 2006. Problem distributions in a CS1 course. In Proceedings of the 8th Australian Conference on Computing Education. 165--173. Google ScholarDigital Library
- D. Sanders and B. Dorn. 2003. Jeroo: A tool for introducing object-oriented programming. In Proceedings of the 34th SIGCSE Technical Symposium on Computer Science Education. 201--204. DOI:http://dx.doi.org/10.1145/792548.611968 Google ScholarDigital Library
- K. Sanders and L. Thomas. 2007. Checklists for grading object-oriented CS1 programs: Concepts and misconceptions. ACM SIGCSE Bull. 39, 3 (2007), 166--170. DOI:http://dx.doi.org/10.1145/1269900.1268834 Google ScholarDigital Library
- C. Schulte and J. Bennedsen. 2006. What do teachers teach in introductory programming? In Proceedings of the 2nd International Workshop on Computing Education Research. 17--28. DOI:http://dx.doi.org/10.1145/1151588.1151593 Google ScholarDigital Library
- A. Sears. 1997. Heuristic walkthroughs: Finding the problems without the noise. Int. J. Hum.-Comput. Interact. 9, 3 (1997), 213--234.Google ScholarCross Ref
- S. Shingō. 1986. Zero Quality Control: Source Inspection and the Poka-Yoke System. Productivity Press, New York, NY, 305 pages.Google Scholar
- R. M. Siegfried. 2006. Visual programming and the blind: The challenge and the opportunity. ACM SIGCSE Bull., 38, 1 (2006), 275--278. DOI:http://dx.doi.org/10.1145/1124706.1121427 Google ScholarDigital Library
- J. M. Slack. 1990. Turbo Pascal with Turtle Graphics. West Publishing Co., St. Paul, MN. Google ScholarDigital Library
- L. Thomas, M. Ratcliffe, and A. Robertson. 2003. Code warriors and code-a-phobes: A study in attitude and pair programming. In Proceedings of the 34th SIGCSE Technical Symposium on Computer Science Education. 363--367. DOI:http://dx.doi.org/10.1145/792548.612007 Google ScholarDigital Library
- B. Thomasson, M. Ratcliffe, and L. Thomas. 2006. Identifying novice difficulties in object oriented design. In Proceedings of the 11th Annual SIGCSE Conference on Innovation and Technology in Computer Science Education. 28--32. DOI:http://dx.doi.org/10.1145/1140123.1140135 Google ScholarDigital Library
- S. Turkle and S. Papert. 1992. Epistemological pluralism and the revaluation of the concrete. J. Math. Behav. 11, 1 (1992), 3--33.Google Scholar
- C. Ware. 2008. Visual Thinking for Design. Morgan Kaufmann, Burlington, MA. Google ScholarDigital Library
- G. M. Weinberg. 1998. The Psychology of Computer Programming (2nd ed.). Dorset House Publishing, New York, NY. Google ScholarDigital Library
Index Terms
- Heuristic Evaluation for Novice Programming Systems
Recommendations
Heuristic evaluation of ambient displays
CHI '03: Proceedings of the SIGCHI Conference on Human Factors in Computing SystemsWe present a technique for evaluating the usability and effectiveness of ambient displays. Ambient displays are abstract and aesthetic peripheral displays portraying non-critical information on the periphery of a user's attention. Although many ...
Heuristic evaluation of programming language features: two parallel programming case studies
PLATEAU '11: Proceedings of the 3rd ACM SIGPLAN workshop on Evaluation and usability of programming languages and toolsUsability is an important feature for programming languages, and user evaluations can provide invaluable feedback on language design. However, user studies which compare programming languages or systems are often very expensive and typically ...
Heuristic-Based Usability Evaluation Support: A Systematic Literature Review and Comparative Study
Interacción '21: Proceedings of the XXI International Conference on Human Computer InteractionHeuristic evaluation is a popular inspection method principally used for the assessment of usability according to expert-based criteria. Since its (practical) emergence in the early 90s, it has become one of the most widely-used method, mainly utilized ...
Comments