An empirical comparison of the accuracy rates of novices using the quorum, perl, and randomo programming languages

ABSTRACT

We present here an empirical study comparing the accuracy rates of novices writing software in three programming languages: Quorum, Perl, and Randomo. The first language, Quorum, we call an evidence-based programming language, where the syntax, semantics, and API designs change in correspondence to the latest academic research and literature on programming language usability. Second, while Perl is well known, we call Randomo a Placebo-language, where some of the syntax was chosen with a random number generator and the ASCII table. We compared novices that were programming for the first time using each of these languages, testing how accurately they could write simple programs using common program constructs (e.g., loops, conditionals, functions, variables, parameters). Results showed that while Quorum users were afforded significantly greater accuracy compared to those using Perl and Randomo, Perl users were unable to write programs more accurately than those using a language designed by chance.

References

1. M. H. Brown and J. Hershberger. Colour and sound in algorithm animation. In Proceedings of the IEEE Workshop on Visual Languages, pages 10--17, Los Alamitos, CA, 1991. IEEE Computer Society Press. doi: 10.1109/WVL.1991.238856.
2. R. Garlick and E. C. Cankaya. Using Alice in CS1: A quantitative experiment. In Proceedings of the Fifteenth Annual Conference on Innovation and Technology in Computer Science Education, ITiCSE '10, pages 165--168, New York, NY, 2010. ACM. ISBN 978-1-60558-820-9. doi: http://doi.acm.org/10.1145/1822090.1822138.
3. S. Hanenberg. Faith, hope, and love: an essay on software science's neglect of human factors. In Proceedings of the ACM International Conference on Object Oriented Programming Systems Languages and Applications, OOPSLA '10, pages 933--946, New York, NY, 2010. ACM. ISBN 978-1-4503-0203-6. doi: http://doi.acm.org/10.1145/1869459.1869536.
4. R. C. Holt and J. R. Cordy. The Turing programming language. Communications of the ACM, 31: 1410--1423, December 1988. ISSN 0001-0782. doi: http://doi.acm.org/10.1145/53580.53581.
5. L. Hubert. Kappa revisited. Psychological Bulletin, 84 (2): 289--297, 1977.
6. C. D. Hundhausen, S. F. Farley, and J. L. Brown. Can direct manipulation lower the barriers to computer programming and promote transfer of training?: An experimental study. ACM Transactions on Computer Human Interaction, 16 (3): 1--40, 2009. ISSN 1073-0516. doi: http://doi.acm.org/10.1145/1592440.1592442.
7. C. Kelleher and R. Pausch. Lowering the barriers to programming: A taxonomy of programming environments and languages for novice programmers. ACM Computing Surveys, 37 (2): 83--137, 2005. ISSN 0360-0300. doi: http://doi.acm.org/10.1145/1089733.1089734.
8. A. J. Ko and B. A. Myers. Finding causes of program output with the java whyline. In CHI '09: Proceedings of the 27th International Conference on Human Factors in Computing Systems, pages 1569--1578, New York, NY, USA, 2009. ACM. ISBN 978-1-60558-246-7. doi: http://doi.acm.org/10.1145/1518701.1518942.
9. S. Markstrum. Staking claims: a history of programming language design claims and evidence: A positional work in progress. In Evaluation and Usability of Programming Languages and Tools, PLATEAU '10, pages 7:1--7:5, New York, NY, USA, 2010. ACM. ISBN 978-1-4503-0547-1. doi: http://doi.acm.org/10.1145/1937117.1937124.
10. L. K. McIver. Syntactic and Semantic Issues in Introductory Programming Education. PhD thesis, Monash University, 2001.
11. B. A. Myers, A. J. Ko, S. Y. Park, J. Stylos, T. D. LaToza, and J. Beaton. More natural end-user software engineering. In WEUSE '08: Proceedings of the 4th International Workshop on End-User Software Engineering, pages 30--34, New York, NY, USA, 2008. ACM. ISBN 978-1-60558-034-0. doi: http://doi.acm.org/10.1145/1370847.1370854.
12. N. Pennington. Comprehension strategies in programming. In G. M. Olson, S. Sheppard, E. Soloway, and B. Shneiderman, editors, Empirical Studies of Programmers: Second Workshop, pages 100--113, Westport, CT, 1987. Greenwood Publishing Group Inc.
13. G. Pothier, E. Tanter, and J. Piquer. Scalable omniscient debugging. In OOPSLA '07: Proceedings of the 22nd Annual ACM SIGPLAN Conference on Object-Oriented Programming Systems and Applications, pages 535--552, New York, NY, 2007. ACM. ISBN 978-1-59593-786-5. doi: http://doi.acm.org/10.1145/1297027.1297067.
14. A. Stefik and E. Gellenbeck. Empirical studies on programming language stimuli. Software Quality Journal, 19 (1): 65--99, 2011. ISSN 0963-9314. doi: http://dx.doi.org/10.1007/s11219-010-9106-7.
15. A. Stefik, C. Hundhausen, and R. Patterson. An empirical investigation into the design of auditory cues to enhance computer program comprehension. International Journal of Human-Computer Studies, 69 (12): 820--838, 2011. ISSN 1071-5819. 10.1016/j.ijhcs.2011.07.002. URL http://www.sciencedirect.com/science/article/pii/S1071581911000814.
16. A. Stefik, C. Hundhausen, and D. Smith. On the design of an educational infrastructure for the blind and visually impaired in computer science. In Proceedings of The 42nd ACM Technical Symposium on Computer Science Education, New York, NY, 2011. ACM.
17. J. Stylos and S. Clarke. Usability implications of requiring parameters in objects' constructors. Software Engineering, 2007. ICSE 2007. 29th International Conference on, pages 529--539, May 2007. ISSN 0270-5257. doi: 10.1109/ICSE.2007.92.
18. J. Stylos and B. A. Myers. The implications of method placement on API learnability. In SIGSOFT '08/FSE-16: Proceedings of the 16th ACM SIGSOFT International Symposium on Foundations of Software Engineering, pages 105--112, New York, NY, USA, 2008. ACM. ISBN 978-1-59593-995-1. doi: http://doi.acm.org/10.1145/1453101.1453117.
19. J. Taylor. Analysing novices analysing prolog: what stories do novices tell themselves about prolog? Instructional Science, 19: 283--309, 1990. ISSN 0020-4277. doi: http://dx.doi.org/10.1007/BF00116442.

Index Terms

• Computing methodologies

  • Artificial intelligence

    • Philosophical/theoretical foundations of artificial intelligence

      • Cognitive science

• Human-centered computing

  • Human computer interaction (HCI)

    • Empirical studies in HCI

• Software and its engineering

  • Software notations and tools

    • General programming languages