Empirical Support for a Causal Relationship Between Gamification and Learning Outcomes

ABSTRACT

Preparing for exams is an important yet stressful time for many students. Self-testing is known to be an effective preparation strategy, yet some students lack motivation to engage or persist in self-testing activities. Adding game elements to a platform supporting self-testing may increase engagement and, by extension, exam performance. We conduct a randomized controlled experiment (n=701) comparing the effect of two game elements -- a points system and a badge system -- used individually and in combination. We find that the badge system elicits significantly higher levels of voluntary self-testing activity and this effect is particularly pronounced amongst a relatively small cohort. Importantly, this increased activity translates to a significant improvement in exam scores. Our data supports a causal relationship between gamification and learning outcomes, mediated by self-testing behavior. This provides empirical support for Landers' theory of gamified learning when the gamified activity is conducted prior to measuring learning outcomes.

References

1. Judd Antin and Elizabeth Churchill. 2011. Badges in Social Media : A Social Psychological Perspective. In CHI 2011 Gamification Workshop Proceedings. ACM, Vancouver, BC, Canada. http://research.yahoo.com/node/3469
2. Michael B. Armstrong and Richard N. Landers. 2017. An Evaluation of Gamified Training: Using Narrative to Improve Reactions and Learning. Simulation & Gaming 48, 4 (2017), 513--538.
3. Ryan Baker, Jason Walonoski, Neil Heffernan, Ido Roll, Albert Corbett, and Kenneth Koedinger. 2008. Why Students Engage in "Gaming the System" Behavior in Interactive Learning Environments. Journal of Interactive Learning Research 19, 2 (April 2008), 185--224. https://www.learntechlib.org/p/24328
4. G. Barata, S. Gama, J. Jorge, and D. Goncalves. 2013a. Engaging Engineering Students with Gamification. In 5th International Conference on Games and Virtual Worlds for Serious Applications (VS-GAMES 2013). 1--8.
5. Gabriel Barata, Sandra Gama, Joaquim Jorge, and Daniel Gonçalves. 2013b. Improving Participation and Learning with Gamification. In Proceedings of the First International Conference on Gameful Design, Research, and Applications (Gamification '13). ACM, New York, NY, USA, 10--17.
6. Reuben M. Baron and David A. Kenny. 1986. The moderator-mediator variable distinction in social psychological research: Conceptual, strategic, and statistical considerations. Journal of Personality and Social Psychology 51 (1986), 1173--1182.
7. Kenneth G Brown. 2001. Using Computers to Deliver Training: Which Employees Learn and Why? Personnel Psychology 54, 2 (2001), 271--296.
8. Patrick Buckley and Elaine Doyle. 2016. Gami?cation and student motivation. Interactive Learning Environments 24, 6 (2016), 1162--1175.
9. Rachel C. Callan, Kristina N. Bauer, and Richard N. Landers. 2015. How to Avoid the Dark Side of Gamification: Ten Business Scenarios and Their Unintended Consequences. Springer International Publishing, Cham, 553--568.
10. Christopher Cheong, France Cheong, and Justin Filippou. 2013. QuickQuiz: A gamified Approach for Enhancing Learning. In Proceedings of the Pacific Asia Conference on Information Systems (PACIS 2013). 1--14.
11. Michelene Chi, Miriam Bassok, Matthew Lewis, Peter Reimann, and Robert Glaser. 1989. Self-Explanations: How Students Study and Use Examples in Learning to Solve Problems. Cognitive Science 13 (1989), 145--182.
12. Michelene Chi, Nicholas Leeuw, Mei-Hung Chiu, and Christian LaVancher. 1994. Eliciting Self-Explanations Improves Understanding. Cognitive Science 18 (1994), 439--477.
13. Luis De-Marcos, Adrián Domínguez, Joseba Saenz-de Navarrete, and Carmen Pagés. 2014. An Empirical Study Comparing Gamification and Social Networking on e-Learning. Computers & Education 75 (2014), 82 -- 91.
14. Simone de Sousa Borges, Vinicius H. S. Durelli, Helena Macedo Reis, and Seiji Isotani. 2014. A Systematic Mapping on Gamification Applied to Education. In Proceedings of the 29th Annual ACM Symposium on Applied Computing (SAC '14). ACM, New York, NY, USA, 216--222.
15. Paul Denny. 2013. The Effect of Virtual Achievements on Student Engagement. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '13). ACM, New York, NY, USA, 763--772.
16. Paul Denny, Andrew Luxton-Reilly, and John Hamer. 2008. The PeerWise System of Student Contributed Assessment Questions. In Proceedings of the Tenth Conference on Australasian Computing Education Volume 78 (ACE '08). Australian Computer Society, Inc., Darlinghurst, Australia, Australia, 69--74. http://dl.acm.org/citation.cfm?id=1379249.1379255
17. Sebastian Deterding. 2012. Gamification: designing for motivation. interactions 19, 4 (July 2012), 14--17.
18. Sebastian Deterding, Dan Dixon, Rilla Khaled, and Lennart Nacke. 2011. From Game Design Elements to Gamefulness: Defining "Gamification". In Proceedings of the 15th International Academic MindTrek Conference: Envisioning Future Media Environments (MindTrek '11). ACM, New York, NY, USA, 9--15.
19. Darina Dicheva, Christo Dichev, Gennady Agre, and Galia Angelova. 2015. Gamification in Education: A Systematic Mapping Study. Educational Technology & Society 18, 3 (2015), 75--88. http://dblp.uni-trier.de/ db/journals/ets/ets18.html#DichevaDAA15
20. John Dunlosky. 2013. Strengthening the Student Toolbox: Study Strategies to Boost Learning. American Educator 37, 3 (September 2013), 12--21.
21. Rosta Farzan, Joan M. DiMicco, David R. Millen, Casey Dugan, Werner Geyer, and Elizabeth A. Brownholtz. 2008. Results from Deploying a Participation Incentive Mechanism Within the Enterprise. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '08). ACM, New York, NY, USA, 563--572.
22. Zachary Fitz-Walter, Dian Tjondronegoro, and Peta Wyeth. 2012. A Gamified Mobile Application for Engaging New Students at University Orientation. In Proceedings of the 24th Australian Computer-Human Interaction Conference (OzCHI '12). ACM, New York, NY, USA, 138--141.
23. Paul W. Foos, Joseph J. Mora, and Sharon Tkacz. 1994. Student Study Techniques and the Generation Effect. Journal of Educational Psychology 86, 4 (1994), 567--576.
24. Seamus F. Forde, Elisa D. Mekler, and Klaus Opwis. 2016. Informational, but Not Intrinsically Motivating Gamification': Preliminary Findings. In Proceedings of the 2016 Annual Symposium on Computer-Human Interaction in Play Companion Extended Abstracts (CHI PLAY Companion '16). ACM, New York, NY, USA, 157--163.
25. Lawrence Frase and Barry Schwartz. 1975. Effect of Question Production and Answering on Prose Recall. Journal of Educational Psychology 67, 5 (1975), 628--635.
26. Lasse Hakulinen, Tapio Auvinen, and Ari Korhonen. 2015. The Effect of Achievement Badges on Students' Behavior: An Empirical Study in a University-Level Computer Science Course. International Journal of Emerging Technologies in Learning (iJET) 10, 1 (2015), 18--29. http://online-journals.org/index.php/i-jet/ article/view/4221
27. Juho Hamari. 2015. Do Badges Increase User Activity? A Field Experiment on the Effects of Gamification. Computers in Human Behavior (2015). (In press)
28. Juho Hamari, Jonna Koivisto, and Harri Sarsa. 2014. Does Gamification Work? -- A Literature Review of Empirical Studies on Gamification. In Proceedings of the 2014 47th Hawaii International Conference on System Sciences (HICSS '14). IEEE Computer Society, Washington, DC, USA, 3025--3034.
29. Michael D. Hanus and Jesse Fox. 2015. Assessing the Effects of Gamification in the Classroom. Comput. Educ. 80, C (Jan. 2015), 152--161.
30. Panagiotis G. Ipeirotis and Evgeniy Gabrilovich. 2014. Quizz: Targeted Crowdsourcing with a Billion (Potential) Users. In Proceedings of the 23rd International Conference on World Wide Web (WWW '14). ACM, New York, NY, USA, 143--154.
31. Yuan Jia, Yikun Liu, Xing Yu, and Stephen Voida. 2017. Designing Leaderboards for Gamification: Perceived Differences Based on User Ranking, Application Domain, and Personality Traits. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (CHI '17). ACM, New York, NY, USA, 1949--1960.
32. J. H. Jung, Christoph Schneider, and Joseph Valacich. 2010. Enhancing the Motivational Affordance of Information Systems: The Effects of Real-Time Performance Feedback and Goal Setting in Group Collaboration Environments. Management Science 56, 4 (April 2010), 724--742.
33. Richard Landers and Rachel C. Callan. 2011. Casual Social Games as Serious Games: The Psychology of Gamification in Undergraduate Education and Employee Training. Springer-Verlag London Ltd.
34. Richard N. Landers. 2014. Developing a Theory of Gamified Learning: Linking Serious Games and Gamification of Learning. Simul. Gaming 45, 6 (Dec. 2014), 752--768.
35. Richard N. Landers and Amy K. Landers. 2014. An Empirical Test of the Theory of Gamified Learning: The Effect of Leaderboards on Time-on-Task and Academic Performance. Simulation & Gaming 45, 6 (2014), 769--785.
36. Joey J. Lee and Jessica Hammer. 2011. Gamification in Education: What, How, Why Bother? Academic Exchange Quarterly 15, 2 (2011).
37. Wei Li, Tovi Grossman, and George Fitzmaurice. 2012. GamiCAD: A Gamified Tutorial System for First Time Autocad Users. In Proceedings of the 25th Annual ACM Symposium on User Interface Software and Technology (UIST '12). ACM, New York, NY, USA, 103--112.
38. Elaine Massung, David Coyle, Kirsten F. Cater, Marc Jay, and Chris Preist. 2013. Using Crowdsourcing to Support Pro-environmental Community Activism. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '13). ACM, New York, NY, USA, 371--380.
39. R. McDaniel, R. Lindgren, and J. Friskics. 2012. Using Badges for Shaping Interactions in Online Learning Environments. In Professional Communication Conference (IPCC), 2012 IEEE International. 1--4.
40. Elisa D. Mekler, Florian Brühlmann, Klaus Opwis, and Alexandre N. Tuch. 2013. Do Points, Levels and Leaderboards Harm Intrinsic Motivation?: An Empirical Analysis of Common Gamification Elements. In Proceedings of the First International Conference on Gameful Design, Research, and Applications (Gamification '13). ACM, New York, NY, USA, 66--73.
41. Elisa D. Mekler, Florian Brühlmann, Alexandre N. Tuch, and Klaus Opwis. 2017. Towards understanding the effects of individual gamification elements on intrinsic motivation and performance. Computers in Human Behavior 71 (2017), 525 -- 534.
42. Benedikt Morschheuser, Juho Hamari, and Jonna Koivisto. 2016. Gamification in Crowdsourcing: A Review. In HICSS, Tung X. Bui and Ralph H. Sprague Jr. (Eds.). IEEE Computer Society, 4375--4384.
43. Siobhan O'Donovan, James Gain, and Patrick Marais. 2013. A Case Study in the Gamification of a University-level Games Development Course. In Proceedings of the South African Institute for Computer Scientists and Information Technologists Conference (SAICSIT '13). ACM, New York, NY, USA, 242--251.
44. Rita Orji, Lennart E. Nacke, and Chrysanne Di Marco. 2017. Towards Personality-driven Persuasive Health Games and Gamified Systems. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (CHI '17). ACM, New York, NY, USA, 1015--1027.
45. Henry L. Roediger and Andrew C. Butler. 2011. The Critical Role of Retrieval Practice in Long-Term Retention. Trends in Cognitive Sciences 15, 1 (2011), 20 -- 27.
46. Henry L. Roediger and Andrew C. Butler. 2013. Retrieval Practice (Testing) Effect. In H. L. Pashler (Ed.), Encyclopedia of the Mind, Sage Publishing Co. (2013), 660--661.
47. Yves Rosseel. 2012. lavaan: An R Package for Structural Equation Modeling. Journal of Statistical Software 48, 2 (2012), 1--36. http://www.jstatsoft.org/v48/i02/
48. Richard M. Ryan and Edward L. Deci. 2000. Intrinsic and Extrinsic Motivations: Classic Definitions and New Directions. Contemporary Educational Psychology 25, 1 (2000), 54 -- 67.
49. Michael Sailer, Jan Ulrich Hense, Sarah Katharina Mayr, and Heinz Mandl. 2017. How gamification motivates: An experimental study of the effects of specific game design elements on psychological need satisfaction. Computers in Human Behavior 69 (2017), 371 -- 380.
50. C. Scott Rigby, Edward L. Deci, Brian C. Patrick, and Richard M. Ryan. 1992. Beyond the Intrinsic-Extrinsic Dichotomy: Self-Determination in Motivation and Learning. Motivation and Emotion 16, 3 (1992), 165--185.
51. Katie Seaborn and Deborah I. Fels. 2015. Gamification in theory and action: A survey. International Journal of Human-Computer Studies 74 (2015), 14 -- 31.
52. Patrick E. Shrout and Niall Bolger. 2002. Mediation in Experimental and Nonexperimental Studies: New Procedures and Recommendations. Psychological Methods 7, 4 (2002), 422--445.
53. Norman Slamecka and Peter Graf. 1978. The Generation Effect: Delineation of a Phenomenon. Journal of Experimental Psychology 4 (1978), 592--604.

Index Terms

• Applied computing

  • Education

• Human-centered computing

• Social and professional topics

  • Professional topics

    • Computing education