April Yi Wang
Steve Oney
Skip Abstract Section
Abstract
Peer assessment, as a form of collaborative learning, can engage students in active learning and improve their learning gains. However, current teaching platforms and programming environments provide little support to integrate peer assessment for in-class programming exercises. We identified challenges in conducting such exercises and adopting peer assessment through formative interviews with instructors of introductory programming courses. To address these challenges, we introduce PuzzleMe, a tool to help Computer Science instructors to conduct engaging in-class programming exercises. PuzzleMe leverages peer assessment to support a collaboration model where students provide timely feedback on their peers' work. We propose two assessment techniques tailored to in-class programming exercises: live peer testing and live peer code review. Live peer testing can improve students' code robustness by allowing them to create and share lightweight tests with peers. Live peer code review can improve code understanding by intelligently grouping students to maximize meaningful code reviews. A two-week deployment study revealed that PuzzleMe encourages students to write useful test cases, identify code problems, correct misunderstandings, and learn a diverse set of problem-solving approaches from peers.
- Susan A Ambrose, Michael W Bridges, Michele DiPietro, Marsha C Lovett, and Marie K Norman. 2010. How learning works: Seven research-based principles for smart teaching. John Wiley & Sons.Google Scholar
- John R. Anderson and Edward Skwarecki. 1986. The automated tutoring of introductory computer programming. Commun. ACM, Vol. 29, 9 (1986), 842--849.Google ScholarDigital Library
- Marcel Borowski, Johannes Zagermann, Clemens N Klokmose, Harald Reiterer, and Roman Radle. 2020. Exploring the Benefits and Barriers of Using Computational Notebooks for Collaborative Programming Assignments. In Proceedings of the 51st ACM Technical Symposium on Computer Science Education. 468--474.Google ScholarDigital Library
- Brian Burg, Amy J. Ko, and Michael D. Ernst. 2015. Explaining Visual Changes in Web Interfaces. In Proceedings of the 28th Annual ACM Symposium on User Interface Software & Technology (Charlotte, NC, USA) (UIST '15). Association for Computing Machinery, New York, NY, USA, 259--268. https://doi.org/10.1145/2807442.2807473Google ScholarDigital Library
- Jeongmin Byun, Jungkook Park, and Alice Oh. 2020. Cocode: Co-learner Screen Sharing for Social Translucence in Online Programming Courses. In Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems. 1--4.Google Scholar
- Julia Cambre, Scott Klemmer, and Chinmay Kulkarni. 2018. Juxtapeer: Comparative Peer Review Yields Higher Quality Feedback and Promotes Deeper Reflection. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (Montreal QC, Canada) (CHI '18). Association for Computing Machinery, New York, NY, USA, Article 294, 13 pages. https://doi.org/10.1145/3173574.3173868Google ScholarDigital Library
- Charles H. Chen and Philip J. Guo. 2019. Improv: Teaching Programming at Scale via Live Coding. In Proceedings of the Sixth (2019) ACM Conference on Learning @ Scale (Chicago, IL, USA) (L@S '19). Association for Computing Machinery, New York, NY, USA, Article 9, 10 pages. https://doi.org/10.1145/3330430.3333627Google ScholarDigital Library
- Yan Chen, Jaylin Herskovitz, Gabriel Matute, April Wang, Sang Won Lee, Walter S Lasecki, and Steve Oney. 2020 a. EdCode: Towards Personalized Support at Scale for Remote Assistance in CS Education. In 2020 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC). IEEE, 1--5.Google ScholarCross Ref
- Yan Chen, Walter S Lasecki, and Tao Dong. 2021. Towards Supporting Programming Education at Scale via Live Streaming. Proceedings of the ACM on Human-Computer Interaction, Vol. 4, CSCW3 (2021), 1--19.Google ScholarDigital Library
- Yan Chen, Maulishree Pandey, Jean Y Song, Walter S Lasecki, and Steve Oney. 2020 b. Improving Crowd-Supported GUI Testing with Structural Guidance. In Proceedings of the SIGCHI conference on human factors in computing systems (CHI '20). Association for Computing Machinery, New York, NY, USA. https://doi.org/10.1145/3313831.3376835Google ScholarDigital Library
- Dave Clarke, Tony Clear, Kathi Fisler, Matthias Hauswirth, Shriram Krishnamurthi, Joe Gibbs Politz, Ville Tirronen, and Tobias Wrigstad. 2014. In-Flow Peer Review. In Proceedings of the Working Group Reports of the 2014 on Innovation & Technology in Computer Science Education Conference (Uppsala, Sweden) (ITiCSE-WGR '14). Association for Computing Machinery, New York, NY, USA, 59--79. https://doi.org/10.1145/2713609.2713612Google ScholarDigital Library
- Diana Cordova and Mark Lepper. 1996. Intrinsic Motivation and the Process of Learning: Beneficial Effects of Contextualization, Personalization, and Choice. Journal of Educational Psychology, Vol. 88 (12 1996), 715--730. https://doi.org/10.1037/0022-0663.88.4.715Google ScholarCross Ref
- Catherine H Crouch and Eric Mazur. 2001. Peer instruction: Ten years of experience and results. American journal of physics, Vol. 69, 9 (2001), 970--977.Google Scholar
- Jill Denner, Linda Werner, Shannon Campe, and Eloy Ortiz. 2014. Pair programming: Under what conditions is it advantageous for middle school students? Journal of Research on Technology in Education, Vol. 46, 3 (2014), 277--296.Google ScholarCross Ref
- Paul Denny, Andrew Luxton-Reilly, Ewan Tempero, and Jacob Hendrickx. 2011. CodeWrite: Supporting Student-Driven Practice of Java. In Proceedings of the 42nd ACM Technical Symposium on Computer Science Education (Dallas, TX, USA) (SIGCSE '11). Association for Computing Machinery, New York, NY, USA, 471--476. https://doi.org/10.1145/1953163.1953299Google ScholarDigital Library
- Collaborative editing library. 2020. ShareDB. https://github.com/share/sharedb Accessed: April, 2020.Google Scholar
- Stephen H. Edwards and Krishnan Panamalai Murali. 2017. CodeWorkout: Short Programming Exercises with Built-in Data Collection. In Proceedings of the 2017 ACM Conference on Innovation and Technology in Computer Science Education (Bologna, Italy) (ITiCSE '17). Association for Computing Machinery, New York, NY, USA, 188--193. https://doi.org/10.1145/3059009.3059055Google ScholarDigital Library
- Carolyn D. Egelman, Emerson Murphy-Hill, Elizabeth Kammer, Margaret Morrow Hodges, Collin Green, Ciera Jaspan, and James Lin. 2020. Predicting Developers' Negative Feelings about Code Review. In 2020 IEEE/ACM 42nd International Conference on Software Engineering (ICSE '20). IEEE. https://doi.org/10.1145/3377811.3380414Google ScholarDigital Library
- Sebastian Elbaum, Suzette Person, Jon Dokulil, and Matt Jorde. 2007. Bug hunt: Making early software testing lessons engaging and affordable. In 29th International Conference on Software Engineering (ICSE'07). IEEE, 688--697.Google ScholarDigital Library
- Jeremiah T Folsom-Kovarik, Sae Schatz, and Denise Nicholson. [n.d.]. Plan ahead: Pricing ITS learner models.Google Scholar
- Elena L. Glassman, Lyla Fischer, Jeremy Scott, and Robert C. Miller. 2015a. Foobaz: Variable Name Feedback for Student Code at Scale. In Proceedings of the 28th Annual ACM Symposium on User Interface Software & Technology (Charlotte, NC, USA) (UIST '15). Association for Computing Machinery, New York, NY, USA, 609--617. https://doi.org/10.1145/2807442.2807495Google ScholarDigital Library
- Elena L. Glassman, Aaron Lin, Carrie J. Cai, and Robert C. Miller. 2016. Learnersourcing Personalized Hints. In Proceedings of the 19th ACM Conference on Computer-Supported Cooperative Work & Social Computing (San Francisco, California, USA) (CSCW '16). Association for Computing Machinery, New York, NY, USA, 1626--1636. https://doi.org/10.1145/2818048.2820011Google ScholarDigital Library
- Elena L. Glassman, Jeremy Scott, Rishabh Singh, Philip J. Guo, and Robert C. Miller. 2015b. OverCode: Visualizing Variation in Student Solutions to Programming Problems at Scale. ACM Trans. Comput.-Hum. Interact., Vol. 22, 2, Article 7 (March 2015), 35 pages. https://doi.org/10.1145/2699751Google ScholarDigital Library
- Max Goldman, Greg Little, and Robert C. Miller. 2011. Real-Time Collaborative Coding in a Web IDE. In Proceedings of the 24th Annual ACM Symposium on User Interface Software and Technology (Santa Barbara, California, USA) (UIST '11). Association for Computing Machinery, New York, NY, USA, 155--164. https://doi.org/10.1145/2047196.2047215Google ScholarDigital Library
- Scott Grissom and Mark J Van Gorp. 2000. A practical approach to integrating active and collaborative learning into the introductory computer science curriculum. In Proceedings of the seventh annual CCSC Midwestern conference on Small colleges. 95--100.Google ScholarDigital Library
- Sumit Gulwani, Ivan Radivcek, and Florian Zuleger. 2018. Automated Clustering and Program Repair for Introductory Programming Assignments. In Proceedings of the 39th ACM SIGPLAN Conference on Programming Language Design and Implementation (Philadelphia, PA, USA) (PLDI 2018). Association for Computing Machinery, New York, NY, USA, 465--480. https://doi.org/10.1145/3192366.3192387Google ScholarDigital Library
- Philip J. Guo. 2015. Codeopticon: Real-Time, One-To-Many Human Tutoring for Computer Programming. In Proceedings of the 28th Annual ACM Symposium on User Interface Software & Technology (Charlotte, NC, USA) (UIST '15). Association for Computing Machinery, New York, NY, USA, 599--608. https://doi.org/10.1145/2807442.2807469Google ScholarDigital Library
- Philip J Guo, Jeffery White, and Renan Zanelatto. 2015. Codechella: Multi-user program visualizations for real-time tutoring and collaborative learning. In 2015 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC). IEEE, 79--87.Google ScholarCross Ref
- Andrew Head, Elena Glassman, Gustavo Soares, Ryo Suzuki, Lucas Figueredo, Loris D'Antoni, and Björn Hartmann. 2017. Writing Reusable Code Feedback at Scale with Mixed-Initiative Program Synthesis. In Proceedings of the Fourth (2017) ACM Conference on Learning @ Scale (Cambridge, Massachusetts, USA) (L@S '17). Association for Computing Machinery, New York, NY, USA, 89--98. https://doi.org/10.1145/3051457.3051467Google ScholarDigital Library
- Catherine M. Hicks, Vineet Pandey, C. Ailie Fraser, and Scott Klemmer. 2016. Framing Feedback: Choosing Review Environment Features That Support High Quality Peer Assessment. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (San Jose, California, USA) (CHI '16). Association for Computing Machinery, New York, NY, USA, 458--469. https://doi.org/10.1145/2858036.2858195Google ScholarDigital Library
- Christopher D Hundhausen, Anukrati Agrawal, and Pawan Agarwal. 2013. Talking about code: Integrating pedagogical code reviews into early computing courses. ACM Transactions on Computing Education (TOCE), Vol. 13, 3 (2013), 1--28.Google ScholarDigital Library
- David Janzen and Hossein Saiedian. 2008. Test-Driven Learning in Early Programming Courses. In Proceedings of the 39th SIGCSE Technical Symposium on Computer Science Education (Portland, OR, USA) (SIGCSE '08). Association for Computing Machinery, New York, NY, USA, 532--536. https://doi.org/10.1145/1352135.1352315Google ScholarDigital Library
- David S. Janzen and Hossein Saiedian. 2006. Test-Driven Learning: Intrinsic Integration of Testing into the CS/SE Curriculum. In Proceedings of the 37th SIGCSE Technical Symposium on Computer Science Education (Houston, Texas, USA) (SIGCSE '06). Association for Computing Machinery, New York, NY, USA, 254--258. https://doi.org/10.1145/1121341.1121419Google ScholarDigital Library
- Juho Kim et al. 2015. Learnersourcing: improving learning with collective learner activity. Ph.D. Dissertation. Massachusetts Institute of Technology.Google Scholar
- Ingo Kollar and Frank Fischer. 2010. Peer assessment as collaborative learning: A cognitive perspective. Learning and Instruction, Vol. 20, 4 (2010), 344--348.Google ScholarCross Ref
- Yasmine Kotturi, Chinmay E Kulkarni, Michael S Bernstein, and Scott Klemmer. 2015. Structure and messaging techniques for online peer learning systems that increase stickiness. In Proceedings of the Second (2015) ACM Conference on Learning@ Scale. 31--38.Google ScholarDigital Library
- Chinmay Kulkarni, Koh Pang Wei, Huy Le, Daniel Chia, Kathryn Papadopoulos, Justin Cheng, Daphne Koller, and Scott R. Klemmer. 2013. Peer and Self Assessment in Massive Online Classes. ACM Trans. Comput.-Hum. Interact., Vol. 20, 6, Article 33 (Dec. 2013), 31 pages. https://doi.org/10.1145/2505057Google ScholarDigital Library
- Chinmay E. Kulkarni, Michael S. Bernstein, and Scott R. Klemmer. 2015. PeerStudio: Rapid Peer Feedback Emphasizes Revision and Improves Performance. In Proceedings of the Second (2015) ACM Conference on Learning @ Scale (Vancouver, BC, Canada) (L@S '15). Association for Computing Machinery, New York, NY, USA, 75--84. https://doi.org/10.1145/2724660.2724670Google ScholarDigital Library
- Patrick R Lowenthal. 2010. Social presence. In Social computing: Concepts, methodologies, tools, and applications. IGI global, 129--136.Google Scholar
- John H Maloney, Kylie Peppler, Yasmin Kafai, Mitchel Resnick, and Natalie Rusk. 2008. Programming by choice: urban youth learning programming with scratch. In Proceedings of the 39th SIGCSE technical symposium on Computer science education. 367--371.Google ScholarDigital Library
- Joseph Bahman Moghadam, Rohan Roy Choudhury, HeZheng Yin, and Armando Fox. 2015. AutoStyle: Toward Coding Style Feedback at Scale. In Proceedings of the Second (2015) ACM Conference on Learning @ Scale (Vancouver, BC, Canada) (L@S '15). Association for Computing Machinery, New York, NY, USA, 261--266. https://doi.org/10.1145/2724660.2728672Google ScholarDigital Library
- Thi Thao Duyen T. Nguyen, Thomas Garncarz, Felicia Ng, Laura A. Dabbish, and Steven P. Dow. 2017. Fruitful Feedback: Positive Affective Language and Source Anonymity Improve Critique Reception and Work Outcomes. In Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing (Portland, Oregon, USA) (CSCW '17). Association for Computing Machinery, New York, NY, USA, 1024--1034. https://doi.org/10.1145/2998181.2998319Google ScholarDigital Library
- Steve Oney, Christopher Brooks, and Paul Resnick. 2018. Creating Guided Code Explanations with chat. codes. Proceedings of the ACM on Human-Computer Interaction, Vol. 2, CSCW (2018), 131.Google Scholar
- Thanaporn Patikorn and Neil T. Heffernan. 2020. Effectiveness of Crowd-Sourcing On-Demand Assistance from Teachers in Online Learning Platforms. In Proceedings of the Seventh ACM Conference on Learning @ Scale (Virtual Event, USA) (L@S '20). Association for Computing Machinery, New York, NY, USA, 115--124. https://doi.org/10.1145/3386527.3405912Google ScholarDigital Library
- Joe Gibbs Politz, Joseph M Collard, Arjun Guha, Kathi Fisler, and Shriram Krishnamurthi. 2016. The Sweep: Essential Examples for In-Flow Peer Review. In Proceedings of the 47th ACM Technical Symposium on Computing Science Education. 243--248.Google ScholarDigital Library
- Joe Gibbs Politz, Shriram Krishnamurthi, and Kathi Fisler. 2014. In-Flow Peer-Review of Tests in Test-First Programming. In Proceedings of the Tenth Annual Conference on International Computing Education Research (Glasgow, Scotland, United Kingdom) (ICER '14). Association for Computing Machinery, New York, NY, USA, 11--18. https://doi.org/10.1145/2632320.2632347Google ScholarDigital Library
- David Preston. 2005. Pair programming as a model of collaborative learning: a review of the research. Journal of Computing Sciences in colleges, Vol. 20, 4 (2005), 39--45.Google ScholarDigital Library
- Thomas W. Price, Joseph Jay Williams, Jaemarie Solyst, and Samiha Marwan. 2020. Engaging Students with Instructor Solutions in Online Programming Homework. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (Honolulu, HI, USA) (CHI '20). Association for Computing Machinery, New York, NY, USA, 1--7. https://doi.org/10.1145/3313831.3376857Google ScholarDigital Library
- Roman R"adle, Midas Nouwens, Kristian Antonsen, James R Eagan, and Clemens N Klokmose. 2017. Codestrates: Literate computing with webstrates. In Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology. 715--725.Google Scholar
- Kelly Rivers and Kenneth R. Koedinger. 2013. Automatic Generation of Programming Feedback; A Data-Driven Approach. In AIED Workshops .Google Scholar
- Kelly Rivers and Kenneth R Koedinger. 2015. Data-driven hint generation in vast solution spaces: a self-improving python programming tutor. International Journal of Artificial Intelligence in Education (2015), 1--28.Google Scholar
- José Miguel Rojas, Thomas D White, Benjamin S Clegg, and Gordon Fraser. 2017. Code defenders: crowdsourcing effective tests and subtle mutants with a mutation testing game. In 2017 IEEE/ACM 39th International Conference on Software Engineering (ICSE). IEEE, 677--688.Google ScholarDigital Library
- Marc J. Rubin. 2013. The Effectiveness of Live-Coding to Teach Introductory Programming. In Proceeding of the 44th ACM Technical Symposium on Computer Science Education (Denver, Colorado, USA) (SIGCSE '13). Association for Computing Machinery, New York, NY, USA, 651--656. https://doi.org/10.1145/2445196.2445388Google ScholarDigital Library
- Jeffrey Saltz and Robert Heckman. 2020. Using Structured Pair Activities in a Distributed Online Breakout Room. Online Learning, Vol. 24, 1 (2020), 227--244.Google ScholarCross Ref
- Rishabh Singh, Sumit Gulwani, and Armando Solar-Lezama. 2013. Automated Feedback Generation for Introductory Programming Assignments. In Proceedings of the 34th ACM SIGPLAN Conference on Programming Language Design and Implementation (Seattle, Washington, USA) (PLDI '13). Association for Computing Machinery, New York, NY, USA, 15--26. https://doi.org/10.1145/2491956.2462195Google ScholarDigital Library
- J. Sitthiworachart and M. Joy. 2003. Web-based peer assessment in learning computer programming. In Proceedings 3rd IEEE International Conference on Advanced Technologies. 180--184. https://doi.org/10.1109/ICALT.2003.1215052Google ScholarCross Ref
- Barbara Leigh Smith and Jean T MacGregor. 1992. What is collaborative learning.Google Scholar
- Joanna Smith, Joe Tessler, Elliot Kramer, and Calvin Lin. 2012. Using Peer Review to Teach Software Testing. In Proceedings of the Ninth Annual International Conference on International Computing Education Research (Auckland, New Zealand) (ICER '12). Association for Computing Machinery, New York, NY, USA, 93--98. https://doi.org/10.1145/2361276.2361295Google ScholarDigital Library
- Stephan Trahasch. 2004. From peer assessment towards collaborative learning. In 34th Annual Frontiers in Education, 2004. FIE 2004. IEEE, F3F--16.Google ScholarCross Ref
- Anne Venables and Liz Haywood. 2003. Programming Students NEED Instant Feedback!. In Proceedings of the Fifth Australasian Conference on Computing Education - Volume 20 (Adelaide, Australia) (ACE '03). Australian Computer Society, Inc., AUS, 267--272.Google Scholar
- Sarah Weir, Juho Kim, Krzysztof Z. Gajos, and Robert C. Miller. 2015. Learnersourcing Subgoal Labels for How-to Videos. In Proceedings of the 18th ACM Conference on Computer Supported Cooperative Work & Social Computing (Vancouver, BC, Canada) (CSCW '15). Association for Computing Machinery, New York, NY, USA, 405--416. https://doi.org/10.1145/2675133.2675219Google ScholarDigital Library
- Joseph Jay Williams, Juho Kim, Anna Rafferty, Samuel Maldonado, Krzysztof Z. Gajos, Walter S. Lasecki, and Neil Heffernan. 2016. AXIS: Generating Explanations at Scale with Learnersourcing and Machine Learning. In Proceedings of the Third (2016) ACM Conference on Learning @ Scale (Edinburgh, Scotland, UK) (L@S '16). Association for Computing Machinery, New York, NY, USA, 379--388. https://doi.org/10.1145/2876034.2876042Google ScholarDigital Library
- Benjamin Xie, Dastyni Loksa, Greg L. Nelson, Matthew J. Davidson, Dongsheng Dong, Harrison Kwik, Alex Hui Tan, Leanne Hwa, Min Li, and Amy J. Ko. 2019. A theory of instruction for introductory programming skills. Computer Science Education, Vol. 29, 2--3 (2019), 205--253. https://doi.org/10.1080/08993408.2019.1565235Google ScholarCross Ref
- Kimberly Michelle Ying and Kristy Elizabeth Boyer. 2020. Understanding Students' Needs for Better Collaborative Coding Tools. In Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems Extended Abstracts (Honolulu, HI, USA) (CHI '20). Association for Computing Machinery, New York, NY, USA, 1--8. https://doi.org/10.1145/3334480.3383068Google ScholarDigital Library
- Alvin Yuan, Kurt Luther, Markus Krause, Sophie Isabel Vennix, Steven P Dow, and Bjorn Hartmann. 2016a. Almost an Expert: The Effects of Rubrics and Expertise on Perceived Value of Crowdsourced Design Critiques. In Proceedings of the 19th ACM Conference on Computer-Supported Cooperative Work & Social Computing (San Francisco, California, USA) (CSCW '16). Association for Computing Machinery, New York, NY, USA, 1005--1017. https://doi.org/10.1145/2818048.2819953Google ScholarDigital Library
- Alvin Yuan, Kurt Luther, Markus Krause, Sophie Isabel Vennix, Steven P Dow, and Bjorn Hartmann. 2016b. Almost an Expert: The Effects of Rubrics and Expertise on Perceived Value of Crowdsourced Design Critiques. In Proceedings of the 19th ACM Conference on Computer-Supported Cooperative Work & Social Computing (San Francisco, California, USA) (CSCW '16). Association for Computing Machinery, New York, NY, USA, 1005--1017. https://doi.org/10.1145/2818048.2819953Google ScholarDigital Library
Index Terms
- PuzzleMe: Leveraging Peer Assessment for In-Class Programming Exercises
Recommendations
Student Perspectives of Peer Assessment in Programming Courses
UKICER '21: Proceedings of the 2021 Conference on United Kingdom & Ireland Computing Education ResearchPeer assessment is a common form of shared learning in which students provide feedback on each other's work. Despite the crucial role that students play in peer assessment, programming students’ perspectives on such practices are relatively under-...
On-line peer assessment and the role of the peer feedback: A study of high school computer course
The purposes of this study were to explore the effects and the validity of on-line peer assessment in high schools and to analyze effects of various types of peer feedback on students. The participating students, a total of 184 10th graders, developed ...
Does evaluating peer assessment accuracy and taking it into account in calculating assessor’s final score enhance online peer assessment quality?
AbstractPeer assessment has an important role in teaching and learning nowadays. However, the existing techniques tend to be limited due to a lack of a suitable means to evaluate the accuracy of peer assessment activities and supervise peer assessment ...
Comments