The Effect of Personalization in Longer-Term Robot Tutoring

Abstract

The benefits of personalized social robots must be evaluated in real-world educational contexts over periods of time longer than a single session to understand their full potential to impact learning outcomes. In this work, we describe a personalization system designed for longer-term personalization that orders curriculum based on an adaptive Hidden Markov Model (HMM) that evaluates students’ skill proficiencies. We present a study investigating the effectiveness of this system in a five-session interaction with a robot tutor, taking place over the course of 2 weeks. Our system is evaluated in the context of native Spanish-speaking first-graders interacting with a social robot tutor while completing an English Language Learning educational task. Participants either received lessons: (1) ordered by our adaptive HMM personalization system which selects a lesson based on a skill that the individual participant needs more practice with (“personalized condition”) or (2) ordered randomly from among the lessons the participant had not yet seen (“non-personalized condition”). We found that participants who received personalized lessons from the robot tutor outperformed participants who received non-personalized lessons on a post-test by 2.0 standard deviations on average, corresponding to a mean learning gain in the 98th percentile.

References

1. Minoo Alemi, Ali Meghdari, and Maryam Ghazisaedy. 2014. Employing humanoid robots for teaching English language in Iranian junior high-schools. Int. J. Human. Robot. 11, 3 (2014).
2. Wilma A. Bainbridge, Justin Hart, Elizabeth S. Kim, and Brian Scassellati. 2008. The effect of presence on human-robot interaction. In Proceedings of the 17th IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN’08). IEEE, 701--706.
3. R. S. Baker, A. T. Corbett, and K. R. Koedinger. 2004. Detecting student misuse of intelligent tutoring systems. In Intelligent Tutoring Systems. Springer, 531--540.
4. Paul Baxter, Emily Ashurst, Robin Read, James Kennedy, and Tony Belpaeme. 2017. Robot education peers in a situated primary school study: Personalisation promotes child learning. PLOS ONE 12, 5 (2017), e0178126.
5. Tony Belpaeme, James Kennedy, Aditi Ramachandran, Brian Scassellati, and Fumihide Tanaka. 2018. Social robots for education: A review. Sci. Robot. 3, 21 (2018). http://robotics.sciencemag.org/content/3/21/eaat5954.
6. Tony Belpaeme, Paul Vogt, Rianne van den Berghe, Kirsten Bergmann, Tilbe Göksun, Mirjam de Haas, Junko Kanero, James Kennedy, Aylin C Küntay, Ora Oudgenoeg-Paz, et al. 2018. Guidelines for designing social robots as second language tutors. Int. J. Soc. Robot. 10, 3 (2018), 325--341.
7. R. M. Callahan. 2005. Tracking and high school English learners: Limiting opportunity to learn. Am. Educ. Res. J. 42, 2 (2005), 305--328.
8. M. Chen and K. Zechner. 2011. Computing and evaluating syntactic complexity features for automated scoring of spontaneous non-native speech. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (HLT’11). Association for Computational Linguistics, Stroudsburg, PA, 722--731. http://dl.acm.org/citation.cfm?id=2002472.2002564
9. Caitlyn Clabaugh, Gisele Ragusa, Fei Sha, and Maja Matarić. 2015. Designing a socially assistive robot for personalized number concepts learning in preschool children. In Proceedings of the 2015 Joint IEEE International Conference on Development and Learning and Epigenetic Robotics (ICDL-EpiRob’15). IEEE, 314--319.
10. C. Conati and H. Maclaren. 2009. Empirically building and evaluating a probabilistic model of user affect. User Model. User-Adapt. Interact. 19, 3 (2009), 267--303.
11. Alexandre Coninx, Paul Baxter, Elletra Oleari, Sara Bellini, Bert Bierman, Olivier Blanson-Henkemans, Lola Canamero, Piero Cosi, Valentin Enescu, Raquel Ros, Antoine Hiolle, Remi Humbert, Bernd Kiefer, Ivana Kruijff-Korbayova, Rosemarijn Looije, Marco Mosconi, Mark Neerincx, Giulio Paci, Georgios Patsis, Clara Pozzi, Francesca Sacchitelli, Hichem Sahli, Alberto Sanna, Giacomo Sommavilla, Fabio Tesser, Yiannis Demiris, and Tony Belpaeme. 2016. Towards long-term social child-robot interaction: Using multi-activity switching to engage young users. J. Hum.-Robot Interact. 5, 1 (2016), 32--67.
12. Albert T. Corbett and John R. Anderson. 1994. Knowledge tracing: Modeling the acquisition of procedural knowledge. User Model. User-Adapt. Interact. 4, 4 (1994), 253--278.
13. Jan de Wit, Thorsten Schodde, Bram Willemsen, Kirsten Bergmann, Mirjam de Haas, Stefan Kopp, Emiel Krahmer, and Paul Vogt. 2018. The effect of a robot’s gestures and adaptive tutoring on children’s acquisition of second language vocabularies. In Proceedings of the 2018 ACM/IEEE International Conference on Human-Robot Interaction. ACM, 50--58.
14. M. C. Desmarais and R. S. Baker. 2012. A review of recent advances in learner and skill modeling in intelligent learning environments. User Model. User-Adapt.Interact. 22, 1--2 (Apr. 2012), 9--38.
15. S. D’Mello. 2012. Monitoring affective trajectories during complex learning. In Encyclopedia of the Sciences of Learning. Springer, 2325--2328.
16. M. Eskenazi. 2009. An overview of spoken language technology for education. Speech Commun. 51, 10 (2009), 832--844.
17. G. D. Forney Jr. 1973. The Viterbi algorithm. Proc. IEEE 61, 3 (1973), 268--278.
18. Goren Gordon and Cynthia Breazeal. 2015. Bayesian active learning-based robot tutor for children’s word-reading skills. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI’15). 1343--1349.
19. Goren Gordon, Samuel Spaulding, Jacqueline Kory Westlund, Jin Joo Lee, Luke Plummer, Marayna Martinez, Madhurima Das, and Cynthia Breazeal. 2016. Affective personalization of a social robot tutor for children’s second language skills. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI’16). 3951--3957.
20. Jillian Greczek, Elaine Short, Caitlyn E. Clabaugh, Katelyn Swift-Spong, and Maja Mataric. 2014. Socially assistive robotics for personalized education for children. In Proceedings of the AAAI Fall Symposium on Artificial Intelligence and Human-Robot Interaction (AI-HRI’14).
21. K. E. Hogan and M. E. Pressley. 1997. Scaffolding Student Learning: Instructional Approaches and Issues. Brookline Books.
22. Iris Howley, Takayuki Kanda, Kotaro Hayashi, and Carolyn Rosé. 2014. Effects of social presence and social role on help-seeking and learning. In Proceedings of the 2014 ACM/IEEE International Conference on Human-Robot Interaction. ACM, 415--422.
23. K. Humes, N. A. Jones, and R. R. Ramirez. 2011. Overview of Race and Hispanic Origin, 2010. U.S. Department of Commerce.
24. J. S. Johnson and E. L. Newport. 1989. Critical period effects in second language learning: The influence of maturational state on the acquisition of English as a second language. Cogn. Psychol. 21, 1 (1989), 60--99.
25. Aidan Jones and Ginevra Castellano. 2018. Adaptive robotic tutors that support self-regulated learning: A longer-term investigation with primary school children. Int. J. Soc. Robot. 10, 3 (2018), 357--370.
26. Takayuki Kanda, Takayuki Hirano, Daniel Eaton, and Hiroshi Ishiguro. 2004. Interactive robots as social partners and peer tutors for children: A field trial. Hum.-Comput. Interact. 19, 1 (2004), 61--84.
27. James Kennedy, Paul Baxter, Emmanuel Senft, and Tony Belpaeme. 2015. Higher nonverbal immediacy leads to greater learning gains in child-robot tutoring interactions. In Proceedings of the International Conference on Social Robotics. Springer, 327--336.
28. James Kennedy, Paul Baxter, Emmanuel Senft, and Tony Belpaeme. 2016. Social robot tutoring for child second language learning. In Proceedings of the 11th ACM/IEEE International Conference on Human-Robot Interaction (HRI’16). IEEE, 231--238.
29. Kenneth R. Koedinger. 2002. Toward evidence for instructional design principles: Examples from cognitive tutor math 6. In Proceedings of the Annual Meeting of the North American Chapter of the International Group for the Psychology of Mathematics Education. ERIC, 21--49.
30. C. C. Kulik, J. A. Kulik, and R. L. Bangert-Drowns. 1990. Effectiveness of mastery learning programs: A meta-analysis. Rev. Educ. Res. 60, 2 (1990), 265--299.
31. Sungjin Lee, Hyungjong Noh, Jonghoon Lee, Kyusong Lee, Gary Geunbae Lee, Seongdae Sagong, and Munsang Kim. 2011. On the effectiveness of robot-assisted language learning. ReCALL 23, 1 (2011), 25--58.
32. Iolanda Leite, Ginevra Castellano, Andre Pereira, Carlos Martinho, and Ana Paiva. 2012. Long-term interactions with empathic robots: Evaluating perceived support in children. In International Conference on Social Robotics. Springer, 298--307.
33. Iolanda Leite, Ginevra Castellano, André Pereira, Carlos Martinho, and Ana Paiva. 2014. Empathic robots for long-term interaction. Int. J. Soc. Robot. 6, 3 (2014), 329--341.
34. M. Levy. 1997. Computer-Assisted Language Learning: Context and Conceptualization. ERIC.
35. M. Levy and G. Stockwell. 2013. CALL Dimensions: Options and Issues in Computer-assisted Language Learning. Routledge.
36. Daniel Leyzberg, Samuel Spaulding, and Brian Scassellati. 2014. Personalizing robot tutors to individuals’ learning differences. In Proceedings of the 2014 ACM/IEEE International Conference on Human-Robot Interaction. ACM, 423--430.
37. D. Leyzberg, S. Spaulding, M. Toneva, and B. Scassellati. 2012. The physical presence of a robot tutor increases cognitive learning gains. In Proceedings of the 34th Annual Conference of the Cognitive Science Society. Cognitive Science Society, Austin, TX.
38. Robert V. Lindsey, Jeffery D. Shroyer, Harold Pashler, and Michael C. Mozer. 2014. Improving students’ long-term knowledge retention through personalized review. Psychol. Sci. 25, 3 (2014), 639--647.
39. K. Littleton and P. Light. 1999. Learning with Computers: Analysing Productive Interaction. Psychology Press.
40. D. C. Merrill, R. J. Reiser, M. Ranney, and J. G. Trafton. 1992. Effective tutoring techniques: A comparison of human tutors and intelligent tutoring systems. J. Learn. Sci. 2, 3 (1992), pp. 277--305.
41. Philip I. Pavlik and John R. Anderson. 2005. Practice and forgetting effects on vocabulary memory: An activation-based model of the spacing effect. Cogn. Sci. 29, 4 (2005), 559--586.
42. Philip I. Pavlik and John R. Anderson. 2008. Using a model to compute the optimal schedule of practice.J. Exp. Psychol. Appl. 14, 2 (2008), 101.
43. André Pereira, Carlos Martinho, Iolanda Leite, and Ana Paiva. 2008. iCat, the chess player: The influence of embodiment in the enjoyment of a game. In Proceedings of the 7th International Joint Conference on Autonomous Agents and Multiagent Systems, Volume 3. International Foundation for Autonomous Agents and Multiagent Systems, 1253--1256.
44. Aaron Powers, Sara Kiesler, Susan Fussell, and Cristen Torrey. 2007. Comparing a computer agent with a humanoid robot. In Proceedings of the 2nd ACM/IEEE International Conference on Human-Robot Interaction (HRI’07). IEEE, 145--152.
45. Aditi Ramachandran, Chien-Ming Huang, and Brian Scassellati. 2017. Give me a break!: Personalized timing strategies to promote learning in robot-child tutoring. In Proceedings of the 2017 ACM/IEEE International Conference on Human-Robot Interaction. ACM, 146--155.
46. Aditi Ramachandran, Alexandru Litoiu, and Brian Scassellati. 2016. Shaping productive help-seeking behavior during robot-child tutoring interactions. In Proceedings of the 11th ACM/IEEE International Conference on Human-Robot Interaction. IEEE Press, 247--254.
47. G. Saunders. 1988. Bilingual Children: From Birth to Teens. ERIC.
48. Thorsten Schodde, Kirsten Bergmann, and Stefan Kopp. 2017. Adaptive robot language tutoring based on Bayesian knowledge tracing and predictive decision-making. In Proceedings of the 2017 ACM/IEEE International Conference on Human-Robot Interaction. ACM, 128--136.
49. Sofia Serholt, Christina Anne Basedow, Wolmet Barendregt, and Mohammad Obaid. 2014. Comparing a humanoid tutor to a human tutor delivering an instructional task to children. In Proceedings of the 14th IEEE-RAS International Conference on Humanoid Robots (Humanoids’14). IEEE, 1134--1141.
50. Daniel Szafir and Bilge Mutlu. 2012. Pay attention!: Designing adaptive agents that monitor and improve user engagement. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI’12). ACM, New York, NY, 11--20.
51. Fumihide Tanaka, Aaron Cicourel, and Javier R. Movellan. 2007. Socialization between toddlers and robots at an early childhood education center. In Proceedings of the National Academy of Sciences 104, 46 (2007), 17954--17958.
52. Muhammad Tanveer. 2007. Investigation of the factors that cause language anxiety for ESL/EFL learners in learning speaking skills and the influence it casts on communication in the target language. Master's thesis. University of Glasgow, Scotland, UK.
53. Adriana Tapus, Cristian Tapus, and Maja Mataric. 2009. The role of physical embodiment of a therapist robot for individuals with cognitive impairments. In Proceedings of the 18th IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN’09). IEEE, 103--107.
54. Wayne P. Thomas and Virginia P. Collier. 2002. A National Study of School Effectiveness for Language Minority Students' Long-Term Academic Achievement. Technical Report. University of California at Santa Cruz, Center for Research on Education, Diversity, and Excellence.
55. U.S. Census Bureau. 2011. 2010 Census. U.S. Department of Commerce.
56. Joshua Wainer, David J. Feil-Seifer, Dylan A. Shell, and Maja J. Mataric. 2007. Embodiment and human-robot interaction: A task-based perspective. In Proceedings of the 16th IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN’07). IEEE, 872--877.
57. L. R. Welch. 2003. Hidden Markov models and the Baum-Welch algorithm. IEEE Inf. Theory Soc. Newslett. 53, 4 (2003), 10--13.
58. S. M. Williams, D. Nix, and P. Fairweather. 2013. Using speech recognition technology to enhance literacy instruction for emerging readers. In Proceedings of the 4th International Conference of the Learning Sciences. 115--120.

Index Terms

• Applied computing

  • Education

    • Interactive learning environments

• Human-centered computing

  • Human computer interaction (HCI)

    • HCI design and evaluation methods

      • User models

      • User studies