Teaching analytics: towards automatic extraction of orchestration graphs using wearable sensors

ABSTRACT

'Teaching analytics' is the application of learning analytics techniques to understand teaching and learning processes, and eventually enable supportive interventions. However, in the case of (often, half-improvised) teaching in face-to-face classrooms, such interventions would require first an understanding of what the teacher actually did, as the starting point for teacher reflection and inquiry. Currently, such teacher enactment characterization requires costly manual coding by researchers. This paper presents a case study exploring the potential of machine learning techniques to automatically extract teaching actions during classroom enactment, from five data sources collected using wearable sensors (eye-tracking, EEG, accelerometer, audio and video). Our results highlight the feasibility of this approach, with high levels of accuracy in determining the social plane of interaction (90%, κ=0.8). The reliable detection of concrete teaching activity (e.g., explanation vs. questioning) accurately still remains challenging (67%, κ=0.56), a fact that will prompt further research on multimodal features and models for teaching activity extraction, as well as the collection of a larger multimodal dataset to improve the accuracy and generalizability of these methods.

References

1. M. Bienkowski, M. Feng, and B. Means. Enhancing teaching and learning through educational data mining and learning analytics: An issue brief. Technical report, U.S. Department of Education, Office of Educational Technology, 2012.
2. N. Blanchard, S. D'Mello, M. Nystrand, and A. M. Olney. Automatic classification of question & answer discourse segments from teacher's speech in classrooms. In Proceedings of the 8th International Conference on Educational Data Mining (EDM 2015), International Educational Data Mining Society, 2015.
3. P. Blikstein. Multimodal learning analytics. In Proceedings of the third international conference on learning analytics and knowledge, pages 102--106. ACM, 2013.
4. L. Breiman. Random forests. Machine learning, 45(1):5--32, 2001.
5. D. Clow. The learning analytics cycle: closing the loop effectively. In Proceedings of the 2nd international conference on learning analytics and knowledge, pages 134--138. ACM, 2012.
6. J. Cohen. Statistical power analysis for the behavioral sciences (rev. Lawrence Erlbaum Associates, Inc., 1977.
7. P. Dillenbourg. Orchestration Graphs: Modeling Scalable Education. EPFL Press, Lausanne, Switzerland, 1 edition, 2015.
8. P. Dillenbourg, S. Järvelä, and F. Fischer. The evolution of research on computer-supported collaborative learning. In Technology-enhanced learning, pages 3--19. Springer, 2009.
9. P. Dillenbourg and P. Tchounikine. Flexibility in macro-scripts for computer-supported collaborative learning. Journal of computer assisted learning, 23(1):1--13, 2007.
10. S. K. D'Mello, A. M. Olney, N. Blanchard, B. Samei, X. Sun, B. Ward, and S. Kelly. Multimodal capture of teacher-student interactions for automated dialogic analysis in live classrooms. In Proceedings of the 2015 ACM on International Conference on Multimodal Interaction, pages 557--566. ACM, 2015.
11. F. Domínguez, K. Chiluiza, V. Echeverria, and X. Ochoa. Multimodal selfies: Designing a multimodal recording device for students in traditional classrooms. In Proceedings of the 2015 ACM on International Conference on Multimodal Interaction, pages 567--574. ACM, 2015.
12. E. Durall and T. Leinonen. Feeler: supporting awareness and reflection about learning through EEG data. In Proceedings of the 5th Workshop on Awareness and Reflection in Technology Enhanced Learning In conjunction with the 10th European Conference on Technology Enhanced Learning, pages 67--73, 2015.
13. A. L. Dyckhoff, D. Zielke, M. Bültmann, M. A. Chatti, and U. Schroeder. Design and implementation of a learning analytics toolkit for teachers. Journal of Educational Technology & Society, 15(3):58--76, 2012.
14. T. Elias. Learning analytics: The definitions, the processes, and the potential. 2011.
15. V. Emin-Martinez, C. Hansen, M. J. Rodríguez-Triana, B. Wasson, Y. Mor, R. Ferguson, and J.-P. Pernin. Towards teacher-led design inquiry of learning. E-learning papers. Special issue on Learning Analytics and Assessment, 36:3--14, 2014.
16. R. Ferguson. Learning analytics: drivers, developments and challenges. International Journal of Technology Enhanced Learning, 4(5-6):304--317, 2012.
17. D. H. Freedman. Why scientific studies are so often wrong: The streetlight effect. Discover Magazine, 26, 2010.
18. J. H. Friedman. Stochastic gradient boosting. Computational Statistics & Data Analysis, 38(4):367--378, 2002.
19. G. Gauthier. Using teaching analytics to inform assessment practices in technology mediated problem solving tasks. In Vatrapu et al. {48}.
20. T. Giannakopoulos and A. Pikrakis. Introduction to Audio Analysis: A MATLAB® Approach. Academic Press, 2014.
21. Y. Hayashi and R. Mizoguchi. Articulation of scenario construction of lessons based on ontological engineering. In Vatrapu et al. {47}.
22. D. Hernández-Leo, J. I. Asensio-Pérez, M. Derntl, L. P. Prieto, and J. Chacón. Ilde: Community environment for conceptualizing, authoring and deploying learning activities. In C. Rensing, S. de Freitas, T. Ley, and P. Muñoz-Merino, editors, Open Learning and Teaching in Educational Communities, volume 8719 of Lecture Notes in Computer Science, pages 490--493. Springer International Publishing, 2014.
23. J. Jovanović, D. Gašević, C. Brooks, V. Devedžić, and M. Hatala. Loco-analyst: A tool for raising teachers' awareness in online learning environments. In Creating New Learning Experiences on a Global Scale, pages 112--126. Springer, 2007.
24. B.-K. Kim, H. Lee, J. Roh, and S.-Y. Lee. Hierarchical committee of deep cnns with exponentially-weighted decision fusion for static facial expression recognition. In Proceedings of the 2015 ACM on International Conference on Multimodal Interaction, ICMI '15, pages 427--434, New York, NY, USA, 2015. ACM.
25. D. Laurillard. Teaching as a design science: Building pedagogical patterns for learning and technology. Routledge, 2012.
26. Y. Mor, R. Ferguson, and B. Wasson. Editorial: Learning design, teacher inquiry into student learning and learning analytics: A call for action. British Journal of Educational Technology, 46(2):221--229, 2015.
27. L.-P. Morency, S. Oviatt, S. Scherer, N. Weibel, and M. Worsley. Icmi 2013 grand challenge workshop on multimodal learning analytics. In Proceedings of the 15th ACM on International conference on multimodal interaction, pages 373--378. ACM, 2013.
28. X. Ochoa, K. Chiluiza, G. Méndez, G. Luzardo, B. Guamán, and J. Castells. Expertise estimation based on simple multimodal features. In Proceedings of the 15th ACM on International conference on multimodal interaction, pages 583--590. ACM, 2013.
29. X. Ochoa, M. Worsley, K. Chiluiza, and S. Luz. Mla'14: Third multimodal learning analytics workshop and grand challenges. In Proceedings of the 16th International Conference on Multimodal Interaction, pages 531--532. ACM, 2014.
30. M. Oliver and K. Trigwell. Can 'blended learning'be redeemed? E-learning and Digital Media, 2(1):17--26, 2005.
31. D. Persico and F. Pozzi. Informing learning design with learning analytics to improve teacher inquiry. British Journal of Educational Technology, 46(2):230--248, 2015.
32. S. J. Preece, J. Y. Goulermas, L. P. Kenney, D. Howard, K. Meijer, and R. Crompton. Activity identification using body-mounted sensors---a review of classification techniques. Physiological measurement, 30(4):R1, 2009.
33. L. P. Prieto, K. Sharma, and P. Dillenbourg. Studying teacher orchestration load in technology-enhanced classrooms. In Design for Teaching and Learning in a Networked World, pages 268--281. Springer, 2015.
34. L. P. Prieto, S. Villagrá-Sobrino, I. M. Jorrín-Abellán, A. Martínez-Monés, and Y. Dimitriadis. Recurrent routines: Analyzing and supporting orchestration in technology-enhanced primary classrooms. Computers & Education, 57(1):1214--1227, 2011.
35. M. Raca and P. Dillenbourg. System for assessing classroom attention. In Proceedings of the Third International Conference on Learning Analytics and Knowledge, pages 265--269. ACM, 2013.
36. S. Rebholz, P. Libbrecht, and W. Müller. Learning analytics as an investigation tool for teaching practitioners. In Vatrapu et al. {47}.
37. V. Rivera-Pelayo, E. Lacić, V. Zacharias, and R. Studer. LIM App: Reflecting on Audience Feedback for Improving Presentation Skills. In D. Hernández-Leo, T. Ley, R. Klamma, and A. Harrer, editors, Scaling up Learning for Sustained Impact, number 8095 in Lecture Notes in Computer Science, pages 514--519. Springer Berlin Heidelberg, Sept. 2013.
38. M. J. Rodríguez-Triana, A. Martínez-Monés, J. I. Asensio-Pérez, and Y. Dimitriadis. Script-aware monitoring model: Using teachers' pedagogical intentions to guide learning analytics. In Vatrapu et al. {47}.
39. R. K. Sawyer. Creative teaching: Collaborative discussion as disciplined improvisation. Educational researcher, 33(2):12--20, 2004.
40. G. Siemens. Sensemaking: Beyond analytics as a technical activity. Presentation at the EDUCAUSE ELI 2012 Online Spring Focus Session.
41. G. Siemens and P. Long. Penetrating the fog: Analytics in learning and education. EDUCAUSE review, 46(5):30, 2011.
42. K. Thompson, R. Martinez-Maldonado, D. Wardak, P. Goodyear, and L. Carvalho. Analysing F2F Collaborative Design and Learning: Experiences in a Design Studio A Place for Learning. In L. P. Prieto, Y. Dimitriadis, A. Harrer, M. Milrad, and J. D. Slotta, editors, Proceedings of the Orchestrated Collaborative Classroom Workshop 2015 co-located with 11th International Conference on Computer Supported Collaborative Learning, pages 25--29, Gothenburg, Sweden, 2015.
43. M. van Harmelen and D. Workman. Analytics for learning and teaching. JISC CETIS Analytics Series, 1(3):1--41, 2012.
44. R. Vatrapu, C. Teplovs, N. Fujita, and S. Bull. Towards visual analytics for teachers' dynamic diagnostic pedagogical decision-making. In Proceedings of the 1st International Conference on Learning Analytics and Knowledge, pages 93--98. ACM, 2011.
45. R. K. Vatrapu. Towards semiology of Teaching Analytics. In Workshop Towards Theory and Practice of Teaching Analytics, at the European Conference on Technology Enhanced Learning, TAPTA'12, Saarbrücken, Germany, 2012.
46. R. K. Vatrapu, K. Kocherla, and K. Pantazos. iklassroom: Real-time, real-place teaching analytics. In Vatrapu et al. {48}.
47. R. K. Vatrapu, P. Reimann, W. Halb, and S. Bull, editors. Proceedings of the Workshop Towards Theory and Practice of Teaching Analytics (TaPTA 2012), number 894 in CEUR Workshop Proceedings, Aachen, 2012.
48. R. K. Vatrapu, P. Reimann, W. Halb, and S. Bull, editors. Proceedings of the 2nd International Workshop on Teaching Analytics (IWTA 2013), number 985 in CEUR Workshop Proceedings, Aachen, 2013.
49. M. Worsley. Multimodal learning analytics: enabling the future of learning through multimodal data analysis and interfaces. In Proceedings of the 14th ACM international conference on Multimodal interaction, pages 353--356. ACM, 2012.
50. M. Worsley and P. Blikstein. Leveraging multimodal learning analytics to differentiate student learning strategies. In Proceedings of the Fifth International Conference on Learning Analytics And Knowledge, pages 360--367. ACM, 2015.
51. B. Xu and M. Recker. Teaching Analytics: A clustering and triangulation study of digital library user data. Educational Technology & Society, 15(3):103--115, 2012.

Index Terms

• Applied computing

  • Education

    • Collaborative learning

  • Enterprise computing