CogniLearn: A Deep Learning-based Interface for Cognitive Behavior Assessment

ABSTRACT

This paper proposes a novel system for assessing physical exercises specifically designed for cognitive behavior monitoring. The proposed system provides decision support to experts for helping with early childhood development. Our work is based on the well-established framework of Head-Toes-Knees-Shoulders (HTKS) that is known for its sufficient psychometric properties and its ability to assess cognitive dysfunctions. HTKS serves as a useful measure for behavioral self-regulation. Our system, CogniLearn, automates capturing and motion analysis of users performing the HTKS game and provides detailed evaluations using state-of-the-art computer vision and deep learning based techniques for activity recognition and evaluation. The proposed system is supported by an intuitive and specifically designed user interface that can help human experts to cross-validate and/or refine their diagnosis. To evaluate our system, we created a novel dataset, that we made open to the public to encourage further experimentation. The dataset consists of 15 subjects performing 4 different variations of the HTKS task and contains in total more than 60,000 RGB frames, of which 4,443 are fully annotated.

References

1. Andriluka, M., Pishchulin, L., Gehler, P. V., and Schiele, B. 2d human pose estimation: New benchmark and state of the art analysis. In 2014 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2014, Columbus, OH, USA, June 23-28, 2014 (2014), 3686--3693.
2. Asiry, O., Shen, H., and Calder, P. Extending attention span of adhd children through an eye tracker directed adaptive user interface. In Proceedings of the ASWEC 2015 24th Australasian Software Engineering Conference, ACM (2015), 149--152.
3. Billard, A., Robins, B., Nadel, J., and Dautenhahn, K. Building robota, a mini-humanoid robot for the rehabilitation of children with autism. Assistive Technology 19, 1 (2007), 37--49.
4. Brox, T., Bruhn, A., Papenberg, N., and Weickert, J. High accuracy optical flow estimation based on a theory for warping. In Computer Vision - ECCV 2004, 8th European Conference on Computer Vision, Prague, Czech Republic, May 11--14, 2004. Proceedings, Part IV (2004), 25--36.
5. Cormier, E. Attention deficit/hyperactivity disorder: a review and update. Journal of pediatric nursing 23, 5 (2008), 345--357.
6. Craven, M. P., and Groom, M. J. Computer games for user engagement in attention deficit hyperactivity disorder (adhd) monitoring and therapy. In 2015 International Conference on Interactive Technologies and Games, IEEE (2015), 34--40.
7. Da Gama, A., Chaves, T., Figueiredo, L., and Teichrieb, V. Poster: improving motor rehabilitation process through a natural interaction based system using kinect sensor. In 3D User Interfaces (3DUI), 2012 IEEE Symposium on, IEEE (2012), 145--146.
8. Dendy, C. A. Executive functionwhat is this anyway?. Retrieved September 18 (2008), 2008.
9. Donahue, J., Anne Hendricks, L., Guadarrama, S., Rohrbach, M., Venugopalan, S., Saenko, K., and Darrell, T. Long-term recurrent convolutional networks for visual recognition and description. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2015), 2625--2634.
10. Donahue, J., Hendricks, L. A., Guadarrama, S., Rohrbach, M., Venugopalan, S., Darrell, T., and Saenko, K. Long-term recurrent convolutional networks for visual recognition and description. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2015, Boston, MA, USA, June 7--12, 2015 (2015), 2625--2634.
11. Dunn, D. W., and Kronenberger, W. G. Attention-deficit/hyperactivity disorder in children and adolescents. Neurologic clinics 21, 4 (2003), 933--940.
12. Gattupalli, S., Ghaderi, A., and Athitsos, V. Evaluation of deep learning based pose estimation for sign language. In Proceedings of the 9th ACM International Conference on PErvasive Technologies Related to Assistive Environments, ACM (2016).
13. Howcroft, J., Klejman, S., Fehlings, D., Wright, V., Zabjek, K., Andrysek, J., and Biddiss, E. Active video game play in children with cerebral palsy: potential for physical activity promotion and rehabilitation therapies. Archives of physical medicine and rehabilitation 93, 8 (2012), 1448--1456.
14. Insafutdinov, E., Pishchulin, L., Andres, B., Andriluka, M., and Schiele, B. Deepercut: A deeper, stronger, and faster multi-person pose estimation model. CoRR abs/1605.03170 (2016).
15. Jadad, A. R., Booker, L., Gauld, M., Kakuma, R., Boyle, M., Cunningham, C. E., Kim, M., and Schachar, R. The treatment of attention-deficit hyperactivity disorder: an annotated bibliography and critical appraisal of published systematic reviews and metaanalyses. The Canadian Journal of Psychiatry 44, 10 (1999), 1025--1035.
16. Jia, Y., Shelhamer, E., Donahue, J., Karayev, S., Long, J., Girshick, R., Guadarrama, S., and Darrell, T. Caffe: Convolutional architecture for fast feature embedding. arXiv preprint arXiv:1408.5093 (2014).
17. Jia, Y., Shelhamer, E., Donahue, J., Karayev, S., Long, J., Girshick, R. B., Guadarrama, S., and Darrell, T. Caffe: Convolutional architecture for fast feature embedding. In Proceedings of the ACM International Conference on Multimedia, MM '14, Orlando, FL, USA, November 03 - 07, 2014 (2014), 675--678.
18. Lange, B., Flynn, S., Proffitt, R., Chang, C.-Y., et al. Development of an interactive game-based rehabilitation tool for dynamic balance training. Topics in stroke rehabilitation (2015).
19. Lange, K. W., Reichl, S., Lange, K. M., Tucha, L., and Tucha, O. The history of attention deficit hyperactivity disorder. ADHD Attention Deficit and Hyperactivity Disorders 2, 4 (2010), 241--255.
20. Leightley, D., Darby, J., Li, B., McPhee, J. S., and Yap, M. H. Human activity recognition for physical rehabilitation. In 2013 IEEE International Conference on Systems, Man, and Cybernetics, IEEE (2013), 261--266.
21. Mayes, R., Bagwell, C., and Erkulwater, J. Adhd and the rise in stimulant use among children. Harvard review of psychiatry 16, 3 (2008), 151--166.
22. McClelland, M. M., and Cameron, C. E. Self-regulation in early childhood: Improving conceptual clarity and developing ecologically valid measures. Child Development Perspectives 6, 2 (2012), 136--142.
23. McClelland, M. M., Cameron, C. E., Duncan, R., Bowles, R. P., Acock, A. C., Miao, A., and Pratt, M. E. Predictors of early growth in academic achievement: The head-toes-knees-shoulders task. Frontiers in psychology 5 (2014).
24. Najjar, L. J. Principles of educational multimedia user interface design. Human Factors: The journal of the human factors and ergonomics society 40, 2 (1998), 311--323.
25. Parrillo, V. N. Encyclopedia of social problems. Sage Publications, 2008.
26. Pirsiavash, H., Vondrick, C., and Torralba, A. Assessing the quality of actions. In European Conference on Computer Vision, Springer (2014), 556--571.
27. Rizzo, A. A., Buckwalter, J. G., Bowerly, T., Van Der Zaag, C., Humphrey, L., Neumann, U., Chua, C., Kyriakakis, C., Van Rooyen, A., and Sisemore, D. The virtual classroom: a virtual reality environment for the assessment and rehabilitation of attention deficits. CyberPsychology & Behavior 3, 3 (2000), 483--499.
28. Ross, R. G. Psychotic and manic-like symptoms during stimulant treatment of attention deficit hyperactivity disorder. American Journal of Psychiatry 163, 7 (2006), 1149--1152.
29. Roy, A. K., Soni, Y., and Dubey, S. Enhancing effectiveness of motor rehabilitation using kinect motion sensing technology. In Global Humanitarian Technology Conference: South Asia Satellite (GHTC-SAS), 2013 IEEE, IEEE (2013), 298--304.
30. Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M. S., Berg, A. C., and Li, F. Imagenet large scale visual recognition challenge. International Journal of Computer Vision 115, 3 (2015), 211--252.
31. Sim, M. G., Khong, E., Hulse, G., et al. When the child with adhd grows up. Australian family physician 33, 8 (2004), 615.
32. Sucar, L. E., Orihuela-Espina, F., Velazquez, R. L., Reinkensmeyer, D. J., Leder, R., and Hernández-Franco, J. Gesture therapy: An upper limb virtual reality-based motor rehabilitation platform. IEEE Transactions on Neural Systems and Rehabilitation Engineering 22, 3 (2014), 634--643.
33. Tsiakas, K., Papakostas, M., Chebaa, B., Ebert, D., Karkaletsis, V., and Makedon, F. An interactive learning and adaptation framework for adaptive robot assisted therapy.
34. Velloso, E., Bulling, A., Gellersen, H., Ugulino, W., and Fuks, H. Qualitative activity recognition of weight lifting exercises. In Proceedings of the 4th Augmented Human International Conference, ACM (2013), 116--123.
35. Wexler, B., and Bell, M. ACTIVATE Program. http://www.c8sciences.com/.
36. Xia, L., and Aggarwal, J. Spatio-temporal depth cuboid similarity feature for activity recognition using depth camera. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2013), 2834--2841.
37. Zhou, L., Liu, Z., Leung, H., and Shum, H. P. H. Posture reconstruction using kinect with a probabilistic model. In Proceedings of the 20th ACM Symposium on Virtual Reality Software and Technology, VRST '14 (2014), 117--125.

Index Terms

• Applied computing

  • Education

• Computing methodologies

  • Artificial intelligence

    • Computer vision

      • Computer vision problems

      • Computer vision tasks

        • Scene understanding

• Human-centered computing

  • Human computer interaction (HCI)