Jittrapol Intarasirisawat
Skip Abstract Section
Skip Supplemental Material SectionAbstract
Early detection of cognitive decline is important for timely intervention and treatment strategies to prevent further deterioration or development of more severe cognitive impairment, as well as identify at risk individuals for research. In this paper, we explore the feasibility of using data collected from built-in sensors of mobile phone and gameplay performance in mobile-game-based cognitive assessments. Twenty-two healthy participants took part in the two-session experiment where they were asked to take a series of standard cognitive assessments followed by playing three popular mobile games in which user-game interaction data were passively collected. The results from bivariate analysis reveal correlations between our proposed features and scores obtained from paper-based cognitive assessments. Our results show that touch gestural interaction and device motion patterns can be used as supplementary features on mobile game-based cognitive measurement. This study provides initial evidence that game related metrics on existing off-the-shelf games have potential to be used as proxies for conventional cognitive measures, specifically for visuospatial function, visual search capability, mental flexibility, memory and attention.
Supplemental Material
Available for Download
zip
intarasirisawat.zip (197.8 KB)
Supplemental movie, appendix, image and software files for, Exploring the Touch and Motion Features in Game-Based Cognitive Assessments
- Anna Anzulewicz, Krzysztof Sobota, and Jonathan T. Delafield-Butt. 2016. Toward the Autism Motor Signature: Gesture patterns during smart tablet gameplay identify children with autism. Scientific reports 6 (2016), 31107.Google Scholar
- Russell A. Barkley. 1997. Behavioral inhibition, sustained attention, and executive functions: constructing a unifying theory of ADHD. Psychological bulletin 121, 1 (1997), 65.Google Scholar
- Jennifer H. Barnett, Andrew D. Blackwell, Barbara J. Sahakian, and Trevor W. Robbins. 2015. The paired associates learning (PAL) test: 30 years of CANTAB translational neuroscience from laboratory to bedside in dementia research. Springer, 449--474.Google Scholar
- Chris Bevan and DanaÃń Stanton Fraser. 2016. Different strokes for different folks? Revealing the physical characteristics of smartphone users from their swipe gestures. International Journal of Human-Computer Studies 88 (apr 2016), 51--61. Google ScholarDigital Library
- Gary Burnett, Elizabeth Crundall, David Large, Glyn Lawson, and Lee Skrypchuk. 2013. A study of unidirectional swipe gestures on in-vehicle touch screens. In Proceedings of the 5th International Conference on Automotive User Interfaces and Interactive Vehicular Applications. ACM, 22--29. Google ScholarDigital Library
- Niamh Caprani, Noel E., and Cathal Gurri. 2012. Touch Screens for the Older User. In Assistive Technologies. InTech.Google Scholar
- A. Collie, D. Darby, and P. Maruff. 2001. Computerised cognitive assessment of athletes with sports related head injury. British journal of sports medicine 35, 5 (Oct 2001), 297--302. LR: 20140613; JID: 0432520; RF: 29; 2001/10/02 10:00 {pubmed}; 2002/01/05 10:01 {medline}; 2001/10/02 10:00 {entrez}; ppublish.Google Scholar
- Ian J. Deary and Geoff Der. 2005. Reaction Time, Age, and Cognitive Ability: Longitudinal Findings from Age 16 to 63 Years in Representative Population Samples. Aging, Neuropsychology, and Cognition 12, 2 (jun 2005), 187--215.Google ScholarCross Ref
- Bruno Dubois, Alessandro Padovani, Philip Scheltens, Andrea Rossi, and Grazia Dell'Agnello. 2015. Timely Diagnosis for Alzheimer's Disease: A Literature Review on Benefits and Challenges. Journal of Alzheimer's Disease 49, 3 (Oct 2015), 617--631.Google ScholarCross Ref
- Eletha Flores, Gabriel Tobon, Ettore Cavallaro, Francesca I. Cavallaro, Joel C. Perry, and Thierry Keller. 2008. Improving patient motivation in game development for motor deficit rehabilitation. In Proceedings of the 2008 International Conference on Advances in Computer Entertainment Technology. ACM, 381--384. Google ScholarDigital Library
- Luciano Gamberini, Francesco Martino, Bruno Seraglia, Anna Spagnolli, Malena Fabregat, Francisco Ibanez, Mariano Alcaniz, and Javier Montesa Andrés. 2009. Eldergames project: An innovative mixed reality table-top solution to preserve cognitive functions in elderly people. In Human System Interactions, 2009. HSI'09. 2nd Conference on. IEEE, 164--169. Google ScholarDigital Library
- Yuan Gao, Nadia Bianchi-Berthouze, and Hongying Meng. 2012. What does touch tell us about emotions in touchscreen-based gameplay? ACM Transactions on Computer-Human Interaction (TOCHI) 19, 4 (2012), 31. Google ScholarDigital Library
- C. Shawn Green and Daphne Bavelier. 2003. Action video game modifies visual selective attention. Nature 423, 6939 (2003), 534.Google Scholar
- Daniel F. Hermens, Jim Lagopoulos, Juliette Tobias-Webb, Tamara De Regt, Glenys Dore, Lisa Juckes, Noeline Latt, and Ian B. Hickie. 2013. Pathways to alcohol-induced brain impairment in young people: a review. Cortex 49, 1 (2013), 3--17.Google ScholarCross Ref
- John R. Hodges. 2007. Cognitive assessment for clinicians. Oxford University Press.Google Scholar
- Emily A. Holmes, Ella L. James, Emma J. Kilford, and Catherine Deeprose. 2010. Key Steps in Developing a Cognitive Vaccine against Traumatic Flashbacks: Visuospatial Tetris versus Verbal Pub Quiz. PLoS ONE 5, 11 (nov 2010), e13706.Google ScholarCross Ref
- Susan Homack and Cynthia A. Riccio. 2004. A meta-analysis of the sensitivity and specificity of the Stroop Color and Word Test with children. Archives of clinical Neuropsychology 19, 6 (2004), 725--743.Google Scholar
- Sameer Jauhar, E. Jane Marshall, and Iain D. Smith. 2014. Alcohol and cognitive impairment. Advances in psychiatric treatment 20, 5 (2014), 304--313.Google Scholar
- Caroline Ketcham and George Stelmach. 2004. Movement control in the older adult. Technology for Adaptive Aging (01 2004), 64--92.Google Scholar
- Alexandra M Kueider, Jeanine M Parisi, Alden L Gross, and George W Rebok. 2012. Computerized cognitive training with older adults: a systematic review. PloS one 7, 7 (2012), e40588.Google ScholarCross Ref
- Robert Langner, Michael B Steinborn, Anjan Chatterjee, Walter Sturm, and Klaus Willmes. 2010. Mental fatigue and temporal preparation in simple reaction-time performance. Acta psychologica 133, 1 (Jan 2010), 64--72.Google Scholar
- Alex Lau-Zhu, Emily A. Holmes, Sally Butterfield, and Joni Holmes. 2017. Selective Association Between Tetris Game Play and Visuospatial Working Memory: A Preliminary Investigation. Applied Cognitive Psychology 31, 4 (jul 2017), 438--445.Google ScholarCross Ref
- Megan E. Lenehan, Mathew J. Summers, Nichole L. Saunders, Jeffery J. Summers, and James C. Vickers. 2016. Does the Cambridge Automated Neuropsychological Test Battery (CANTAB) distinguish between cognitive domains in healthy older adults? Assessment 23, 2 (2016), 163--172.Google ScholarCross Ref
- Qian Liu, Xinyi Zhu, Albert Ziegler, and Jiannong Shi. 2015. The effects of inhibitory control training for preschoolers on reasoning ability and neural activity. Scientific Reports 5, 1 (sep 2015).Google Scholar
- Monica Luciana. 2003. Practitioner review: computerized assessment of neuropsychological function in children: clinical and research applications of the Cambridge Neuropsychological Testing Automated Battery (CANTAB). Journal of Child Psychology and Psychiatry 44, 5 (2003), 649--663.Google ScholarCross Ref
- Jim Lumsden, Elizabeth A Edwards, Natalia S Lawrence, David Coyle, and Marcus R Munafò. 2016. Gamification of Cognitive Assessment and Cognitive Training: A Systematic Review of Applications and Efficacy. JMIR Serious Games 4, 2 (jul 2016), e11.Google ScholarCross Ref
- V. Manera, P. D. Petit, A. Derreumaux, I. Orvieto, M. Romagnoli, G. Lyttle, R. David, and P. H. Robert. 2015. 'Kitchen and cooking,' a serious game for mild cognitive impairment and Alzheimer's disease: a pilot study. Frontiers in aging neuroscience 7 (Mar 17 2015), 24.Google Scholar
- P Mavrogiorgou, R Mergl, P Tigges, J El Husseini, A SchrÃűter, G Juckel, M Zaudig, and U Hegerl. 2001. Kinematic analysis of handwriting movements in patients with obsessive-compulsive disorder. Journal of neurology, neurosurgery, and psychiatry 70, 5 (May 2001), 605--12.Google ScholarCross Ref
- Ziad S. Nasreddine, Natalie A. Phillips, ValÃľrie BÃľdirian, Simon Charbonneau, Victor Whitehead, Isabelle Collin, Jeffrey L. Cummings, and Howard Chertkow. 2005. The Montreal Cognitive Assessment, MoCA: a brief screening tool for mild cognitive impairment. Journal of the American Geriatrics Society 53, 4 (2005), 695--699.Google ScholarCross Ref
- K. Onoda, T. Hamano, Y. Nabika, A. Aoyama, H. Takayoshi, T. Nakagawa, M. Ishihara, S. Mitaki, T. Yamaguchi, H. Oguro, K. Shiwaku, and S. Yamaguchi. 2013. Validation of a new mass screening tool for cognitive impairment: Cognitive Assessment for Dementia, iPad version. Clinical interventions in aging 8 (2013), 353--360.Google Scholar
- R. C. Petersen, J.C. Stevens, M. Ganguli, E. G. Tangalos, J.L. Cummings, and S. T. DeKosky. 2001. Practice parameter: Early detection of dementia: Mild cognitive impairment (an evidence-based review): Report of the Quality Standards Subcommittee of the American Academy of Neurology. Neurology 56, 9 (may 2001), 1133--1142.Google ScholarCross Ref
- Trevor W. Robbins, M.James, Adrian M. Owen, Barbara J. Sahakian, L. McInnes, and P. Rabbitt. 1994. Cambridge Neuropsychological Test Automated Battery (CANTAB): a factor analytic study of a large sample of normal elderly volunteers. Dementia and geriatric cognitive disorders 5, 5 (1994), 266--281.Google Scholar
- Philippe Robert, Alexandra König, Hélene Amieva, Sandrine Andrieu, François Bremond, Roger Bullock, Mathieu Ceccaldi, Bruno Dubois, Serge Gauthier, and Paul-Ariel Kenigsberg. 2014. Recommendations for the use of Serious Games in people with Alzheimer's Disease, related disorders and frailty. Frontiers in aging neuroscience 6 (2014), 54.Google Scholar
- Ned Sacktor, Michael P. McDermott, Karen Marder, Giovanni Schifitto, Ola A. Selnes, Justin C. McArthur, Yaakov Stern, Steve Albert, Donna Palumbo, and Karl Kieburtz. 2002. HIV-associated cognitive impairment before and after the advent of combination therapy. Journal of neurovirology 8, 2 (2002), 136--142.Google ScholarCross Ref
- Akane Sano and Rosalind W. Picard. 2013. Stress recognition using wearable sensors and mobile phones. In Affective Computing and Intelligent Interaction (ACII), 2013 Humaine Association Conference on. IEEE, 671--676. Google ScholarDigital Library
- A. Schröter, R. Mergl, K. Bürger, H. Hampel, H.-J. Möller, and U. Hegerl. 2003. Kinematic Analysis of Handwriting Movements in Patients with Alzheimer's Disease, Mild Cognitive Impairment, Depression and Healthy Subjects. Dementia and Geriatric Cognitive Disorders 15, 3 (2003), 132--142.Google ScholarCross Ref
- A. Schweiger, GM Doniger, T. Dwolatzky, D. Jaffe, and ES Simon. 2003. Reliability of a novel computerized neuropsychological battery for mild cognitive impairment. Acta Neuropsychologica 1, 4 (2003), 407--413.Google Scholar
- Yoshihito Shigihara, Masaaki Tanaka, Akira Ishii, Seiki Tajima, Etsuko Kanai, Masami Funakura, and Yasuyoshi Watanabe. 2013. Two different types of mental fatigue produce different styles of task performance. Neurology, Psychiatry and Brain Research 19, 1 (feb 2013), 5--11.Google ScholarCross Ref
- F. C. Soares, T. C. de Oliveira, L. D. de Macedo, A. M. Tomas, D. L. Picanco-Diniz, J. Bento-Torres, N. V. Bento-Torres, and C. W. Picanco-Diniz. 2014. CANTAB object recognition and language tests to detect aging cognitive decline: an exploratory comparative study. Clinical interventions in aging 10 (Dec 19 2014), 37--48.Google Scholar
- Kevin M. Spencer and Marie T. Banich. 2005. Hemispheric biases and the control of visuospatial attention: an ERP study. BMC Neuroscience 6, 1 (24 Aug 2005), 51.Google Scholar
- Statista. {n.d.}. U.S. average age of video gamers 2018 | Statistic. url: https://www.statista.com/statistics/189582/age-of-us-video-game-players-since-2010/. https://www.statista.com/statistics/189582/age-of-us-video-game-players-since-2010/ Accessed: 22-March-2019.Google Scholar
- Alvaro D. Taveira and Sang D. Choi. 2009. Review Study of Computer Input Devices and Older Users. International Journal of Human-Computer Interaction 25, 5 (jun 2009), 455--474.Google ScholarCross Ref
- H. Gerry Taylor and Kenneth M. Heilman. 1980. Left-Hemisphere Motor Dominance in Righthandersi. Cortex 16, 4 (Dec. 1980), 587--603.Google Scholar
- Peter Tigges, Roland Mergl, Thomas Frodl, Eva M. Meisenzahl, Jürgen Gallinat, Andreas Schröter, Michael Riedel, Norbert Müller, Hans-Jürgen Möller, and Ulrich Hegerl. 2000. Digitized analysis of abnormal hand-motor performance in schizophrenic patients. Schizophrenia Research 45, 1-2 (sep 2000), 133--143.Google ScholarCross Ref
- Tom N. Tombaugh. 2004. Trail Making Test A and B: normative data stratified by age and education. Archives of clinical neuropsychology 19, 2 (2004), 203--214.Google Scholar
- Tom N. Tombaugh and Nancy J. McIntyre. 1992. The miniâĂŘmental state examination: a comprehensive review. Journal of the American Geriatrics Society 40, 9 (1992), 922--935.Google ScholarCross Ref
- T. Tong, M. Chignell, M. C. Tierney, and J. Lee. 2016. A Serious Game for Clinical Assessment of Cognitive Status: Validation Study. JMIR serious games 4, 1 (May 27 2016), e7.Google Scholar
- Unity3D. {n.d.}. Match 3 Jelly Garden Complete Project Asset Store. url: https://assetstore.unity.com/packages/templates/systems/match-3-jelly-garden-complete-project-43260. https://assetstore.unity.com/packages/templates/systems/match-3-jelly-garden-complete-project-43260 Accessed: 11-July-2018.Google Scholar
- Unity3D. {n.d.}. Object/Ball Slicing Example Game Unity3D Asset Store. url: https://assetstore.unity.com/packages/templates/object-ball-slicing-example-game-76330. https://assetstore.unity.com/packages/templates/object-ball-slicing-example-game-76330 Accessed: 11-July-2018.Google Scholar
- Unity3D. {n.d.}. Tetris Template Mobile Ready Unity3D Asset Store. url: https://assetstore.unity.com/packages/templates/tetris-template-mobile-ready-72717. https://assetstore.unity.com/packages/templates/tetris-template-mobile-ready-72717 Accessed: 11-July-2018.Google Scholar
- Geoffrey F. Woodman and Steven J. Luck. 2004. Visual search is slowed when visuospatial working memory is occupied. Psychonomic bulletin & review 11, 2 (2004), 269--274.Google Scholar
- Anthony S. Zigmond and R. Philip Snaith. 1983. The hospital anxiety and depression scale. Acta Psychiatrica Scandinavica 67, 6 (1983), 361--370.Google ScholarCross Ref
- Gokhan Zorluoglu, Mustafa E. Kamasak, Leyla Tavacioglu, and Pinar O. Ozanar. 2015. A mobile application for cognitive screening of dementia. Computer methods and programs in biomedicine 118, 2 (2015), 252--262. Google ScholarDigital Library
- Stelios Zygouris, Dimitrios Giakoumis, Konstantinos Votis, Stefanos Doumpoulakis, Konstantinos Ntovas, Sofia Segkouli, Charalampos Karagiannidis, Dimitrios Tzovaras, and Magda Tsolaki. 2015. Can a virtual reality cognitive training application fulfill a dual role? Using the virtual supermarket cognitive training application as a screening tool for mild cognitive impairment. Journal of Alzheimer's Disease 44, 4 (2015), 1333--1347.Google ScholarCross Ref
- Stelios Zygouris and Magda Tsolaki. 2015. Computerized cognitive testing for older adults: a review. American Journal of Alzheimer's Disease & Other Dementias® 30, 1 (2015), 13--28.Google Scholar
Index Terms
- Exploring the Touch and Motion Features in Game-Based Cognitive Assessments
Recommendations
Monitoring Cognitive Performance with a Serious Game: A Longitudinal Case Study on Online Cognitive Assessment Using Serious Games
CHI EA '21: Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing SystemsThis case study involves: the design and evaluation of serious games; the use of longitudinal research and remote testing in an international setting. Current methods for cognitive assessment tend to be inconvenient, costly and infrequently performed. ...
Developing a serious game for cognitive assessment: choosing settings and measuring performance
Chinese CHI '14: Proceedings of the Second International Symposium of Chinese CHIGamification and serious games are becoming increasingly important for training, wellness, and other applications. How can games be developed for non-traditional gaming populations such as the elderly, and how can gaming be applied in non-traditional ...
Can Card Games Be Used to Assess Mild Cognitive Impairment?: A Study of Klondike Solitaire and Cognitive Functions
CHI PLAY '17 Extended Abstracts: Extended Abstracts Publication of the Annual Symposium on Computer-Human Interaction in PlayThis paper investigates whether Klondike Solitaire can be used as an assessment tool for mild cognitive impairment (MCI). It does so by providing a structured analysis and mapping player actions to cognitive functions. For Klondike Solitaire, 22 player ...
Comments