Investigating the Influence of Avatar Facial Characteristics on the Social Behaviors of Children with Autism

ABSTRACT

Autism spectrum disorder (ASD) is characterized by unusual social communication and interaction. These traits are often targets for intervention, particularly computer-based interventions (CBIs). We examined whether interactive behaviors in children with autism could be influenced by modifying the facial characteristics of computer avatars and how behavior toward avatars compared to that toward video. Participants spoke with a therapist over a modified videoconferencing system that permitted manipulation of her appearance (i.e., using cartoon or more realistic avatars versus video) and motion (i.e., exaggerating or damping facial movements). We measured the participants' speech, gaze, and gestures. In the first study, we found that the appearance complexity of the avatar did not significantly affect any social interaction behaviors. However, the results of the second study suggest that exaggerated facial motion can improve nonverbal social behaviors, such as gaze and gesture. These findings have implications for character design in CBIs for ASD.

References

1. American Psychiatric Association. 2000. DSM-IV-TR: Diagnostic and Statistical Manual of Mental Disorders, 4th Edition, Text Revision. American Psychiatric Association.
2. American Psychiatric Association. 2013. Diagnostic and Statistical Manual of Mental Disorders, 5th Edition. American Psychiatric Association.
3. Simon Baron-Cohen, Ofer Golan, and Emma Ashwin. 2009. Can emotion recognition be taught to children with autism spectrum conditions? Philosophical Transactions of the Royal Society B: Biological Sciences 364, 1535 (2009), 3567--3574.
4. Renae Beaumont and Kate Sofronoff. 2008. A multi-component social skills intervention for children with Asperger syndrome: The Junior Detective Training Program. Journal of Child Psychology and Psychiatry 49, 7 (2008), 743--753.
5. Armando Bertone, Laurent Mottron, Patricia Jelenic, and Jocelyn Faubert. 2003. Motion perception in autism: A complex issue. Journal of Cognitive Neuroscience 15, 2 (2003), 218--225.
6. Armando Bertone, Laurent Mottron, Patricia Jelenic, and Jocelyn Faubert. 2005. Enhanced and diminished visuo-spatial information processing in autism depends on stimulus complexity. Brain 128, 10 (2005), 2430--2441.
7. Steven M. Boker, Jeffrey F. Cohn, Barry-John Theobald, Iain Matthews, Timothy R. Brick, and Jeffrey R. Spies. 2009. Effects of damping head movement and facial expression in dyadic conversation using real-time facial expression tracking and synthesized avatars. Philosophical Transactions of the Royal Society B: Biological Sciences 364, 1535 (2009), 3485--3495.
8. Alexis Bosseler and Dominic W. Massaro. 2003. Development and evaluation of a computer-animated tutor for vocabulary and language learning in children with autism. Journal of Autism and Developmental Disorders 33, 6 (2003), 653--672.
9. Hennie Brugman and Albert Russel. 2004. Annotating Multimedia/Multi-modal resources with ELAN. In Proceedings of the 4th International Conference on Language Resources and Language Evaluation (LREC 2004), F. Ferreira R. Costa M. Lino, M. Xavier and R. Silva (Eds.). European Language Resources Association, Paris.
10. Ariane V. S. Buescher, Zuleyha Cidav, Martin Knapp, and David S. Mandell. 2014. Costs of autism spectrum disorders in the United Kingdom and the USA. JAMA Pediatrics 168, 8 (2014), 721--728.
11. Elizabeth J. Carter and Jennifer Hyde. 2015. Designing autism research for maximum impact. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (CHI '15). ACM, 2801--2804.
12. Elizabeth J. Carter, Diane L. Williams, Jessica K. Hodgins, and Jill F. Lehman. 2014. Are children with autism more responsive to animated characters? A study of interactions with humans and human-controlled avatars. Journal of Autism and Developmental Disorders 44, 10 (2014), 2475--2485.
13. Catherine Lord, Susan Risi, Linda Lambrecht, Edwin H. Cook, Bennett L. Leventhal, Pamela C. DiLavore, Andrew Pickles, and Michael Rutter. 2000. The Autism Diagnostic Observation Schedule-Generic: A standard measure of social and communication deficits associated with the spectrum of autism. Journal of Autism and Developmental Disorders 30, 3 (2000), 205--223.
14. Centers for Disease Control and Prevention. 2014. Prevalence of autism spectrum disorder among children aged 8 years -- Autism and Developmental Disabilities Monitoring Network, 11 Sites, USA, 2010. Morbidity and Mortality Weekly Report 63, SS02 (2014), 1--21.
15. Shen Hsing A. Chen and Vera Bernard-Opitz. 1993. Comparison of personal and computer-assisted instruction for children with autism. Mental Retardation 31, 6 (1993), 368--376.
16. Timothy F. Cootes, Gareth J. Edwards, and Christopher J. Taylor. 2001. Active appearance models. IEEE Transactions on Pattern Analysis and Machine Intelligence 23, 6 (2001), 681--685.
17. Timothy F. Cootes, Gavin V. Wheeler, Kevin N. Walker, and Christopher J. Taylor. 2002. View-based active appearance models. Image and Vision Computing 20, 9 (2002), 657--664.
18. Doreen Granpeesheh and Dennis R. Dixon and Jonathan Tarbox and Andrew M. Kaplan and Arthur E. Wilke. 2009. The effects of age and treatment intensity on behavioral intervention outcomes for children with autism spectrum disorders. Research in Autism Spectrum Disorders 3, 4 (2009), 1014--1022.
19. ELAN. 2004. Max Planck Institute for Psycholinguistics, The Language Archive, Nijmegen, Netherlands. (2004). http://tla.mpi.nl/tools/tla-tools/elan/.
20. Sue Fletcher-Watson. 2014. A targeted review of computer-assisted learning for people with autism spectrum disorder: Towards a consistent methodology. Review Journal of Autism and Developmental Disorders 1, 2 (2014), 87--100.
21. Uta Frith. Autism: Explaining the Enigma. Oxford, UK.
22. Francesca Happe. 1999. Autism: Cognitive deficit or cognitive style? Trends in Cognitive Sciences 3, 6 (1999), 216--222.
23. Francesca Happe and Uta Frith. 2006. The weak coherence account: Detail-focused cognitive style in autism spectrum disorders. Journal of Autism and Developmental Disorders 36, 1 (2006), 5--25.
24. Jennifer Hyde, Elizabeth J Carter, Sara Kiesler, and Jessica K Hodgins. 2013. Perceptual effects of damped and exaggerated facial motion in animated characters. In 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG) 2013. IEEE, 1--6.
25. Jennifer Hyde, Elizabeth J. Carter, Sara Kiesler, and Jessica K. Hodgins. 2014. Assessing naturalness and emotional intensity: A perceptual study of animated facial motion. In Proceedings of the Symposium on Applied Perception (SAP '14). ACM, 15--22.
26. Jennifer Hyde, Elizabeth J. Carter, Sara Kiesler, and Jessica K. Hodgins. 2015. Using an interactive avatar's facial expressiveness to increase persuasiveness and socialness. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (CHI '15). ACM, 2801--2804.
27. Jennifer Hyde, Sara Kiesler, Jessica K. Hodgins, and Elizabeth J. Carter. 2014. Conversing with children: Cartoon and video people elicit similar conversational behaviors. In Proceedings of the 32nd Annual Conference on Human Factors in Computing Systems (CHI '14). ACM, 1787--1796.
28. John Lasseter. 1987. Principles of traditional animation applied to 3D computer animation. In Proceedings of the 14th Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH '87), Vol. 21. ACM, 35--44.
29. Dominic W. Massaro and Alexis Bosseler. 2006. Read my lips: The importance of the face in a computer-animated tutor for vocabulary learning by children with autism. Autism 10, 5 (2006), 495--510.
30. Iain Matthews and Simon Baker. 2004. Active appearance models revisited. International Journal of Computer Vision 60, 2 (2004), 135--164.
31. Micah O. Mazurek, Paul T. Shattuck, Mary Wagner, and Benjamin P. Cooper. 2012. Prevalence and correlates of screen-based media use among youths with autism spectrum disorders. Journal of Autism and Developmental Disorders 42, 8 (2012), 1757--1767.
32. Beth A. Mineo, William Ziegler, Susan Gill, and Donna Salkin. 2009. Engagement with electronic screen media among students with autism spectrum disorders. Journal of Autism and Developmental Disorders 39, 1 (2009), 172--187.
33. Monique Moore and Sandra Calvert. 2000. Vocabulary acquisition for children with autism: Teacher or computer instruction. Journal of Autism and Developmental Disorders 30, 4 (2000), 359--362.
34. Laurent Mottron and Jacob A Burack. 2001. Enhanced perceptual functioning in the development of autism. In The Development of Autism: Perspectives from Theory and Research, Jacob A. Burack, Tony Charman, Nurit Yirmiya, and Philip R. Zelazo (Eds.). Lawrence Erlbaum Associates Publishers, Mahwah, NJ, USA.
35. Laurent Mottron, Michelle Dawson, Isabelle Soulieres, Benedicte Hubert, and Jake Burack. 2006. Enhanced perceptual functioning in autism: An update, and eight principles of autistic perception. Journal of Autism and Developmental Disorders 36, 1 (2006), 27--43.
36. Justin O'Brien, Janine Spencer, Christine Girges, Alan Johnston, and Harold Hill. 2014. Impaired perception of facial motion in autism spectrum disorder. PLOS One 9, 7 (2014), e102173.
37. Gael I. Orsmond and Hsin-Yu Kuo. 2011. The daily lives of adolescents with an autism spectrum disorder: Discretionary time use and activity partners. Autism 15, 5 (2011), 579--599.
38. Anthony J Plienis and Raymond G Romanczyk. 1985. Analyses of performance, behavior, and predictors for severely disturbed children: A comparison of adult vs. computer instruction. Analysis and Intervention in Developmental Disabilities 5, 4 (1985), 345--356.
39. Bertram O. Ploog, Alexa Scharf, DeShawn Nelson, and Patricia J. Brooks. 2013. Use of computer-assisted technologies (CAT) to enhance social, communicative, and language development in children with autism spectrum disorders. Journal of Autism and Developmental Disorders 43, 2 (2013), 301--322.
40. Sathiyaprakash Ramdoss, Russell Lang, Austin Mulloy, Jessica Franco, Mark O'Reilly, Robert Didden, and Giulio Lancioni. 2011. Use of computer-based interventions to teach communication skills to children with autism spectrum disorders: A systematic review. Journal of Behavioral Education 20, 1 (2011), 55--76.
41. Sathiyaprakash Ramdoss, Wendy Machalicek, Mandy Rispoli, Austin Mulloy, Russell Lang, and Mark O'Reilly. 2012. Computer-based interventions to improve social and emotional skills in individuals with autism spectrum disorders: A systematic review. Developmental Neurorehabilitation 15, 2 (2012), 119--135.
42. Keiran M. Rump, Joyce L. Giovannelli, Nancy J. Minshew, and Mark S. Strauss. 2009. The development of emotion recognition in individuals with autism. Child Development 80, 5 (2009), 1434--1447.
43. Wataru Sato, Shota Uono, and Motomi Toichi. 2013. Atypical recognition of dynamic changes in facial expressions in autism spectrum disorders. Research in Autism Spectrum Disorders 7, 7 (2013), 906--912.
44. Andrew Sears and Vicki L Hanson. 2012. Representing users in accessibility research. ACM Transactions on Accessible Computing (TACCESS) 4, 2 (2012), 7.
45. Howard C. Shane and Patti Ducoff Albert. 2008. Electronic screen media for persons with autism spectrum disorders: Results of a survey. Journal of Autism and Developmental Disorders 38, 8 (2008), 1499--1508.
46. Tristram Smith, Lawrence Scahill, Geraldine Dawson, Donald Guthrie, Catherine Lord, Samuel Odom, Sally Rogers, and Ann Wagner. 2007. Designing research studies on psychosocial interventions in autism. Journal of Autism and Developmental Disorders 37, 2 (2007), 354--366.
47. Jennifer Tam, Elizbeth J. Carter, Sara Kiesler, and Jessica K. Hodgins. 2012. Video increases the perception of naturalness during remote interactions with latency. In Proceedings of the 30th Annual ACM Conference on Human Factors in Computing Systems (CHI '12), Extended Abstracts. ACM, 2045--2050.
48. Carole Tardif, France Laine, Melissa Rodriguez, and Bruno Gepner. 2007. Slowing down presentation of facial movements and vocal sounds enhances facial expression recognition and induces facial-vocal imitation in children with autism. Journal of Autism and Developmental Disorders 37, 8 (2007), 1469--1484.
49. Andrea Tartaro and Justine Cassell. 2008. Playing with virtual peers: Bootstrapping contingent discourse in children with autism. In Proceedings of the 8th International Conference for the Learning Sciences-Volume 2. International Society of the Learning Sciences, 382--389.
50. Tomas Tjus, Mikael Heimann, and Keith E Nelson. 2001. Interaction patterns between children and their teachers when using a specific multimedia and communication strategy observations from children with autism and mixed intellectual disabilities. Autism 5, 2 (2001), 175--187.
51. Shota Uono, Wataru Sato, and Motomi Toichi. 2009. Dynamic fearful gaze does not enhance attention orienting in individuals with Aspergers disorder. Brain and Cognition 71, 3 (2009), 229--233.
52. Quan Wang, Feridun M. Celebi, Lilli Flink, Gabriella Greco, Carla Wall, Emily Prince, Sharlene Lansiquot, Katarzyna Chawarska, Elizabeth S. Kim, Laura Boccanfuso, Lauren DiNicola, and Frederick Shic. 2015. Interactive eye tracking for gaze strategy modification. In Proceedings of the 14th International Conference on Interaction Design and Children (IDC '15). ACM, 247--250.

Index Terms

• Information systems

  • Information systems applications

    • Multimedia information systems