Tomislav Pejsa
Sean Andrist
Bilge Mutlu
Skip Abstract Section
Abstract
To facilitate natural interactions between humans and embodied conversational agents (ECAs), we need to endow the latter with the same nonverbal cues that humans use to communicate. Gaze cues in particular are integral in mechanisms for communication and management of attention in social interactions, which can trigger important social and cognitive processes, such as establishment of affiliation between people or learning new information. The fundamental building blocks of gaze behaviors are gaze shifts: coordinated movements of the eyes, head, and body toward objects and information in the environment. In this article, we present a novel computational model for gaze shift synthesis for ECAs that supports parametric control over coordinated eye, head, and upper body movements. We employed the model in three studies with human participants. In the first study, we validated the model by showing that participants are able to interpret the agent’s gaze direction accurately. In the second and third studies, we showed that by adjusting the participation of the head and upper body in gaze shifts, we can control the strength of the attention signals conveyed, thereby strengthening or weakening their social and cognitive effects. The second study shows that manipulation of eye--head coordination in gaze enables an agent to convey more information or establish stronger affiliation with participants in a teaching task, while the third study demonstrates how manipulation of upper body coordination enables the agent to communicate increased interest in objects in the environment.
- Y. Alkan, B. B. Biswal, and T. L. Alvarez. 2011. Differentiation between vergence and saccadic functional activity within the human frontal eye fields and midbrain revealed through fMRI. PLoS One 6, 11 (Nov. 2011).Google Scholar
Cross Ref
- C. André-Deshays, A. Berthoz, and M. Revel. 1988. Eye-head coupling in humans. Experimental Brain Research 69, 2 (1988), 399--406.Google ScholarCross Ref
- S. Andrist, B. Mutlu, and M. Gleicher. 2013. Conversational gaze aversion for virtual agents. In Proceedings of 13th International Conference on Intelligent Virtual Agents (IVA’13), R. Aylett, B. Krenn, C. Pelachaud, and H. Shimodaira (Eds.). Vol. 8108. Springer, 249--262.Google Scholar
- S. Andrist, T. Pejsa, B. Mutlu, and M. Gleicher. 2012a. Designing effective gaze mechanisms for virtual agents. In Proceedings of ACM/CHI 2012 Conference on Human Factors in Computing Systems. ACM, 705--714. Google ScholarDigital Library
- S. Andrist, T. Pejsa, B. Mutlu, and M. Gleicher. 2012b. A head-eye coordination model for animating gaze shifts of virtual characters. In Proceedings of 4th ICMI Workshop on Eye Gaze in Intelligent Human Machine Interaction (Gaze-In’12). ACM, 4:1--4:6. Google ScholarDigital Library
- M. Argyle and M. Cook. 1976. Gaze and Mutual Gaze. Cambridge University Press.Google Scholar
- A. T. Bahill, M. R. Clark, and L. Stark. 1975. The main sequence, a tool for studying human eye movements. Mathematical Biosciences 24, 34 (1975), 191--204.Google ScholarCross Ref
- G. Bailly, S. Raidt, and F. Elisei. 2010. Gaze, conversational agents and face-to-face communication. Speech Communication 52, 6 (June 2010), 598--612. Google ScholarDigital Library
- G. R. Barnes. 1979. Vestibulo-ocular function during co-ordinated head and eye movements to acquire visual targets. Journal of Physiology 287, 1 (1979), 127--147.Google ScholarCross Ref
- S. A. Beebe. 1976. Effects of Eye Contact, Posture and Vocal Inflection upon Credibility and Comprehension. Ph.D. Dissertation. University of Missouri-Columbia.Google Scholar
- A. R. Bentivoglio, S. B. Bressman, E. Cassetta, D. Carretta, P. Tonali, and A. Albanese. 1997. Analysis of blink rate patterns in normal subjects. Movement Disorders: Official Journal of the Movement Disorder Society 12, 6 (1997), 1028--1034.Google ScholarCross Ref
- J. K. Burgoon, D. A. Coker, and R. A. Coker. 1986. Communicative effects of gaze behavior. Human Communication Research 12, 4 (1986), 495--524.Google ScholarCross Ref
- A. Cafaro, R. Gaito, and H. Vilhjálmsson. 2009. Animating idle gaze in public places. In Proceedings of 9th International Conference on Intelligent Virtual Agents. Springer, 250--256. Google ScholarDigital Library
- J. Cassell, O. Torres, and S. Prevost. 1999. Turn taking vs. discourse structure: How best to model multimodal conversation. Machine Conversations, Y. Wilks (Ed.), Kluwer, 143--154.Google Scholar
- H. H. Clark and S. A. Brennan. 1991. Grounding in communication. In Perspectives on Socially Shared Cognition, L. B. Resnick, J. M. Levine, and S. D. Teasley (Eds.). APA Books.Google Scholar
- B. D’Entremont, A. Yazbeck, A. Morgan, and S. MacAulay. 2007. Early gaze-following and the understanding of others. In Gaze Following: Its Development and Significance, R. Flomm, K. Lee, and D. Muir (Eds.), Lawrence Erlbaum Associates, 77--94.Google Scholar
- C. Evinger, K. A. Manning, J. J. Pellegrini, M. A. Basso, A. S. Powers, and P. A. Sibony. 1994. Not looking while leaping: The linkage of blinking and saccadic gaze shifts. Experimental Brain Research 100, 2 (1994), 337--344.Google ScholarCross Ref
- J. Fox and J. N. Bailenson. 2009. Virtual virgins and vamps: The effects of exposure to female characters’ sexualized appearance and gaze in an immersive virtual environment. Sex Roles 61, 3 (2009), 147--157.Google ScholarCross Ref
- E. G. Freedman and D. L. Sparks. 2000. Coordination of the eyes and head: Movement kinematics. Experimental Brain Research 131, 1 (2000), 22--32.Google ScholarCross Ref
- A. Frischen, A. P. Bayliss, and S. P. Tipper. 2007. Gaze cueing of attention: Visual attention, social cognition, and individual differences. Psychological Bulletin 133, 4 (2007), 694--724.Google ScholarCross Ref
- R. Fry and G. F. Smith. 1975. The effects of feedback and eye contact on performance of a digit-coding task. Journal of Social Psychology 96, 1 (1975), 145--146.Google ScholarCross Ref
- J. H. Fuller. 1992. Head movement propensity. Experimental Brain Research 92, 1 (1992), 152--164.Google ScholarCross Ref
- C. Fullwood and G. Doherty-Sneddon. 2006. Effect of gazing at the camera during a video link on recall. Applied Ergonomics 37, 2 (2006), 167--175.Google ScholarCross Ref
- M. Garau, M. Slater, S. Bee, and M. A. Sasse. 2001. The impact of eye gaze on communication using humanoid avatars. In Proceedings of ACM/CHI 2001 Conference on Human factors in Computing Systems. ACM, 309--316. Google ScholarDigital Library
- G. N. Goldberg, C. A. Kiesler, and B. E. Collins. 1969. Visual behavior and face-to-face distance during interaction. Sociometry 32, 1 (1969), 43--53.Google ScholarCross Ref
- J. E. Goldring, M. C. Dorris, B. D. Corneil, P. A. Ballantyne, and D. R. Munoz. 1996. Combined eye-head gaze shifts to visual and auditory targets in humans. Experimental Brain Research 111, 1 (1996), 68--78.Google ScholarCross Ref
- H. H. Goossens and A. J. V. Opstal. 1997. Human eye-head coordination in two dimensions under different sensorimotor conditions. Experimental Brain Research 114, 3 (1997), 542--560.Google ScholarCross Ref
- Z. M. Griffin. 2001. Gaze durations during speech reflect word selection and phonological encoding. Cognition 82, 1 (2001), B1--B14.Google ScholarCross Ref
- H. Grillon and D. Thalmann. 2009. Simulating gaze attention behaviors for crowds. Computer Animation and Virtual Worlds 20, 23 (June 2009), 111--119. Google ScholarDigital Library
- E. Gu and N. I. Badler. 2006. Visual attention and eye gaze during multiparty conversations with distractions. In Proceedings of 6th International Conference on Intelligent Virtual Agents (IVA’06). Springer-Verlag, 193--204. Google ScholarDigital Library
- D. Guitton and M. Volle. 1987. Gaze control in humans: Eye-head coordination during orienting movements to targets within and beyond the oculomotor range. Journal of Neurophysiology 58, 3 (1987), 427--459.Google ScholarCross Ref
- M. J. Harris and R. Rosenthal. 2005. No more teachers' dirty looks: Effects of teacher nonverbal behavior on student outcomes. Applications of Nonverbal Communication, R. E. Riggio and R. S. Feldman (Eds.), Lawrence Erlbaum Mahwah, 157--192.Google Scholar
- R. M. Heck. 2007. Automated Authoring of Quality Human Motion for Interactive Environments. Ph.D. Dissertation. University of Wisconsin--Madison. Google ScholarDigital Library
- D. Heylen, I. Van Es, E. Van Dijk, A. Nijholt, and B. Van Dijk. 2002. Experimenting with the gaze of a conversational agent. In Proceedings of International CLASS Workshop on Natural, Intelligent and Effective Interaction in Multimodal Dialogue Systems. Kluwer Academic Publishers, 93--100.Google Scholar
- J. K. Hietanen. 1999. Does your gaze direction and head orientation shift my visual attention? Neuroreport 10, 16 (1999), 3443.Google Scholar
- J. K. Hietanen. 2002. Social attention orienting integrates visual information from head and body orientation. Psychological Research 66, 3 (2002), 174--179.Google ScholarCross Ref
- M. A. Hollands, N. V. Ziavra, and A. M. Bronstein. 2004. A new paradigm to investigate the roles of head and eye movements in the coordination of whole-body movements. Experimental Brain Research 154, 2 (2004), 261--266.Google ScholarCross Ref
- D. H. Kelley and J. Gorham. 1988. Effects of immediacy on recall of information. Communication Education 37, 3 (1988).Google Scholar
- A. Kendon. 1973. The role of visible behaviour in the organization of social interaction. In Social Communication and Movement, M. V. Cranach and I. Vine (Eds.), Academic Press, 29--74.Google Scholar
- A. Kendon. 2010. Spacing and orientation in co-present interaction. In Development of Multimodal Interfaces: Active Listening and Synchrony. Lecture Notes in Computer Science, Vol. 5967. Springer, Berlin, 1--15. Google ScholarDigital Library
- S. C. Khullar and N. I. Badler. 2001. Where to look? Automating attending behaviors of virtual human characters. Autonomous Agents and Multi-Agent Systems 4, 1 (2001), 9--23. Google ScholarDigital Library
- K. H. Kim, R. Brent Gillespie, and B. J. Martin. 2007. Head movement control in visually guided tasks: Postural goal and optimality. Computers in Biology and Medicine 37, 7 (2007), 1009--1019. Google ScholarDigital Library
- A. Kittur, E. H. Chi, and B. Suh. 2008. Crowdsourcing user studies with Mechanical Turk. In Proceedings of ACM/CHI 2008 Conference on Human factors in Computing Systems (CHI’08). ACM, 453--456. Google ScholarDigital Library
- H. Kuzuoka, Y. Suzuki, J. Yamashita, and K. Yamazaki. 2010. Reconfiguring spatial formation arrangement by robot body orientation. In Proceedings of 5th ACM/IEEE International Conference on Human-Robot Interaction (HRI’10). 285--292. Google ScholarDigital Library
- B. J. Lance and S. C. Marsella. 2010. The expressive gaze model: Using gaze to express emotion. IEEE Computer Graphics & Applications 30, 4 (2010), 62--73. Google ScholarDigital Library
- S. R. H. Langton, R. J. Watt, and V. Bruce. 2000. Do the eyes have it? Cues to the direction of social attention. Trends in Cognitive Sciences 4, 2 (2000), 50--59.Google ScholarCross Ref
- J. Lee, S. Marsella, D. Traum, J. Gratch, and B. Lance. 2007. The rickel gaze model: A window on the mind of a virtual human. In Proceedings of 7th International Conference on Intelligent Virtual Agents (IVA’07). Springer, 296--303. Google ScholarDigital Library
- S. P. Lee, J. B. Badler, and N. I. Badler. 2002. Eyes alive. ACM Transactions on Graphics 21, 637--644. Google ScholarDigital Library
- Z. Li, X. Mao, and L. Liu. 2009. Providing expressive eye movement to virtual agents. In Proceedings of the 2009 International Conference on Multimodal Interfaces (ICMI-MLMI’09). 241--244. Google ScholarDigital Library
- C. Liu, D. L. Kay, and J. Y. Chai. 2011. Awareness of partner’s eye gaze in situated referential grounding: An empirical study. In Proceedings of the 2nd Workshop on Eye Gaze in Intelligent Human Machine Interaction.Google Scholar
- M. M. Lusk and R. K. Atkinson. 2007. Animated pedagogical agents: Does their degree of embodiment impact learning from static or animated worked examples? Applied Cognitive Psychology 21, 6 (2007), 747--764.Google Scholar
- M. K. McCluskey and K. E. Cullen. 2007. Eye, head, and body coordination during large gaze shifts in rhesus monkeys: Movement kinematics and the influence of posture. Journal of Neurophysiology 97, 4 (2007), 2976--2991.Google ScholarCross Ref
- A. Mehrabian. 1966. Immediacy: An indicator of attitudes in linguistic communication. Journal of Personality 34, 1 (1966), 26--34.Google ScholarCross Ref
- A. S. Meyer, A. M. Sleiderink, and W. J. M. Levelt. 1998. Viewing and naming objects: Eye movements during noun phrase production. Cognition 66, 2 (1998), B25--B33.Google ScholarCross Ref
- J. Mumm and B. Mutlu. 2011. Designing motivational agents: The role of praise, social comparison, and embodiment in computer feedback. Computers in Human Behavior 27, 5 (Sept. 2011), 1643--1650. Google Scholar
- D. L. Mumme, E. W. Bushnell, J. A. DiCorcia, and L. A. Lariviere. 2007. Infants’ use of gaze cues to interpret others’ actions and emotional reactions. In Gaze Following: Its Development and Significance. Lawrence Erlbaum Associates, 143--170.Google Scholar
- B. Mutlu, J. Forlizzi, and J. Hodgins. 2006. A storytelling robot: Modeling and evaluation of human-like gaze behavior. In Proceedings of 6th IEEE-RAS International Conference on Humanoid Robots. IEEE, 518--523.Google Scholar
- B. Mutlu, T. Kanda, J. Forlizzi, J. Hodgins, and H. Ishiguro. 2012. Conversational gaze mechanisms for humanlike robots. ACM Transactions on Interactive Intelligent Systems 1, 2 (Jan. 2012), 12:1--12:33. Google ScholarDigital Library
- Y. I. Nakano, G. Reinstein, T. Stocky, and J. Cassell. 2003. Towards a model of face-to-face grounding. In Proceedings of the 41st Annual Meeting on Association for Computational Linguistics - Volume 1 (ACL’03). Association for Computational Linguistics, 553--561. Google ScholarDigital Library
- J. P. Otteson and C. R. Otteson. 1980. Effect of teacher’s gaze on children’s story recall. Perceptual and Motor Skills 50, 1 (1980), 35--42.Google ScholarCross Ref
- T. Pejsa, B. Mutlu, and Gleicher M. 2013. Stylized and performative gaze for character animation. Computer Graphics Forum 32, 2 (2013), 143--152.Google ScholarCross Ref
- T. Pejsa and I. S. Pandzic. 2010. State of the art in example-based motion synthesis for virtual characters in interactive applications. Computer Graphics Forum 29, 1 (2010), 202--226.Google ScholarCross Ref
- C. Pelachaud and M. Bilvi. 2003. Modelling gaze behavior for conversational agents. In Proceedings of 4th International Workshop on Intelligent Virtual Agents (IVA’03). Springer, 93--100.Google Scholar
- J. Pelz, M. Hayhoe, and R. Loeber. 2001. The coordination of eye, head, and hand movements in a natural task. Experimental Brain Research 139, 3 (2001), 266--277.Google ScholarCross Ref
- D. I. Perrett and N. J. Emery. 1994. Understanding the intentions of others from visual signals: Neurophysiological evidence. Cahiers de Psychologie Cognitive/Current Psychology of Cognition 13, 5 (1994), 683--694.Google Scholar
- C. Peters and A. Qureshi. 2010. A head movement propensity model for animating gaze shifts and blinks of virtual characters. Computers & Graphics 34, 6 (2010), 677--687. Google ScholarDigital Library
- I. Pomianowska, F. Germeys, K. Verfaillie, and F. N. Newell. 2011. The role of social cues in the deployment of spatial attention: Head-body relationships automatically activate directional spatial codes in a Simon task. Frontiers in Integrative Neuroscience 6, 4 (2011).Google Scholar
- M. Scaife and J. S. Bruner. 1975. The capacity for joint visual attention in the infant. Nature 253, 1 (1975), 265--266.Google ScholarCross Ref
- E. A. Schegloff. 1998. Body torque. Social Research 65, 3 (1998), 535--596.Google Scholar
- J. V. Sherwood. 1987. Facilitative effects of gaze upon learning. Perceptual and Motor Skills 64, 3 (1987), 1275--1278.Google ScholarCross Ref
- A. Tartaro. 2007. Authorable virtual peers for children with autism. In CHI’07 Extended Abstracts on Human Factors in Computing Systems (CHI EA’07). ACM, New York, NY, 1677--1680. Google ScholarDigital Library
- M. Thiebaux, B. Lance, and S. Marsella. 2009. Real-time expressive gaze animation for virtual humans. In Proceedings of the 8th International Conference on Autonomous Agents and Multiagent Systems - Volume 1 (AAMAS’09). International Foundation for Autonomous Agents and Multiagent Systems, 321--328. Google ScholarDigital Library
- T. Uemura, Y. Arai, and C. Shimazaki. 1980. Eye-head coordination during lateral gaze in normal subjects. Acta Oto-Laryngologica 90, 3--4 (1980), 191--198.Google ScholarCross Ref
- A. Woodward. 2005. Infants’ understanding of the actions involved in joint attention. In Joint Attention: Communication and Other Minds. Oxford University Press, 110--128.Google Scholar
- W. H. Zangemeister and L. Stark. 1982. Types of gaze movement: Variable interactions of eye and head movements. Experimental Neurology 77, 3 (1982), 563--577.Google ScholarCross Ref
Index Terms
- Gaze and Attention Management for Embodied Conversational Agents
Recommendations
Designing effective gaze mechanisms for virtual agents
CHI '12: Proceedings of the SIGCHI Conference on Human Factors in Computing SystemsVirtual agents hold great promise in human-computer interaction with their ability to afford embodied interaction using nonverbal human communicative cues. Gaze cues are particularly important to achieve significant high-level outcomes such as improved ...
Embodied conversational agents: computing and rendering realistic gaze patterns
PCM'06: Proceedings of the 7th Pacific Rim conference on Advances in Multimedia Information ProcessingWe describe here our efforts for modeling multimodal signals exchanged by interlocutors when interacting face-to-face. This data is then used to control embodied conversational agents able to engage into a realistic face-to-face interaction with human ...
Personality Analysis of Embodied Conversational Agents
IVA '18: Proceedings of the 18th International Conference on Intelligent Virtual AgentsPeople tend to personify machines. Giving machines the ability to actually produce social information can help improve human-machine interactions. Embodied Conversational Agents (ECAs) are virtual software agents that can process and produce speech, ...
Comments