Visual display, pointing, and natural language: the power of multimodal interaction

ABSTRACT

This paper examines user behavior during multimodal human-computer interaction (HCI). It discusses how pointing, natural language, and graphical layout should be integrated to enhance the usability of multimodal systems. Two experiments were run to study simulated systems capable of understanding written natural language and mouse-supported pointing gestures. Results allowed to: (a) develop a taxonomy of communication acts aimed at identifying targets; (b) determine the conditions under which specific referent identification strategies are likely to be produced; (c) suggest guidelines for designing effective multimodal interfaces; (d) show that performance is strongly influenced by interface graphical layout and by user expertise. Our study confirms the value of simulation as a tool for building HCI models and supports the basic idea that linguistic, visual, and motor cues can be integrated to favor effective multimodal communication.

References

1. Brennan, S. E. Conversation With and Through Computers. User Modeling and User-Adapted Interaction, 1, 1991, 67--86.
2. Buxton, B. The Natural Language of Interaction: A Perspective on Nonverbal Dialogues. In B. Laurel (Ed.) The Art of Human-Computer Interface Design, Addison-Wesley, 1991, 405--416.
3. Caramazza, A. and Hillis, A. E. Lexical Organization of Nouns and Verbs in Brain. Nature, 349, 1991, 788--790.
4. Cassell, J. and Prevost, S. Distribution of Semantic Features Across Speech and Gesture by Humans and Machine. In L. Messing (Ed.) Proc. of the Workshop of the Integration of Gesture in Language and Speech, University of Delaware, 1996.
5. Cohen, P. R., Dalrymple M., Moran, D. B., Pereira, F. C. N., Sullivan, J. W, Gargan, R. A., Schlossberg, J. L., and Tyler., S. W. Synergistic Use of Direct Manipulation and Natural Language. In CHI'89 Conf. Proc., ACM, Addison-Wesley, New York, 227--234.
6. Clark, H. H., Schreuder, R., and Buttrick, S. Common Ground and the Understanding of Demostrative Reference. Journal of Verbal Learning and Verbal Behavior, 22, 1983, 245--258.
7. Dahlback, N and Johnsson, A. Empirical studies of discourse representation for natural language interfaces. In Proc. of 4th European Conf. of ACL, 1989, 291--298.
8. De Angeli, A., Petrelli, D. and Gerbino, W. Interface Features Affecting Deixis Production: A Simulation Study. In L. Messing (Ed.) Proc. of the Workshop of the Integration of Gesture in Language and Speech, University of Delaware, 1996, 195--204.
9. De Angeli, A., Petrelli, D., Gerbino, W. & Cassano, G. How Computer Literacy Affects Natural Language Interaction. Technical Report of the Cognitive Technology Laboratory, University of Trieste, 1997.
10. Fraser, N. M. and Gilbert, G. N. Simulating Speech System. Computer, Speech and Languages, 5, 1991, 81--99.
11. Gentner, D. and Nielsen, J. The Anti-Mac Interface. Communication of the ACM, 39(8), 1996, 70--82.
12. Gerbino W. Solving by Redundancy and Misunderstanding by Simplification. In V. Cantoni, V. Di Gesù, A. Setti and D. Tegolo (Eds.) Human and Machine Perception: Information Fusion. New York, Plenum, 1997, 147--154.
13. Glenberg, A., and McDaniel, M. Mental Models, Pictures, and Text: Integration of Spatial and Verbal Information. Memory and Cognition, 20 (5), 1992, 458--460.
14. Gullberg, M. Deictic Gesture and Strategy in Second Language Narrative. In L. Messing (Ed.) Proc. of the Workshop of the Integration of Gesture in Language and Speech, University of Delaware, 1996, 155--174.
15. Johnson, A. and Dahlback, N. Talking To a Computer is Not Like Talking To Your Best Friend. Proc. of Scandinavian Conf. of Artificial Intelligence, 1988, 53--68.
16. Kendon, A. Gesticulation and Speech: Two Aspects of the Process of Utterances. In M. R. Key (Ed.) Nonverbal Communication and Language, The Hague: Mounton, 1980.
17. Levelt, W. J., Richardson, G., and La Heij, W. Pointing and Voicing in Deictic Expressions. Journal of Memory and Language, 24, (1985), 133--164.
18. Levialdi, S., Mussio, P., and Mastronardi, G. Characters, Pixels, and Phonemes. In V. Cantoni, V. Di Gesù, A. Setti and D. Tegolo (Eds.) Human and Machine Perception: Information Fusion. New York, Plenum, 1997.
19. Levinson, S. C. Pragmatics. Cambridge: Cambridge University Press, 1983.
20. Lewkowicz, D. J. and Lickliter, R. The Development of Intersensory Perception: Comparative Perspectives. Hillsdale, NJ: LEA, 1994.
21. Mac Aogain, E. and Reilly, R. Discourse Theory and Interface Design: The Case of Pointing With the Mouse. Intl. Journal of Man-Machine Studies, 32, 1990, 591--602.
22. Maybury, M. T. (Ed.), Intelligent Multimedia Interfaces. Cambridge, Mass: MIT Press, 1993.
23. McNeill, D. Hand and Mind: What Gestures Reveal about Thought. Chicago: University of Chicago Press, 1992.
24. McNeill, D. Language as Gesture (Gesture as Language). In L. Messing (Ed.) Proc. of the Workshop of the Integration of Gesture in Language and Speech, University Of Delaware, 1996, 1--20.
25. Naughton, K. Spontaneous Gesture and Sign: A Study of ASL Signs Co-occurring with Speech. In L. Messing (Ed.) Proc. of the Workshop of the Integration of Gesture in Language and Speech, University Of Delaware, 1996, 125--34.
26. Norman, D. A. and Draper, S. W. User Centered System Design: New Perspectives on Human-Computer Interaction. Hillsdale, New Jersey: Lawrence Erlbaum Associates, 1986.
27. Oviatt, S. Multimodal Interfaces for Dynamic Interactive Maps. CHI'96 Conf. Proc., New York: ACM Press, 1996, 95--102.
28. Oviatt, S. and Olsen, E. Integration Themes in Multimodal Human-Computer Interaction. In Proc. of the Intl. Conf. on Spoken Language Processing, Yokohama, Japan, 1994, 551--554.
29. Oviatt, S. and vanGent, R. Error Resolution During Multimodal Human-Computer Interaction. In Proc. of the Intl. Conf. on Spoken Language Processing, Philadelfia, USA, 1996.
30. Oviatt, S., Cohen, P. R. and Wang, M. Toward Interface Design for Human Language Technology: Modality and Structure as Determinants of Linguistic Complexity. Speech Communication, 15, 1994, 283--300.
31. Oviatt, S., Cohen, P. R., Fong, M. and Frank, M. A Rapid Semi-automatic Simulation Technique for Investigating Interactive Speech and Handwriting. Proc. of the Intl. Conf. on Spoken Language Processing, vol. 2, Alberta, 1992, 1351--1354.
32. Oviatt, S., De Angeli, A. and Kuhn, K. Integration and Synchronization of Input Modes During Multimodal Human-Computer Interaction. In CHI'96 Conf. Proc., New York: ACM Press, 1997, 415--422.
33. Petrelli, D., De Angeli, A., Gerbino, W. and Cassano, G. Referring in Multimodal Systems: The Importance of User Expertise and System Features. In Proc. of the Workshop on Referring Phenomena in a Multimedia Context and Their Computational Treatment, ACL-EACL, 1997, 14--19.
34. Schmauks, D. Natural and Simulated Pointing. Proc. of the 3rd European ACL Conf., Kopenhagen, Danmark, 1987, 179--185.
35. Schmauks, D. and Reithinger, N. Generating Multimodal Output-Conditions, Advantages and Problems. In Proc. of the Intl. Conf. on Computational Linguistics, Budapest, Hungary, 1988, 584--588.
36. Schmauks, d. and Wille M. Integration of Communicative Hand Movements into Human-Computer-Interaction. Computers and the Humanities, 25, 1991, 129--140.
37. Shneiderman, B. Designing the User Interface: Strategies for Effective Human-Computer Interaction. Reading, MA: Addison-Wesley, second edition, 1992.
38. Siroux, J., Guyomard, M., Multon, F. and Remondeau, C. Oral and Gestural Activities of the Users in the Georal System. In Proc. of the Intl. Conf. on Cooperative Multimodal Communication, vol. 2, 1995, 287--298.
39. Stock, O. A Third Modality? Artificial Intelligence Review, 9, 1995, 129--146.
40. van Vliet, P. J., Kletke, M. G. and Chakraborty, G. The measurement of computer literacy: a comparison of self-appraisal and objective tests. Intl. Journal of Human-Computer Studies, 40, 1994, 835--875.

• Hardware

  • Power and energy

    • Power estimation and optimization

• Human-centered computing