Emotional input for character-based interactive storytelling

ABSTRACT

In most Interactive Storytelling systems, user interaction is based on natural language communication with virtual agents, either through isolated utterances or through dialogue. Natural language communication is also an essential element of interactive narratives in which the user is supposed to impersonate one of the story's characters. Whilst techniques for narrative generation and agent behaviour have made significant progress in recent years, natural language processing remains a bottleneck hampering the scalability of Interactive Storytelling systems. In this paper, we introduce a novel interaction technique based solely on emotional speech recognition. It allows the user to take part in dialogue with virtual actors without any constraints on style or expressivity, by mapping the recognised emotional categories to narrative situations and virtual characters feelings. Our Interactive Storytelling system uses an emotional planner to drive characters' behaviours. The main feature of this approach is that characters' feelings are part of the planning domain and are at the heart of narrative representations. The emotional speech recogniser analyses the speech signal to produce a variety of features which can be used to define ad-hoc categories on which to train the system. The content of our interactive narrative is an adaptation of one chapter of the XIX^th century classic novel, Madame Bovary, which is well suited to a formalisation in terms of characters' feelings. At various stages of the narrative, the user can address the main character or respond to her, impersonating her lover. The emotional category extracted from the user utterance can be analysed in terms of the current narrative context, which includes characters' beliefs, feelings and expectations, to produce a specific influence on the target character, which will become visible through a change in its behaviour, achieving a high level of realism for the interaction. A limited number of emotional categories is sufficient to drive the narrative across multiple courses of actions, since it comprises over thirty narrative functions. We report results from a fully implemented prototype, both in terms of proof of concept and of usability through a preliminary user study.

References

1. ]]Bonet, B., Geffner, H. 1999. Planning as Heuristic Search: New Results, Proceedings of the European Conference on Planning (ECP'99), pp. 360--372.
2. ]]Bonet, B., Geffner, H. 2001. Planning as Heuristic Search. Artificial Intelligence Special Issue on Heuristic Search, 129, n. 1, 2001, pp. 5--33.
3. ]]Cavazza, M., Charles, F. and Mead, S. J., 2002. Character-based Interactive Storytelling. IEEE Intelligent Systems, 17, 4, 2002, pp. 17--24.
4. ]]Chang, C.-C. and Lin, C.-J. 2001. LIB-SVM: a library for support vector machines. Software available at http://www.csie.ntu.edu.tw/cjlin/libsvm.
5. ]]Diener, H. and Oertel, K. 2006. Experimental Approach to Affective Interaction in Games. Edutainment 2006: pp. 507--518.
6. ]]Flaubert, G. 1856. Madame Bovary. La revue de Paris (Ed.), France. (in French).
7. ]]Freeman, D. 2003. Creating Emotion in Games: The Craft and Art of Emotioneering. New Riders. 2003.
8. ]]Gratch, J. Why you should buy an emotional planner, Proceedings of the Autonomous Agents 1999 Workshop on Emotion-based Agent Architectures (EBAA'99).
9. ]]Jones, C. and Sutherland, J. 2007. Acoustic emotion recognition for affective computer gaming. In Peter, C. and Beale, R., editors, Affect and Emotion in Human-Computer Interaction, volume 4868 of LNCS. Springer, Heidelberg, Germany.
10. ]]Kenny, G., P., Parsons, T., D., Gratch, J., Rizzo, A., 2008. Evaluation of Justina: A Virtual Patient with PTSD. In Proceedings of the conference on IVA 2008, pp. 394--408.
11. ]]Korf, R. E. Real-time heuristic search. Artificial Intelligence, 42:2--3, 1990, pp. 189--211.
12. ]]Leclerc, Y (Ed.). Plans et Scenarios de Madame Bovary, CNRS Editions, France, 1995 (in French).
13. ]]Liu Y., Koenig S. and Furcy D., Speeding Up the Calculation of Heuristics for Heuristic Search-Based Planning, In Proceedings of the Eighteenth National Conference on Artificial Intelligence, pp. 484--491, 2002.
14. ]]Louchart, S., Romano, D., M., Aylett, R., and Pickering, J. 2004. Speaking and acting - interacting language and action for an expressive character. In Proceedings for the AISB workshop, University of Leeds, UK.
15. ]]Lyons, G., Sharma, P., Baker, M., O'Malley, S., and Shanahan, A. 2003. A computer gamebased EMG biofeedback system for muscle rehabilitation. In: Engineering in Medicine and Biology Society, 2003. Proceedings of the 25th Annual International Conference of the IEEE. volume 2. S. 1625--1628. Sept. 2003.
16. ]]Marsella, S. and Gratch, J. 2002. A Step Towards Irrationality: Using Emotion to Change Belief, in Proceedings of the First International Joint Conference on Autonomous Agents and Multi-Agent Systems, Bologna, Italy, July 2002.
17. ]]Marsella, S. and Gratch, J. 2003. Modeling coping behavior in virtual humans: don't worry, be happy. In Proceedings of the Second international Joint Conference on Autonomous Agents and Multiagent Systems (Melbourne, Australia, July 14--18, 2003). AAMAS '03. ACM, New York, NY, pp. 313--320.
18. ]]Mateas, M., Stern, A., 2004. Natural Language Understanding in Façade: Surface-Text Processing. TIDSE 2004: 3--13.
19. ]]Mohamad, Y., Velasco, C., A, Berlage, T. 2003. An Approach toward building of adaptive training and therapeutic systems considering the users affective state. HCI International Proceedings 2003, Crete, Greece.
20. ]]Murray, J. 1997. Hamlet on the Holodeck: The Future of Narrative in Cyberspace. MIT Press, Cambridge, 1997.
21. ]]Nazir, A., Lim, M., Y., Kriegel, M., Aylett, R., Cawsey, A., Enz, S., Rizzo, P. and Hall, L. 2008. ORIENT: An Inter-Cultural Role-Play Game. In Narrative in Interactive Learning Environments NILE 2008 Conference.
22. ]]Paiva, A., Chaves, R., Piedade, M., Bullock, A., Andersson, G., and Höök, K. 2003. Sentoy: A tangible interface to control the emotions of a synthetic character. In: Proc. of AAMAS 2003. pp. 1088--1089. 2003.
23. ]]Pizzi, D., Charles, F., Lugrin J-L., and Cavazza, M., 2007. Interactive Storytelling with Literary Feelings. In: Paiva, A. C. R., Prada, R., Picard, R. W. (eds.) ACII 2007. LNCS, vol. 4738, pp. 630--641.
24. ]]Rank, S., Petta, P. and Trappl R. 2006. Features of Emotional Planning in Software Agents, in Dellariccia G., et al.(eds.), Decision Theory and Multi-Agent Planning, Springer Wien/New York, 2006.
25. ]]Rehm, M., Vogt, T., Wissner, M. and Bee, N. 2008. Dancing the Night Away --- Controlling a Virtual Karaoke Dancer by Multimodal Expressive Cues. In Proceedings of AAMAS, 2008.
26. ]]Riedl, M. O., Young, R. M. 2004. An Intent-Driven Planner for Multi-Agent Story Generation. Third ACM Joint Conference on Autonomous Agents and Multi-Agent Systems, New York, USA, 2004, pp. 186--193.
27. ]]Swartout, W., Gratch, J., Hill, R. Hovy, E., Marsella, S. Rickel, J. and Traum, D., 2006. Toward Virtual Humans in AI Magazine, v. 27(1).
28. ]]Traum, D., Rickel, J., Gratch, J., and Marsella, S. 2003. Negotiation over tasks in hybrid human-agent teams for simulation-based training. In Proceedings of the Second international Joint Conference on Autonomous Agents and Multiagent Systems (Melbourne, Australia, July 14--18, 2003). AAMAS '03. ACM, New York, NY, pp. 441--448.
29. ]]Velten, E. 1968. A laboratory task for induction of mood states. Behavior Research & Therapy, (6):473--482.
30. ]]Vogt, T., André, E. and Bee, N. 2008. EmoVoice - A framework for online recognition of emotions from voice. In Proceedings of Workshop on Perception and Interactive Technologies for Speech-Based Systems, Springer, Kloster Irsee, Germany, (June 2008).
31. ]]Weiß, S., Müller, W., Spierling, U., Steimle, F., 2005. Scenejo - An Interactive Storytelling Platform. In: Virtual Storytelling, Using Virtual Reality Technologies for Storytelling, Third International Conference, Proceedings, Strasbourg, France, pp. 77--80.
32. ]]Wilting, J., Krahmer, E., and Swerts, M. 2006. Real vs. acted emotional speech. In Proceedings of Interspeech 2006 | ICSLP, Pittsburgh, PA, USA.
33. ]]Young, R., M. 2000. Creating Interactive Narrative Structures: The Potential for AI Approaches. AAAI Spring Symposium in Artificial Intelligence and Entertainment. AAAI Press, 2000.
34. ]]Young, R., M. 2001. An Overview of the Mimesis Architecture: Integrating Intelligent Narrative Control into an Existing Gaming Environment, The Working Notes of the AAAI Spring Symposium on Artificial Intelligence and Interactive Entertainment, Stanford, CA, March 2001.

Index Terms

• Information systems

  • Information systems applications

    • Multimedia information systems