Clavius: bi-directional parsing for generic multimodal interaction

ABSTRACT

We introduce a new multi-threaded parsing algorithm on unification grammars designed specifically for multimodal interaction and noisy environments. By lifting some traditional constraints, namely those related to the ordering of constituents, we overcome several difficulties of other systems in this domain. We also present several criteria used in this model to constrain the search process using dynamically loadable scoring functions. Some early analyses of our implementation are discussed.

References

1. ]]Ageno, A., Rodriguez, H. 2000 Extending Bidirectional Chart Parsing with a Stochastic Model, in Proc. of TSD 2000, Brno, Czech Republic.
2. ]]Alexandersson, J. and Becker, T. 2001 Overlay as the Basic Operation for Discourse Processing in a Multimodal Dialogue System in Proc. of the 2nd IJCAI Workshop on Knowledge and Reasoning in Practical Dialogue Systems, Seattle, WA.
3. ]]Bolt, R. A. 1980 "Put-that-there": Voice and gesture at the graphics interface in Proc. of SIGGRAPH 80 ACM Press, New York, NY.
4. ]]Boussemart, Y., Rioux, F., Rudzicz, F., Wozniewski, M., Cooperstock, J. 2004 A Framework for 3D Visualisation and Manipulation in an Immersive Space using an Untethered Bimanual Gestural Interface in Proc. of VRST 2004 ACM Press, Hong Kong.
5. ]]Dowding, J. et al. 1993 Gemini: A Natural Language System For Spoken-Language Understanding in Meeting of the ACL, ACL, Morristown, NJ.
6. ]]Holzapfel, H., Nickel, K., Stiefelhagen, R. 2004 Implementation and evaluation of a constraint-based multimodal fusion system for speech and 3D pointing gestures, in ICMI '04: Proc. of the 6th intl. conference on Multimodal interfaces, ACM Press, New York, NY.
7. ]]Johnston, M. 1998 Unification-based multimodal parsing, in Proc. of the 36th annual meeting of the ACL, ACL, Morristown, NJ.
8. ]]Johnston, M., Bangalore, S. 2000 Finite-state multimodal parsing and understanding in Proc. of the 18th conference on Computational linguistics ACL, Morristown, NJ.
9. ]]Kettebekov, S., et al. 2002 Prosody Based Co-analysis of Deictic Gestures and Speech in Weather Narration Broadcast, in Workshop on Multimodal Resources and Multimodal System Evaluation. (LREC 2002), Las Palmas, Spain.
10. ]]McNeill, D. 1992 Hand and mind: What gestures reveal about thought University of Chicago Press and CSLI Publications, Chicago, IL.
11. ]]Rocio, V., Lopes, J. G. 1998 Partial Parsing, Deduction and Tabling in TAPD 98
12. ]]Tomita, M. 1985 An Efficient Context-Free Parsing Algorithm for Natural Languages, in Proc. Ninth Intl. Joint Conf. on Artificial Intelligence, Los Angeles, CA.

• Hardware

  • Power and energy

    • Power estimation and optimization