Baptiste Caramiaux
Atau Tanaka
Skip Abstract Section
Abstract
This article presents a gesture recognition/adaptation system for human--computer interaction applications that goes beyond activity classification and that, as a complement to gesture labeling, characterizes the movement execution. We describe a template-based recognition method that simultaneously aligns the input gesture to the templates using a Sequential Monte Carlo inference technique. Contrary to standard template-based methods based on dynamic programming, such as Dynamic Time Warping, the algorithm has an adaptation process that tracks gesture variation in real time. The method continuously updates, during execution of the gesture, the estimated parameters and recognition results, which offers key advantages for continuous human--machine interaction. The technique is evaluated in several different ways: Recognition and early recognition are evaluated on 2D onscreen pen gestures; adaptation is assessed on synthetic data; and both early recognition and adaptation are evaluated in a user study involving 3D free-space gestures. The method is robust to noise, and successfully adapts to parameter variation. Moreover, it performs recognition as well as or better than nonadapting offline template-based methods.
- M. S. Arulampalam, S. Maskell, N. Gordon, and T. Clapp. 2002. A tutorial on particle filters for online nonlinear/non-Gaussian bayesian tracking. IEEE Transactions on Signal Processing 50, 2, 174--188. Google Scholar
Digital Library
- S. Barrass and G. Kramer. 1999. Using sonification. Multimedia Systems 7, 1, 23--31. Google ScholarDigital Library
- O. Bau and W. E. Mackay. 2008. OctoPocus: a dynamic guide for learning gesture-based command sets. In Proceedings of the 21st Annual ACM Symposium on User Interface Software and Technology. ACM, 37--46. Google ScholarDigital Library
- F. Bevilacqua, F. Baschet, and S. Lemouton. 2012. The augmented string quartet: Experiments and gesture following. Journal of New Music Research 41, 1, 103--119.Google ScholarCross Ref
- F. Bevilacqua, F. Guédy, N. Schnell, E. Fléty, and N. Leroy. 2007. Wireless sensor interface and gesture-follower for music pedagogy. In Proceedings of the 7th International Conference on New Interfaces for Musical Expression. ACM, 124--129. Google ScholarDigital Library
- F. Bevilacqua, N. Schnell, and S. Fdili Alaoui. 2011a. Gesture capture: Paradigms in interactive music/dance systems. In Emerging Bodies: The Performance of Worldmaking in Dance and Choreography, G. Klein, S. Noeth (Eds.), Transaction Publishers, New Brunswick, NJ, 183--194.Google Scholar
- F. Bevilacqua, N. Schnell, N. Rasamimanana, B. Zamborlin, and F. Guédy. 2011b. Online gesture analysis and control of audio processing. In Musical Robots and Interactive Multimodal Systems, J. Solis, K Ng (Eds), Springer-Verlag, Berlin, 127--142.Google Scholar
- F. Bevilacqua, B. Zamborlin, A. Sypniewski, N. Schnell, F. Guédy, and N. Rasamimanana. 2010. Continuous realtime gesture following and recognition. In Embodied Communication and Human-Computer Interaction, Volume 5934 of Lecture Notes in Computer Science. Springer, Berlin, 73--84. Google ScholarDigital Library
- J. Bilmes. 2002. What HMMs Can Do. Technical Report. University of Washington, Department of Electrical Engineering, Seattle, WA.Google Scholar
- M. Black and A. Jepson. 1998a. A probabilistic framework for matching temporal trajectories: Condensation-based recognition of gestures and expressions. In Proceedings of the European Conference on Computer Vision (ECCV'98). Springer, 909--924. Google ScholarDigital Library
- M. J. Black and A. D. Jepson. 1998b. Recognizing temporal trajectories using the condensation algorithm. In Proceedings of the 3rd IEEE International Conference on Automatic Face and Gesture Recognition. IEEE, 16--21. Google ScholarDigital Library
- A. F. Bobick and A. D. Wilson. 1997. A state-based approach to the representation and recognition of gesture. IEEE Transactions on Pattern Analysis and Machine Intelligence 19, 12 (1997), 1325--1337. Google ScholarDigital Library
- E. O. Boyer, B. M. Babayan, F. Bevilacqua, M. Noisternig, O. Warusfel, A. Roby-Brami, S. Hanneton, and I. Viaud-Delmon. 2013. From ear to hand: The role of the auditory-motor loop in pointing to an auditory source. Frontiers in Computational Neuroscience 7.Google Scholar
- M. Brand and A. Hertzmann. 2000. Style machines. In Proceedings of the 27th Annual Conference on Computer Graphics and Interactive Techniques. ACM Press/Addison-Wesley Publishing Co., 183-- 192. Google ScholarDigital Library
- L. Bretzner, I. Laptev, and T. Lindeberg. 2002. Hand gesture recognition using multi-scale colour features, hierarchical models and particle filtering. In Proceedings of the 5th IEEE International Conference on Automatic Face and Gesture Recognition. IEEE, 423--428. Google ScholarDigital Library
- B. Caramiaux. 2012. Studies on the Relationship between Gesture and Sound in Musical Performance. Ph.D. Dissertation. University Paris VI, IRCAM Centre Pompidou.Google Scholar
- B. Caramiaux, F. Bevilacqua, and N. Schnell. 2010. Analysing gesture and sound similarities with a HMM-based divergence measure. In Proceedings of the 6th Sound and Music Conference. Barcelona, Spain.Google Scholar
- B. Caramiaux, F. Bevilacqua, and A. Tanaka. 2013. Beyond recognition: Using gesture variation for continuous interaction. In Proceedings of the CHI Conference on Human Factors in Computing Systems, Extended Abstracts, Alt.CHI. ACM, New York, NY, 2109--2118. Google ScholarDigital Library
- G. Caridakis, K. Karpouzis, N. Drosopoulos, and S. Kollias. 2009. Adaptive gesture recognition in human computer interaction. In Proceedings of the 10th Workshop on Image Analysis for Multimedia Interactive Services (WIAMIS'09). IEEE, 270--274.Google Scholar
- R. Chavarriaga, H. Bayati, and J. D. Millán. 2013. Unsupervised adaptation for acceleration-based activity recognition: Robustness to sensor displacement and rotation. Personal and Ubiquitous Computing 17, 3, 479--490. Google ScholarDigital Library
- R. Douc and O. Cappé. 2005. Comparison of resampling schemes for particle filtering. In Proceedings of the 4th International Symposium on Image and Signal Processing and Analysis (ISPA'05). IEEE, 64--69.Google Scholar
- A. Doucet, N. De Freitas, and N. Gordon. 2001. Sequential Monte Carlo Methods in Practice. Springer Verlag.Google Scholar
- P. Dourish. 2004. Where the Action Is: The Foundations of Embodied Interaction. MIT Press, Cambridge, MA.Google Scholar
- K. Forbes and E. Fiume. 2005. An efficient search algorithm for motion data using weighted PCA. In Proceedings of the 2005 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. ACM, New York, NY, 67--76. Google ScholarDigital Library
- D. M. Gavrila and L. S. Davis. 1995. Towards 3-D model-based tracking and recognition of human movement: A multi-view approach. In Proceedings of the International Workshop on Automatic Face and Gesture recognition. Citeseer, 272--277.Google Scholar
- A. Heloir, N. Courty, S. Gibet, and F. Multon. 2006. Temporal alignment of communicative gesture sequences. Computer Animation and Virtual Worlds 17, 3--4, 347--357. Google ScholarDigital Library
- O. Höner and T. Hermann. 2005. Listen to the ball! Sonification-based sport games for people with visual impairment. In Proceedings of the 15th International Symposium Adapted Physical Activity.Google Scholar
- M. Isard and A. Blake. 1998. Condensation conditional density propagation for visual tracking. International Journal of Computer Vision 29, 1, 5--28. Google ScholarDigital Library
- S. Jordà. 2008. On stage: The reactable and other musical tangibles go real. International Journal of Arts and Technology 1, 3, 268--287.Google ScholarCross Ref
- A. Licsár and T. Szirányi. 2005. User-adaptive hand gesture recognition system with interactive training. Image and Vision Computing 23, 12, 1102--1114. Google ScholarDigital Library
- J. Liu, L. Zhong, J. Wickramasuriya, and V. Vasudevan. 2009. Uwave: Accelerometer-based personalized gesture recognition and its applications. Pervasive and Mobile Computing 5, 6, 657--675. Google ScholarDigital Library
- D. Merrill and J. A. Paradiso. 2005. Personalization, expressivity, and learnability of an implicit mapping strategy for physical interfaces. In Proceedings of the CHI Conference on Human Factors in Computing Systems, Extended Abstracts. 2152--2161.Google Scholar
- S. Mitra and T. Acharya. 2007. Gesture recognition: A survey. IEEE Transactions on Systems, Man, and Cybernetics, Part C: Applications and Reviews 37, 3, 311--324. Google ScholarDigital Library
- L. R. Rabiner. 1989. A tutorial on hidden Markov models and selected applications in speech recognition. Proceedings of the IEEE, 257--286.Google ScholarCross Ref
- N. Rasamimanana and F. Bevilacqua. 2012. Urban musical game. In Proceedings of the 2012 Annual Conference on Human Factors in Computing Systems (CHI'12).Google Scholar
- N. Rasamimanana, F. Bevilacqua, N. Schnell, F. Guedy, E. Flety, C. Maestracci, B. Zamborlin, J. L. Frechin, and U. Petrevski. 2011. Modular musical objects towards embodied control of digital music. In Proceedings of the 5th International Conference on Tangible, Embedded, and Embodied Interaction. ACM, New York, NY, 9--12. Google ScholarDigital Library
- D. Rocchesso, P. Polotti, and S. Delle Monache. 2009. Designing continuous sonic interaction. International Journal of Design 3, 3.Google Scholar
- D. Rubine. 1991. Specifying gestures by example. In Proceedings of the 18th Annual Conference on Computer Graphics and Interactive Techniques. ACM, New York, NY, 329--337. Google ScholarDigital Library
- C. Shan, T. Tan, and Y. Wei. 2007. Real-time hand tracking using a mean shift embedded particle filter. Pattern Recognition 40, 7, 1958--1970. Google ScholarDigital Library
- B. Verplank, C. Sapp, and M. Mathews. 2001. A course on controllers. In Proceedings of the Workshop at the ACM Conference on Computer-Human Interaction (CHI'2001) on New Interfaces for Musical Expression (NIME). Google ScholarDigital Library
- Y. Visell and J. Cooperstock. 2007. Enabling gestural interaction by means of tracking dynamical systems models and assistive feedback. In IEEE International Conference on Systems, Man and Cybernetics. IEEE, 3373--3378.Google Scholar
- P. Viviani and T. Flash. 1995. Minimum-jerk, two-thirds power law, and isochrony: Converging approaches to movement planning. Journal of Experimental Psychology: Human Perception and Performance 21, 1, 32.Google ScholarCross Ref
- Z. Wei, T. Tao, D. ZhuoShu, and E. Zio. 2013. A dynamic particle filter-support vector regression method for reliability prediction. Reliability Engineering & System Safety 119, 109--116.Google ScholarCross Ref
- D. Weinland, R. Ronfard, and E. Boyer. 2011. A survey of vision-based methods for action representation, segmentation and recognition. Computer Vision and Image Understanding 115, 2, 224--241. Google ScholarDigital Library
- A. D. Wilson. 2000. Adaptive Models for Gesture Recognition. Ph.D. Dissertation. Massachusetts Institute of Technology, Cambridge MA. Google ScholarDigital Library
- A. D. Wilson and A. F. Bobick. 1999. Parametric hidden Markov models for gesture recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence 21, 9, 884--900. Google ScholarDigital Library
- A. D. Wilson and A. F. Bobick. 2000. Realtime online adaptive gesture recognition. In Proceedings of the 15th International Conference on Pattern Recognition (ICPR), Vol. 1. IEEE, 270--275. Google ScholarDigital Library
- J. O. Wobbrock, A. D. Wilson, and Y. Li. 2007. Gestures without libraries, toolkits or training: A $1 recognizer for user interface prototypes. In Proceedings of the 20th Annual ACM Symposium on User Interface Software and Technology. ACM, New York, NY, 159--168. Google ScholarDigital Library
- Y. Yacoob and M. J. Black. 1998. Parameterized modeling and recognition of activities. In Proceedings of the 6th International Conference on Computer Vision. IEEE, 120--127. Google ScholarDigital Library
- B. Zamborlin, F. Bevilacqua, M. Gillies, and M. d'Inverno. 2014. Fluid gesture interaction design: Applications of continuous recognition for the design of modern gestural interfaces. ACM Transactions on Interactive Intelligent Systems (TiiS) 3, 4, 22. Google ScholarDigital Library
- S. K. Zhou, R. Chellappa, and B. Moghaddam. 2004. Visual tracking and recognition using appearance-adaptive models in particle filters. IEEE Transactions on Image Processing 13, 11, 1491--1506. Google ScholarDigital Library
Index Terms
- Adaptive Gesture Recognition with Variation Estimation for Interactive Systems
Recommendations
Multi-scenario gesture recognition using Kinect
CGAMES '12: Proceedings of the 2012 17th International Conference on Computer Games: AI, Animation, Mobile, Interactive Multimedia, Educational & Serious Games (CGAMES)Hand gesture recognition (HGR) is an important research topic because some situations require silent communication with sign languages. Computational HGR systems assist silent communication, and help people learn a sign language. In this article, a ...
Fundamentals of Gesture Production, Recognition, and Analysis
CHI EA '17: Proceedings of the 2017 CHI Conference Extended Abstracts on Human Factors in Computing SystemsThis course will introduce participants to practical aspects of human gesture production, gesture recognition, and gesture analysis. Notions such as gesture training sets, user-dependent and user-independent training, and variability of gesture ...
Designing, Engineering, and Evaluating Gesture User Interfaces
CHI EA '18: Extended Abstracts of the 2018 CHI Conference on Human Factors in Computing SystemsThis course will introduce participants to the three main stages of the development life cycle of gesture-based interactions: (ul) how to design a gesture user interface (UI) by carefully considering key aspects, such as gesture recognition techniques, ...
Comments