Hao Lü
ABSTRACT
Multi-touch gestures have become popular on a wide range of touchscreen devices, but the programming of these gestures remains an art. It is time-consuming and error-prone for a developer to handle the complicated touch state transitions that result from multiple fingers and their simultaneous movements. In this paper, we present Gesture Coder, which by learning from a few examples given by the developer automatically generates code that recognizes multi-touch gestures, tracks their state changes and invokes corresponding application actions. Developers can easily test the generated code in Gesture Coder, refine it by adding more examples, and once they are satisfied with its performance integrate the code into their applications. We evaluated our learning algorithm exhaustively with various conditions over a large set of noisy data. Our results show that it is sufficient for rapid prototyping and can be improved with higher quality and more training data. We also evaluated Gesture Coder's usability through a within-subject study in which we asked participants to implement a set of multi-touch interactions with and without Gesture Coder. The results show overwhelmingly that Gesture Coder significantly lowers the threshold of programming multi-touch gestures.
Supplemental Material
- Android. http://www.android.com.Google Scholar
- Anthony, L. and Wobbrock, J.O. A lightweight multistroke recognizer for user interface prototypes. Proc. GI 2010, 245--252. Google ScholarDigital Library
- Appert, C. and Bau, O. Scale detection for a priori gesture recognition. Proc. CHI 2010, 879--882. Google ScholarDigital Library
- Appert, C. and Zhai, S. Using strokes as command shortcuts. Proc. CHI 2009, 2289--2298. Google ScholarDigital Library
- Apple iPod Nano 6th Generation. http://www.apple.com/ipodnano/.Google Scholar
- Ashbrook, D. and Starner, T. MAGIC: a motion gesture design tool. Proc. CHI 2010, 2159--2168. Google ScholarDigital Library
- Brushes. http://www.brushesapp.com/.Google Scholar
- Cypher, A., ed. Watch What I Do: Programming by Demonstration. MIT Press, 1993. Google ScholarDigital Library
- Elias, J.G., Westerman, W.C., and Haggerty, M.M. Multi-Touch Gesture Dictionary. US Patent, 2007.Google Scholar
- Google Nexus. http://www.google.com/nexus/.Google Scholar
- Hartmann, B., Abdulla, L., Mittal, M., and Klemmer, S.R. Authoring sensor-based interactions by demonstration with direct manipulation and pattern recognition. Proc. CHI 2007, 145--154. Google ScholarDigital Library
- Hoste, L. Software engineering abstractions for the multi-touch revolution. Proc. ICSE 2010, 509--510. Google ScholarDigital Library
- Kin, K., Hartmann, B., DeRose, T., and Agrawala, M. Proton: Multitouch Gestures as Regular Expressions. Proc. CHI 2012. Google ScholarDigital Library
- Kurtenbach, G. and Buxton, W. User learning and performance with marking menus. Proc. CHI 1994, 258--264. Google ScholarDigital Library
- Lieberman, H., E. Your Wish Is My Command: Programming by Example. Morgan Kaufmann, 2001.Google ScholarDigital Library
- Long, Jr., A.C. Quill: a gesture design tool for penbased user interfaces. Doctoral Dissertation, University of California, Berkeley, 2001. Google ScholarDigital Library
- Microsoft Surface. http://www.microsoft.com/surface/.Google Scholar
- Motorola Xoom. http://developer.motorola.com/products/xoom/.Google Scholar
- OS X Lion: About Multi-Touch gestures. http://support.apple.com/kb/HT4721.Google Scholar
- Perceptive Pixel Multi-Touch Collaboration Wall. http://www.perceptivepixel.com.Google Scholar
- Quinlan, J.R. C4.5: programs for machine learning. Morgan Kaufmann, 1993. Google ScholarDigital Library
- Rubine, D. Specifying gestures by example. ACM SIGGRAPH Computer Graphics 25, 4 (1991), 329--337. Google ScholarDigital Library
- Scholliers, C., Hoste, L., Signer, B., and De Meuter, W. Midas: a declarative multi-touch interaction framework. Proc. TEI 2011, 49--56. Google ScholarDigital Library
- Shneiderman, B. Direct manipulation: a step beyond programming languages. Computer 16, 8 (1983), 57--69. Google ScholarDigital Library
- Weka. http://www.cs.waikato.ac.nz/ml/weka/.Google Scholar
- Wobbrock, J.O., Wilson, A.D., and Li, Y. Gestures without libraries, toolkits or training: a $1 recognizer for user interface prototypes. Proc. UIST 2007, 159--168. Google ScholarDigital Library
- Zhai, S. and Kristensson, P.-O. Shorthand writing on stylus keyboard. Proc. CHI 2003, 97--104. Google ScholarDigital Library
- iOS developer library. http://developer.apple.com/library/ios/.Google Scholar
Index Terms
- Gesture coder: a tool for programming multi-touch gestures by demonstration
Recommendations
Gestures in the wild: studying multi-touch gesture sequences on interactive tabletop exhibits
CHI '11: Proceedings of the SIGCHI Conference on Human Factors in Computing SystemsIn this paper we describe our findings from a field study that was conducted at the Vancouver Aquarium to investigate how visitors interact with a large interactive table exhibit using multi-touch gestures. Our findings show that the choice and use of ...
Gesture studio: authoring multi-touch interactions through demonstration and declaration
CHI '13: Proceedings of the SIGCHI Conference on Human Factors in Computing SystemsThe prevalence of multi-touch devices opens the space for rich interactions. However, the complexity for creating multi-touch interactions hinders this potential. In this paper, we present Gesture Studio, a tool for creating multi-touch interaction ...
Gesture-based interaction: a new dimension for mobile user interfaces
AVI '12: Proceedings of the International Working Conference on Advanced Visual InterfacesToday, smart phones with touchscreens and sensors are the predominant, fastest growing class of consumer computing devices. However, because these devices are used in diverse situations, and have unique capabilities and form factors, they also raise new ...
Comments