Air+touch: interweaving touch & in-air gestures

ABSTRACT

We present Air+Touch, a new class of interactions that interweave touch events with in-air gestures, offering a unified input modality with expressiveness greater than each input modality alone. We demonstrate how air and touch are highly complementary: touch is used to designate targets and segment in-air gestures, while in-air gestures add expressivity to touch events. For example, a user can draw a circle in the air and tap to trigger a context menu, do a finger 'high jump' between two touches to select a region of text, or drag and in-air 'pigtail' to copy text to the clipboard. Through an observational study, we devised a basic taxonomy of Air+Touch interactions, based on whether the in-air component occurs before, between or after touches. To illustrate the potential of our approach, we built four applications that showcase seven exemplar Air+Touch interactions we created.

References

1. Banerjee, A., Burstyn, J., Girouard, A., and Vertegaal, R. Pointable: an in-air pointing technique to manipulate outof-reach targets on tabletops. In Proc. ITS '11, 11--20.
2. Baudisch, P. and Chu, G. Back-of-device interaction allows creating very small touch devices. In Proc. CHI '09, 1923-
3. Butler, A., Izadi, S., and Hodges, S. SideSight: multi"touch" interaction around small devices. In Proc. UIST '08, 201--204.
4. Buxton W. Chunking and phrasing and the design of human-computer dialogues. In Human-Computer Interaction. Morgan Kaufmann, San Francisco, CA, USA 494--499.
5. Grossman, T., Hinckley, K., Baudisch, P., Agrawala, M., and Balakrishnan, R. Hover widgets: using the tracking state to extend the capabilities of pen-operated devices. In Proc. CHI '06, 861--870.
6. Harrison, C. and Hudson, S.E. Abracadabra: wireless, highprecision, and unpowered finger input for very small mobile devices. In Proc. UIST '09, 121--124.
7. Hilliges, O., Izadi, S., Wilson, A.D., Hodges, S., GarciaMendoza, A. and Butz, A. Interactions in the air: adding further depth to interactive tabletops. In Proc. UIST '09, 139--148.
8. Hinckley, K., Baudisch, P., Ramos, G. and Guimbretiere, F. Design and analysis of delimiters for selection-action pen gesture phrases in Scriboli. In Proc. CHI '05, 451--460.
9. Hinckley, K., Chen, X. and Benko, H. Motion and context sensing techniques for pen Computing. In Proc. GI '13.
10. Hinckley, K. and Song, H. Sensor synaesthesia: touch in motion, and motion in touch. In Proc. CHI '11, 801--810.
11. Hinckley, K., Yatani, K., Pahud, M, Coddington, N., Rodenhouse, J., Wilson, A., Benko, H. and Buxton, B. Pen + touch = new tools. In Proc. UIST '10, 27--36.
12. Hudson, S.E., Harrison, C., Harrison, B.L., and LaMarca, A. Whack gestures: inexact and inattentive interaction with mobile devices. In Proc. TEI '10, 109--113.
13. Jones, B., Sodhi, R., Forsyth, D., Bailey, B., and Maciocci, G. Around device interaction for multiscale navigation. In Proc. MobileHCI '12, 83--92.
14. Ketabdar, H., Yüksel, K.A., and Roshandel, M. MagiTact: Interaction with Mobile Devices Based on Compass (Magnetic) Sensor. In Proc. IUI '10, 413--414.
15. Kratz, S., Rohs, M., Guse, D., Müller, J., Bailly, G. and Nischt, M. PalmSpace: continuous around-device gestures vs. multitouch for 3D rotation tasks on mobile devices. In Proc. AVI '12, 181--188.
16. Kratz, S. and Rohs, M. HoverFlow: expanding the design space of around-device interaction. In Proc. MobileHCI '09, 1--8.
17. Marquardt, N., Jota, R., Greenberg, S. and Jorge, J.A. The continuous interaction space. In Proc. INTERACT '11, 461--476.
18. Niikura, T., Hirobe, Y., Cassinelli, A., Watanabe, Y., Komuro, T. and Ishikawa, M. In-air typing interface for mobile devices with vibration feedback. In Proc. SIGGRAPH '10, 1--1.
19. PMD Technologies. Camboard Nano camera. http://pmdtec.com/products_services/reference_design.php
20. Quinn, P. and Cockburn, A. Zoofing!: faster list selections with pressure-zoom-flick-scrolling. In Proc. OZCHI '09. 185--192.
21. Ramos, G. and Balakrishnan, R. Zliding: fluid zooming and sliding for high precision parameter manipulation. In Proc. UIST '05. 143--152.
22. Samsung. Galaxy S4. http://www.samsung.com/galaxys4
23. Synaptics. 3D Touch. http://blog.synaptics.com/?p=401
24. Wigdor, D., Forlines, C., Baudisch, P., Barnwell, J. and Shen, C. Lucid touch: a see-through mobile device. In Proc. UIST '07, 269--278.
25. Wigdor, D., Leigh, D., Forlines, C., Shipman, S., Barnwell, J., Balakrishnan, R. and Shen, C. Under the table interaction. In Proc. UIST '06, 259--268.
26. Wobbrock, J.O., Wilson, A.D. and Li, Y. Gestures without libraries, toolkits or training. In Proc. UIST '07, 159--168.

Index Terms

• Human-centered computing

  • Human computer interaction (HCI)

    • Interaction devices

      • Touch screens

    • Interaction paradigms

      • Graphical user interfaces