skip to main content
10.1145/2858036.2858355acmconferencesArticle/Chapter ViewAbstractPublication PageschiConference Proceedingsconference-collections
research-article

Finger-Aware Shortcuts

Published:07 May 2016Publication History

ABSTRACT

We evaluate and demonstrate finger, hand, and posture identification as keyboard shortcuts. By detecting the hand and finger used to press a key, and open or closed hand postures, a key press can have multiple command mappings. A formative study reveals performance and preference patterns when using different fingers and postures to press a key. The results are used to develop a computer vision algorithm to identify fingers and hands on a keyboard captured by a built-in lap top camera and reflector. This algorithm is built into a background service to enable system-wide finger-aware shortcut keys in any application. A controlled experiment uses the service to compare the performance of Finger-Aware Shortcuts with existing methods. The results show Finger-Aware Shortcuts are comparable with a common class of shortcuts using multiple modifier keys. Finally, application demonstrations illustrate different use cases and mappings for Finger-Aware Shortcuts and extend the idea to two-handed key presses, continuous parameter control, and menu selection.

Skip Supplemental Material Section

Supplemental Material

p4274-zheng.mp4
suppl.mov

Supplemental video

References

  1. Gilles Bailly, Thomas Pietrzak, Jonathan Deber, and Daniel J. Wigdor. 2013. Métamorphe: Augmenting Hotkey Usage with Actuated Keys. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '13). 563-572. Google ScholarGoogle ScholarDigital LibraryDigital Library
  2. Ruth Ben'ary. 1989. Touch Typing in Ten Lessons (revised edition ed.). Perigee Books, NY, NY, USA.Google ScholarGoogle Scholar
  3. Florian Block, Hans Gellersen, and Nicolas Villar. 2010. Touch-Display Keyboards: Transforming Keyboards into Interactive Surfaces. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '10). 1145-1154. Google ScholarGoogle ScholarDigital LibraryDigital Library
  4. Géry Casiez, Nicolas Roussel, and Daniel Vogel. 2012. 1 C Filter: A Simple Speed-based Low-pass Filter for Noisy Input in Interactive Systems. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '12). 2527-2530. Google ScholarGoogle ScholarDigital LibraryDigital Library
  5. Paul H. Dietz, Benjamin Eidelson, Jonathan Westhues, and Steven Bathiche. 2009. A Practical Pressure Sensitive Computer Keyboard. In Proceedings of the 22nd Annual ACM Symposium on User Interface Software and Technology (UIST '09). 55-58. Google ScholarGoogle ScholarDigital LibraryDigital Library
  6. Wilbert O. Galitz. 2007. The Essential Guide to User Interface Design: An Introduction to GUI Design Principles and Techniques (3, illustrated ed.). John Wiley & Sons, NY, NY, USA. Google ScholarGoogle ScholarDigital LibraryDigital Library
  7. Alix Goguey, Géry Casiez, Thomas Pietrzak, Daniel Vogel, and Nicolas Roussel. 2014. Adoiraccourcix: Multi-touch Command Selection Using Finger Identification. In Proceedings of the 26th Conference on L'Interaction Homme-Machine (IHM '14). 28-37. Google ScholarGoogle ScholarDigital LibraryDigital Library
  8. Alix Goguey, Mathieu Nancel, Géry Casiez, and Daniel Vogel. 2016. The Performance and Preference of Different Fingers and Chords for Pointing, Dragging, and Object Transformation. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '16). 12p. Google ScholarGoogle ScholarDigital LibraryDigital Library
  9. Tovi Grossman, Pierre Dragicevic, and Ravin Balakrishnan. 2007. Strategies for Accelerating On-line Learning of Hotkeys. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '07). 1591-1600. Google ScholarGoogle ScholarDigital LibraryDigital Library
  10. Robert J. K. Jacob, Audrey Girouard, Leanne M. Hirshfield, Michael S. Horn, Orit Shaer, Erin Treacy Solovey, and Jamie Zigelbaum. 2008. Reality-Based Interaction: A Framework for Post-WIMP Interfaces. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '08). 201-210. Google ScholarGoogle ScholarDigital LibraryDigital Library
  11. John Karat, James E. McDonald, and Matt Anderson. 1986. A Comparison of Menu Selection Techniques: Touch Panel, Mouse and Keyboard. International Journal of Man-Machine Studies 25, 1 (July 1986), 73-88. Google ScholarGoogle ScholarDigital LibraryDigital Library
  12. David Kim, Shahram Izadi, Jakub Dostal, Christoph Rhemann, Cem Keskin, Christopher Zach, Jamie Shotton, Timothy Large, Steven Bathiche, Matthias Niessner, D. Alex Butler, Sean Fanello, and Vivek Pradeep. 2014. RetroDepth: 3D Silhouette Sensing for High-precision Input on and Above Physical Surfaces. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '14). 1377-1386. Google ScholarGoogle ScholarDigital LibraryDigital Library
  13. Donald E. Knuth and Andrew Binstock. 2008. Interview with Donald Knuth. (25 Apr 2008). Retrieved Sep 20, 2015 from http://www.informit.com/articles/article.aspx?p=1193856.Google ScholarGoogle Scholar
  14. Brian Krisler and Richard Alterman. 2008. Training Towards Mastery: Overcoming the Active User Paradox. In Proceedings of the 5th Nordic Conference on Human-computer Interaction: Building Bridges (NordiCHI '08). 239-248. Google ScholarGoogle ScholarDigital LibraryDigital Library
  15. Gordon Kurtenbach and William Buxton. 1994. User Learning and Performance with Marking Menus. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '94). 258-264. Google ScholarGoogle ScholarDigital LibraryDigital Library
  16. David M. Lane, H. Albert Napier, S. Camille Peres, and Anikó Sándor. 2005. Hidden Costs of Graphical User Interfaces: Failure to Make the Transition from Menus and Icon Toolbars to Keyboard Shortcuts. International Journal of Human-Computer Interaction 18, 2 (May 2005), 133-144.Google ScholarGoogle ScholarCross RefCross Ref
  17. Jinha Lee, Alex Olwal, Hiroshi Ishii, and Cati Boulanger. 2013. SpaceTop: Integrating 2D and Spatial 3D Interactions in a See-through Desktop Environment. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '13). 189-192. Google ScholarGoogle ScholarDigital LibraryDigital Library
  18. Sylvain Malacria, Gilles Bailly, Joel Harrison, Andy Cockburn, and Carl Gutwin. 2013a. Promoting Hotkey Use through Rehearsal with ExposeHK. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '13). 573-582. Google ScholarGoogle ScholarDigital LibraryDigital Library
  19. Sylvain Malacria, Joey Scarr, Andy Cockburn, Carl Gutwin, and Tovi Grossman. 2013b. Skillometers: Reflective Widgets That Motivate and Help Users to Improve Performance. In Proceedings of the 26th Annual ACM Symposium on User Interface Software and Technology (UIST '13). 321-330. Google ScholarGoogle ScholarDigital LibraryDigital Library
  20. Shahzad Malik and Joe Laszlo. 2004. Visual Touchpad: A Two-handed Gestural Input Device. In Proceedings of the 6th International Conference on Multimodal Interfaces (ICMI '04). 289-296. Google ScholarGoogle ScholarDigital LibraryDigital Library
  21. Hugh McLoone, Ken Hinckley, and Edward Cutrell. 2003. Bimanual Interaction on the Microsoft Office Keyboard. In Proceedings of IFIP TC 13 International Conference on Human-Computer Interaction (INTERACT '03). 49-56.Google ScholarGoogle Scholar
  22. Craig S. Miller, Svetlin Denkov, and Richard C. Omanson. 2011. Categorization Costs for Hierarchical Keyboard Commands. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '11). 2765-2768. Google ScholarGoogle ScholarDigital LibraryDigital Library
  23. Thomas A. Mysliwiec. 1994. FingerMouse: A Freehand Computer Pointing Interface. In Proceedings of IEEE International Conference on Automatic Face and Gesture Recognition (FG '95). 372-377.Google ScholarGoogle Scholar
  24. Michael Ortega and Laurence Nigay. 2009. AirMouse: Finger Gesture for 2D and 3D Interaction. In Proceedings of the 12th IFIP TC 13 International Conference on Human-Computer Interaction (INTERACT '09). 214-227. Google ScholarGoogle ScholarDigital LibraryDigital Library
  25. S. Camille Peres, Michael D. Fleetwood, Minmin Yang, Franklin P. Tamborello, and Danielle Paige Smith. 2005. Pros, Cons, and Changing Behavior: An Application in the Use of the Keyboard to Issue Commands. In Proceedings of the Human Factors and Ergonomics Society Annual Meeting (HFES '05), Vol. 49. 637-641.Google ScholarGoogle ScholarCross RefCross Ref
  26. Thomas Pietrzak, Sylvain Malacria, and Gilles Bailly. 2014. CtrlMouse et TouchCtrl: Duplicating Mode Delimiters on the Mouse. In Proceedings of the 26th Conference on L'Interaction Homme-Machine (IHM '14). 38-47. Google ScholarGoogle ScholarDigital LibraryDigital Library
  27. Jun Rekimoto, Takaaki Ishizawa, Carsten Schwesig, and Haruo Oba. 2003. PreSense: Interaction Techniques for Finger Sensing Input Devices. In Proceedings of the 16th Annual ACM Symposium on User Interface Software and Technology (UIST '03). 203-212. Google ScholarGoogle ScholarDigital LibraryDigital Library
  28. Atsushi Sugiura and Yoshiyuki Koseki. 1998. A User Interface Using Fingerprint Recognition: Holding Commands and Data Objects on Fingers. In Proceedings of the 11th Annual ACM Symposium on User Interface Software and Technology (UIST '98). 71-79. Google ScholarGoogle ScholarDigital LibraryDigital Library
  29. Susanne Tak, Piet Westendorp, and Iris van Rooij. 2013. Satisficing and the Use of Keyboard Shortcuts: Being Good Enough Is Enough? Interacting with Computers 25, 5 (Sept. 2013), 404-416.Google ScholarGoogle ScholarCross RefCross Ref
  30. Stuart Taylor, Cem Keskin, Otmar Hilliges, Shahram Izadi, and John Helmes. 2014. Type-Hover-Swipe in 96 Bytes: A Motion Sensing Mechanical Keyboard. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '14). 1695-1704. Google ScholarGoogle ScholarDigital LibraryDigital Library
  31. Jingtao Wang and John Canny. 2004. FingerSense: Augmenting Expressiveness to Physical Pushing Button by Fingertip Identification. In CHI '04 Extended Abstracts on Human Factors in Computing Systems (CHI EA '04). 1267-1270. Google ScholarGoogle ScholarDigital LibraryDigital Library
  32. Malte Weiss, Julie Wagner, Yvonne Jansen, Roger Jennings, Ramsin Khoshabeh, James D. Hollan, and Jan Borchers. 2009. SLAP Widgets: Bridging the Gap Between Virtual and Physical Controls on Tabletops. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '09). 481-490. Google ScholarGoogle ScholarDigital LibraryDigital Library
  33. Andrew D. Wilson. 2006. Robust Computer Vision-based Detection of Pinching for One and Two-handed Gesture Input. In Proceedings of the 19th Annual ACM Symposium on User Interface Software and Technology (UIST '06). 255-258. Google ScholarGoogle ScholarDigital LibraryDigital Library
  34. Andrew D. Wilson and Edward Cutrell. 2005. FlowMouse: A Computer Vision-Based Pointing and Gesture Input Device. In Proceedings of IFIP TC 13 International Conference on Human-Computer Interaction (INTERACT '05). 565-578. Google ScholarGoogle ScholarDigital LibraryDigital Library
  35. Jacob O. Wobbrock, Leah Findlater, Darren Gergle, and James J. Higgins. 2011. The Aligned Rank Transform for Nonparametric Factorial Analyses Using Only Anova Procedures. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '11). 143-146. Google ScholarGoogle ScholarDigital LibraryDigital Library
  36. Jacob O. Wobbrock, Andrew D. Wilson, and Yang Li. 2007. Gestures without Libraries, Toolkits or Training: A $1 Recognizer for User Interface Prototypes. In Proceedings of the 20th Annual ACM Symposium on User Interface Software and Technology (UIST '07). 159-168. Google ScholarGoogle ScholarDigital LibraryDigital Library
  37. Erdem Yoruk, Ender Konukoglu, Bulent Sankur, and Jérôme Darbon. 2006. Shape-based Hand Recognition. Image Processing, IEEE Transactions on 15, 7 (July 2006), 1803-1815. Google ScholarGoogle ScholarDigital LibraryDigital Library
  38. Haimo Zhang and Yang Li. 2014. GestKeyboard: Enabling Gesture-based Interaction on Ordinary Physical Keyboard. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '14). 1675-1684. Google ScholarGoogle ScholarDigital LibraryDigital Library

Index Terms

  1. Finger-Aware Shortcuts

    Recommendations

    Comments

    Login options

    Check if you have access through your login credentials or your institution to get full access on this article.

    Sign in
    • Published in

      cover image ACM Conferences
      CHI '16: Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems
      May 2016
      6108 pages
      ISBN:9781450333627
      DOI:10.1145/2858036

      Copyright © 2016 ACM

      Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

      Publisher

      Association for Computing Machinery

      New York, NY, United States

      Publication History

      • Published: 7 May 2016

      Permissions

      Request permissions about this article.

      Request Permissions

      Check for updates

      Qualifiers

      • research-article

      Acceptance Rates

      CHI '16 Paper Acceptance Rate565of2,435submissions,23%Overall Acceptance Rate6,199of26,314submissions,24%

      Upcoming Conference

      CHI '24
      CHI Conference on Human Factors in Computing Systems
      May 11 - 16, 2024
      Honolulu , HI , USA

    PDF Format

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader