ABSTRACT
We evaluate and demonstrate finger, hand, and posture identification as keyboard shortcuts. By detecting the hand and finger used to press a key, and open or closed hand postures, a key press can have multiple command mappings. A formative study reveals performance and preference patterns when using different fingers and postures to press a key. The results are used to develop a computer vision algorithm to identify fingers and hands on a keyboard captured by a built-in lap top camera and reflector. This algorithm is built into a background service to enable system-wide finger-aware shortcut keys in any application. A controlled experiment uses the service to compare the performance of Finger-Aware Shortcuts with existing methods. The results show Finger-Aware Shortcuts are comparable with a common class of shortcuts using multiple modifier keys. Finally, application demonstrations illustrate different use cases and mappings for Finger-Aware Shortcuts and extend the idea to two-handed key presses, continuous parameter control, and menu selection.
Supplemental Material
- Gilles Bailly, Thomas Pietrzak, Jonathan Deber, and Daniel J. Wigdor. 2013. Métamorphe: Augmenting Hotkey Usage with Actuated Keys. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '13). 563-572. Google ScholarDigital Library
- Ruth Ben'ary. 1989. Touch Typing in Ten Lessons (revised edition ed.). Perigee Books, NY, NY, USA.Google Scholar
- Florian Block, Hans Gellersen, and Nicolas Villar. 2010. Touch-Display Keyboards: Transforming Keyboards into Interactive Surfaces. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '10). 1145-1154. Google ScholarDigital Library
- Géry Casiez, Nicolas Roussel, and Daniel Vogel. 2012. 1 C Filter: A Simple Speed-based Low-pass Filter for Noisy Input in Interactive Systems. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '12). 2527-2530. Google ScholarDigital Library
- Paul H. Dietz, Benjamin Eidelson, Jonathan Westhues, and Steven Bathiche. 2009. A Practical Pressure Sensitive Computer Keyboard. In Proceedings of the 22nd Annual ACM Symposium on User Interface Software and Technology (UIST '09). 55-58. Google ScholarDigital Library
- Wilbert O. Galitz. 2007. The Essential Guide to User Interface Design: An Introduction to GUI Design Principles and Techniques (3, illustrated ed.). John Wiley & Sons, NY, NY, USA. Google ScholarDigital Library
- Alix Goguey, Géry Casiez, Thomas Pietrzak, Daniel Vogel, and Nicolas Roussel. 2014. Adoiraccourcix: Multi-touch Command Selection Using Finger Identification. In Proceedings of the 26th Conference on L'Interaction Homme-Machine (IHM '14). 28-37. Google ScholarDigital Library
- Alix Goguey, Mathieu Nancel, Géry Casiez, and Daniel Vogel. 2016. The Performance and Preference of Different Fingers and Chords for Pointing, Dragging, and Object Transformation. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '16). 12p. Google ScholarDigital Library
- Tovi Grossman, Pierre Dragicevic, and Ravin Balakrishnan. 2007. Strategies for Accelerating On-line Learning of Hotkeys. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '07). 1591-1600. Google ScholarDigital Library
- Robert J. K. Jacob, Audrey Girouard, Leanne M. Hirshfield, Michael S. Horn, Orit Shaer, Erin Treacy Solovey, and Jamie Zigelbaum. 2008. Reality-Based Interaction: A Framework for Post-WIMP Interfaces. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '08). 201-210. Google ScholarDigital Library
- John Karat, James E. McDonald, and Matt Anderson. 1986. A Comparison of Menu Selection Techniques: Touch Panel, Mouse and Keyboard. International Journal of Man-Machine Studies 25, 1 (July 1986), 73-88. Google ScholarDigital Library
- David Kim, Shahram Izadi, Jakub Dostal, Christoph Rhemann, Cem Keskin, Christopher Zach, Jamie Shotton, Timothy Large, Steven Bathiche, Matthias Niessner, D. Alex Butler, Sean Fanello, and Vivek Pradeep. 2014. RetroDepth: 3D Silhouette Sensing for High-precision Input on and Above Physical Surfaces. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '14). 1377-1386. Google ScholarDigital Library
- Donald E. Knuth and Andrew Binstock. 2008. Interview with Donald Knuth. (25 Apr 2008). Retrieved Sep 20, 2015 from http://www.informit.com/articles/article.aspx?p=1193856.Google Scholar
- Brian Krisler and Richard Alterman. 2008. Training Towards Mastery: Overcoming the Active User Paradox. In Proceedings of the 5th Nordic Conference on Human-computer Interaction: Building Bridges (NordiCHI '08). 239-248. Google ScholarDigital Library
- Gordon Kurtenbach and William Buxton. 1994. User Learning and Performance with Marking Menus. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '94). 258-264. Google ScholarDigital Library
- David M. Lane, H. Albert Napier, S. Camille Peres, and Anikó Sándor. 2005. Hidden Costs of Graphical User Interfaces: Failure to Make the Transition from Menus and Icon Toolbars to Keyboard Shortcuts. International Journal of Human-Computer Interaction 18, 2 (May 2005), 133-144.Google ScholarCross Ref
- Jinha Lee, Alex Olwal, Hiroshi Ishii, and Cati Boulanger. 2013. SpaceTop: Integrating 2D and Spatial 3D Interactions in a See-through Desktop Environment. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '13). 189-192. Google ScholarDigital Library
- Sylvain Malacria, Gilles Bailly, Joel Harrison, Andy Cockburn, and Carl Gutwin. 2013a. Promoting Hotkey Use through Rehearsal with ExposeHK. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '13). 573-582. Google ScholarDigital Library
- Sylvain Malacria, Joey Scarr, Andy Cockburn, Carl Gutwin, and Tovi Grossman. 2013b. Skillometers: Reflective Widgets That Motivate and Help Users to Improve Performance. In Proceedings of the 26th Annual ACM Symposium on User Interface Software and Technology (UIST '13). 321-330. Google ScholarDigital Library
- Shahzad Malik and Joe Laszlo. 2004. Visual Touchpad: A Two-handed Gestural Input Device. In Proceedings of the 6th International Conference on Multimodal Interfaces (ICMI '04). 289-296. Google ScholarDigital Library
- Hugh McLoone, Ken Hinckley, and Edward Cutrell. 2003. Bimanual Interaction on the Microsoft Office Keyboard. In Proceedings of IFIP TC 13 International Conference on Human-Computer Interaction (INTERACT '03). 49-56.Google Scholar
- Craig S. Miller, Svetlin Denkov, and Richard C. Omanson. 2011. Categorization Costs for Hierarchical Keyboard Commands. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '11). 2765-2768. Google ScholarDigital Library
- Thomas A. Mysliwiec. 1994. FingerMouse: A Freehand Computer Pointing Interface. In Proceedings of IEEE International Conference on Automatic Face and Gesture Recognition (FG '95). 372-377.Google Scholar
- Michael Ortega and Laurence Nigay. 2009. AirMouse: Finger Gesture for 2D and 3D Interaction. In Proceedings of the 12th IFIP TC 13 International Conference on Human-Computer Interaction (INTERACT '09). 214-227. Google ScholarDigital Library
- S. Camille Peres, Michael D. Fleetwood, Minmin Yang, Franklin P. Tamborello, and Danielle Paige Smith. 2005. Pros, Cons, and Changing Behavior: An Application in the Use of the Keyboard to Issue Commands. In Proceedings of the Human Factors and Ergonomics Society Annual Meeting (HFES '05), Vol. 49. 637-641.Google ScholarCross Ref
- Thomas Pietrzak, Sylvain Malacria, and Gilles Bailly. 2014. CtrlMouse et TouchCtrl: Duplicating Mode Delimiters on the Mouse. In Proceedings of the 26th Conference on L'Interaction Homme-Machine (IHM '14). 38-47. Google ScholarDigital Library
- Jun Rekimoto, Takaaki Ishizawa, Carsten Schwesig, and Haruo Oba. 2003. PreSense: Interaction Techniques for Finger Sensing Input Devices. In Proceedings of the 16th Annual ACM Symposium on User Interface Software and Technology (UIST '03). 203-212. Google ScholarDigital Library
- Atsushi Sugiura and Yoshiyuki Koseki. 1998. A User Interface Using Fingerprint Recognition: Holding Commands and Data Objects on Fingers. In Proceedings of the 11th Annual ACM Symposium on User Interface Software and Technology (UIST '98). 71-79. Google ScholarDigital Library
- Susanne Tak, Piet Westendorp, and Iris van Rooij. 2013. Satisficing and the Use of Keyboard Shortcuts: Being Good Enough Is Enough? Interacting with Computers 25, 5 (Sept. 2013), 404-416.Google ScholarCross Ref
- Stuart Taylor, Cem Keskin, Otmar Hilliges, Shahram Izadi, and John Helmes. 2014. Type-Hover-Swipe in 96 Bytes: A Motion Sensing Mechanical Keyboard. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '14). 1695-1704. Google ScholarDigital Library
- Jingtao Wang and John Canny. 2004. FingerSense: Augmenting Expressiveness to Physical Pushing Button by Fingertip Identification. In CHI '04 Extended Abstracts on Human Factors in Computing Systems (CHI EA '04). 1267-1270. Google ScholarDigital Library
- Malte Weiss, Julie Wagner, Yvonne Jansen, Roger Jennings, Ramsin Khoshabeh, James D. Hollan, and Jan Borchers. 2009. SLAP Widgets: Bridging the Gap Between Virtual and Physical Controls on Tabletops. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '09). 481-490. Google ScholarDigital Library
- Andrew D. Wilson. 2006. Robust Computer Vision-based Detection of Pinching for One and Two-handed Gesture Input. In Proceedings of the 19th Annual ACM Symposium on User Interface Software and Technology (UIST '06). 255-258. Google ScholarDigital Library
- Andrew D. Wilson and Edward Cutrell. 2005. FlowMouse: A Computer Vision-Based Pointing and Gesture Input Device. In Proceedings of IFIP TC 13 International Conference on Human-Computer Interaction (INTERACT '05). 565-578. Google ScholarDigital Library
- Jacob O. Wobbrock, Leah Findlater, Darren Gergle, and James J. Higgins. 2011. The Aligned Rank Transform for Nonparametric Factorial Analyses Using Only Anova Procedures. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '11). 143-146. Google ScholarDigital Library
- Jacob O. Wobbrock, Andrew D. Wilson, and Yang Li. 2007. Gestures without Libraries, Toolkits or Training: A $1 Recognizer for User Interface Prototypes. In Proceedings of the 20th Annual ACM Symposium on User Interface Software and Technology (UIST '07). 159-168. Google ScholarDigital Library
- Erdem Yoruk, Ender Konukoglu, Bulent Sankur, and Jérôme Darbon. 2006. Shape-based Hand Recognition. Image Processing, IEEE Transactions on 15, 7 (July 2006), 1803-1815. Google ScholarDigital Library
- Haimo Zhang and Yang Li. 2014. GestKeyboard: Enabling Gesture-based Interaction on Ordinary Physical Keyboard. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '14). 1675-1684. Google ScholarDigital Library
Index Terms
- Finger-Aware Shortcuts
Recommendations
FingerArc and FingerChord: Supporting Novice to Expert Transitions with Guided Finger-Aware Shortcuts
UIST '18: Proceedings of the 31st Annual ACM Symposium on User Interface Software and TechnologyKeyboard shortcuts can be more efficient than graphical input, but they are underused by most users. To alleviate this, we present "Guided Finger-Aware Shortcuts" to reduce the gulf between graphical input and shortcut activation. The interaction ...
Finger identification and hand gesture recognition techniques for natural user interface
APCHI '13: Proceedings of the 11th Asia Pacific Conference on Computer Human InteractionThe natural user interface using hand gesture have been popular field in Human-Computer-Interaction(HCI). Many research papers have been proposed in this field. They proposed vision-based, glove-based and depth-based approach for hand gesture ...
Finger controller: natural user interaction using finger gestures
HCI'13: Proceedings of the 15th international conference on Human-Computer Interaction: interaction modalities and techniques - Volume Part IVWe present a new natural user interaction technique using finger gesture recognition and finger identification with Kinect depth data. We developed a gesture version drawing, multi-touch and mapping on 3d space interactions. We implemented three type ...
Comments