AudioTouch: Minimally Invasive Sensing of Micro-Gestures via Active Bio-Acoustic Sensing

ABSTRACT

We present AudioTouch, a minimally invasive approach for sensing micro-gestures using active bio-acoustic sensing. It only requires attaching two piezo-electric elements, acting as a surface mounted speaker and microphone, on the back of the hand. It does not require any instrumentation on the palm or fingers; therefore, it does not encumber interactions with physical objects. The signal is rich enough to detect small differences in micro-gestures with standard machine-learning classifiers. This approach also allows for the discrimination of different levels of touch-force, further expanding the interaction vocabulary. We conducted four experiments to evaluate the performances of AudioTouch: a user study for measuring the gesture recognition accuracy, a follow-up study investigating the ability to discriminate different levels of touch-force, an experiment assessing the cross-session robustness, and, a systematic evaluation assessing the effect of sensor placement on the back of the hand.

References

1. Shuhei Aoyama, Buntarou Shizuki, and Jiro Tanaka. 2016. Thumb-Slide: An Interaction Technique for Smartwatches Using a Thumb Slide Movement. In Proceedings of the 2016 CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA '16). ACM, New York, NY, USA, 2403--2409. https://doi.org/10.1145/2851581.2892435
2. Riley Booth and Peter Goldsmith. 2017. Detecting Finger Gestures with a Wrist Worn Piezoelectric Sensor Array. In Proceedings of the 2017 IEEE International Conference on Systems, Man, and Cybernetics (SMC '17). 3665--3670.
3. Liwei Chan, Yi-Ling Chen, Chi-Hao Hsieh, Rong-Hao Liang, and Bing-Yu Chen. 2015. CyclopsRing: Enabling Whole-Hand and Context-aware Interactions through a Fisheye Ring. In Proceedings of the 28th Annual ACM Symposium on User Interface Software and Technology (UIST '15). ACM, New York, NY, USA, 549--556. https://doi.org/10.1145/2807442.2807450
4. CyberGlove Systems Inc. 2009. CyberTouch Glove. http://www. cyberglovesystems.com/cybertouch. Last accessed: 18.03.2017.
5. Artem Dementyev and Joseph A. Paradiso. 2014. WristFlex: Low-power Gesture Input with Wrist-worn Pressure Sensors. In Proceedings of the 27th Annual ACM Symposium on User Interface Software and Technology (UIST '14). ACM, New York, NY, USA, 161--166. https://doi.org/10.1145/2642918.2647396
6. Travis Deyle, Szabolcs Palinko, Erika Shehan Poole, and Thad Starner. 2007. Hambone: A Bio-Acoustic Gesture Interface. In Proceedings of the 11th IEEE International Symposium on Wearable Computers. IEEE, 3--10.
7. Laura Dipietro, Angelo M. Sabatini, and Paolo Dario. 2008. A Survey of Glove-Based Systems and Their Applications. IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews) 38, 4 (July 2008), 461--482. https://doi.org/10.1109/TSMCC.2008.923862
8. Mayank Goel, Jacob Wobbrock, and Shwetak Patel. 2012. GripSense: Using Built-in Sensors to Detect Hand Posture and Pressure on Commodity Mobile Phones. In Proceedings of the 25th Annual ACM Symposium on User Interface Software and Technology (UIST '12). ACM, New York, NY, USA, 545--554. https://doi.org/10.1145/2380116.2380184
9. Jun Gong, Yang Zhang, Xia Zhou, and Xing-Dong Yang. 2017. Pyro: Thumb-Tip Gesture Recognition Using Pyroelectric Infrared Sensing. In Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology (UIST '17). ACM, New York, NY, USA, 553--563. https://doi.org/10.1145/3126594.3126615
10. Mark Hall, Eibe Frank, Geoffrey Holmes, Bernhard Pfahringer, Peter Reutemann, and Ian H. Witten. 2009. The WEKA Data Mining Software: An Update. SIGKDD Explorations Newsletter 11, 1 (2009), 10--18. https://doi.org/10.1145/1656274.1656278
11. Teng Han, Khalad Hasan, Keisuke Nakamura, Randy Gomez, and Pourang Irani. 2017. SoundCraft: Enabling Spatial Interactions on Smartwatches using Hand Generated Acoustics. In Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology (UIST '17). ACM, New York, NY, USA, 579--591. https://doi.org/10.1145/3126594.3126612
12. Chris Harrison, Hrvoje Benko, and Andrew D. Wilson. 2011. Omni-Touch: Wearable Multitouch Interaction Everywhere. In Proceedings of the 24th Annual ACM Symposium on User Interface Software and Technology (UIST '11). ACM, New York, NY, USA, 441--450. https://doi.org/10.1145/2047196.2047255
13. Chris Harrison and Scott E. Hudson. 2008. Scratch Input: Creating Large, Inexpensive, Unpowered and Mobile Finger Input Surfaces. In Proceedings of the 21st Annual ACM Symposium on User Interface Software and Technology (UIST '08). ACM, New York, NY, USA, 205--208. https://doi.org/10.1145/1449715.1449747
14. Chris Harrison, Desney Tan, and Dan Morris. 2010. Skinput: Appropriating the Body As an Input Surface. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '10). ACM, New York, NY, USA, 453--462. https://doi.org/10.1145/1753326.1753394
15. Chris Harrison, Robert Xiao, and Scott Hudson. 2012. Acoustic Barcodes: Passive, Durable and Inexpensive Notched Identification Tags. In Proceedings of the 25th Annual ACM Symposium on User Interface Software and Technology (UIST '12). ACM, New York, NY, USA, 563--568. https://doi.org/10.1145/2380116.2380187
16. Seongkook Heo and Geehyuk Lee. 2011. Force Gestures: Augmenting Touch Screen Gestures with Normal and Tangential Forces. In Proceedings of the 24th Annual ACM Symposium on User Interface Software and Technology (UIST '11). ACM, New York, NY, USA, 621--626. https://doi.org/10.1145/2047196.2047278
17. Seongkook Heo and Geehyuk Lee. 2011. ForceTap: Extending the Input Vocabulary of Mobile Touch Screens by adding Tap Gestures. In Proceedings of the 13th International Conference on Human Computer Interaction with Mobile Devices and Services (MobileHCI '11). ACM, New York, NY, USA, 113--122. https://doi.org/10.1145/2037373.2037393
18. Donny Huang, Xiaoyi Zhang, T. Scott Saponas, James Fogarty, and Shyamnath Gollakota. 2015. Leveraging Dual-Observable Input for Fine-Grained Thumb Interaction using Forearm EMG. In Proceedings of the 28th Annual ACM Symposium on User Interface Software and Technology (UIST '15). ACM, New York, NY, USA, 523--528. https://doi.org/10.1145/2807442.2807506
19. Da-Yuan Huang, Liwei Chan, Shuo Yang, Fan Wang, Rong-Hao Liang, De-Nian Yang, Yi-Ping Hung, and Bing-Yu Chen. 2016. DigitSpace: Designing Thumb-to-Fingers Touch Interfaces for One-Handed and Eyes-Free Interactions. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI '16). ACM, New York, NY, USA, 1526--1537. https://doi.org/10.1145/2858036.2858483
20. Sungjae Hwang, Andrea Bianchi, and Kwangyun Wohn. 2012. MicPen: Pressure-sensitive Pen Interaction using Microphone with Standard Touchscreen. In Proceedings of the 30th International SIGCHI Conference Extended Abstracts on Human Factors in Computing System (CHI EA '12). ACM, New York, NY, USA, 1847--1852. https://doi.org/10.1145/2212776.2223717
21. Sungjae Hwang and Kwang-yun Wohn. 2012. PseudoButton: Enabling Pressure-Sensitive Interaction by Repurposing Microphone on Mobile Device. In Proceedings of the 30th International SIGCHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA '12). ACM, New York, NY, USA, 1565--1570. https://doi.org/10.1145/2212776.2223673
22. Hsin-Liu (Cindy) Kao, Christian Holz, Asta Roseway, Andres Calvo, and Chris Schmandt. 2016. DuoSkin: Rapidly Prototyping On-skin User Interfaces Using Skin-friendly Materials. In Proceedings of the 2016 ACM International Symposium on Wearable Computers (ISWC '16). ACM, New York, NY, USA, 16--23. https://doi.org/10.1145/2971763.2971777
23. Hiroyuki Kato and Kentaro Takemura. 2016. Hand Pose Estimation based on Active Bone-conducted Sound Sensing. In Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing: Adjunct (UbiComp '16). ACM, New York, NY, USA, 109--112. https://doi.org/10.1145/2968219.2971403
24. David Kim, Otmar Hilliges, Shahram Izadi, Alex D. Butler, Jiawen Chen, Iason Oikonomidis, and Patrick Olivier. 2012. Digits: Freehand 3D Interactions Anywhere using a Wrist-worn Gloveless Sensor. In Proceedings of the 25th Annual ACM Symposium on User Interface Software and Technology (UIST '12). ACM, New York, NY, USA, 167--176. https://doi.org/10.1145/2380116.2380139
25. Gierad Laput, Eric Brockmeyer, Scott E. Hudson, and Chris Harrison. 2015. Acoustruments: Passive, Acoustically-Driven, Interactive Controls for Handheld Devices. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (CHI '15). ACM, New York, NY, USA, 2161--2170. https://doi.org/10.1145/2702123.2702414
26. Gierad Laput, Robert Xiao, and Chris Harrison. 2016. ViBand: High-Fidelity Bio-Acoustic Sensing Using Commodity Smartwatch Accelerometers. In Proceedings of the 29th Annual Symposium on User Interface Software and Technology (UIST '16). ACM, New York, NY, USA, 321--333. https://doi.org/10.1145/2984511.2984582
27. LEAP MOTION, INC. 2012. LEAP MOTION. https://www.leapmotion. com. Last accessed: 01.02.2019.
28. Hong Li, Wei Yang, Jianxin Wang, Yang Xu, and Liusheng Huang. 2016. WiFinger: Talk to Your Smart Devices with Finger-grained Gesture. In Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp '16). ACM, New York, NY, USA, 250--261. https://doi.org/10.1145/2971648.2971738
29. Jaime Lien, Nicholas Gillian, M. Emre Karagozler, Patrick Amihood, Carsten Schwesig, Erik Olson, Hakim Raja, and Ivan Poupyrev. 2016. Soli: Ubiquitous Gesture Sensing with Millimeter Wave Radar. ACM Transactions on Graphics 35, 4, Article 142 (July 2016), 19 pages. https://doi.org/10.1145/2897824.2925953
30. Jhe-Wei Lin, Chiuan Wang, Yi Yao Huang, Kuan-Ting Chou, Hsuan-Yu Chen, Wei-Luan Tseng, and Mike Y. Chen. 2015. BackHand: Sensing Hand Gestures via Back of the Hand. In Proceedings of the 28th Annual ACM Symposium on User Interface Software & Technology (UIST '15). ACM, New York, NY, USA, 557--564. https://doi.org/10.1145/2807442.2807462
31. Jess McIntosh, Asier Marzo, and Mike Fraser. 2017. SensIR: Detecting Hand Gestures with a Wearable Bracelet Using Infrared Transmission and Reflection. In Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology (UIST '17). ACM, New York, NY, USA, 593--597. https://doi.org/10.1145/3126594.3126604
32. Jess McIntosh, Asier Marzo, Mike Fraser, and Carol Phillips. 2017. EchoFlex: Hand Gesture Recognition Using Ultrasound Imaging. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (CHI '17). ACM, New York, NY, USA, 1923--1934. https://doi.org/10.1145/3025453.3025807
33. Jess McIntosh, Charlie McNeill, Mike Fraser, Frederic Kerber, Markus Löchtefeld, and Antonio Krüger. 2016. EMPress: Practical Hand Gesture Classification with Wrist-Mounted EMG and Pressure Sensing. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI '16). ACM, New York, NY, USA, 2332--2342. https://doi.org/10.1145/2858036.2858093
34. Adiyan Mujibiya, Xiang Cao, Desney S. Tan, Dan Morris, Shwetak N. Patel, and Jun Rekimoto. 2013. The Sound of Touch: On-body Touch and Gesture Sensing Based on Transdermal Ultrasound Propagation. In Proceedings of the 2013 ACM International Conference on Interactive Tabletops and Surfaces (ITS '13). ACM, New York, NY, USA, 189--198. https://doi.org/10.1145/2512349.2512821
35. NeuroDigital Technologies. 2015. Gloveone Glove. https://www.neurodigital.es/gloveone/. Last accessed: 01.02.2019.
36. North Inc. 2014. Myo. https://www.bynorth.com/about. Last accessed: 01.02.2019.
37. Makoto Ono, Buntarou Shizuki, and Jiro Tanaka. 2013. Touch & Activate: Adding Interactivity to Existing Objects Using Active Acoustic Sensing. In Proceedings of the 26th Annual ACM Symposium on User Interface Software and Technology (UIST '13). ACM, New York, NY, USA, 31--40. https://doi.org/10.1145/2501988.2501989
38. Makoto Ono, Buntarou Shizuki, and Jiro Tanaka. 2015. Sensing Touch Force using Active Acoustic Sensing. In Proceedings of the 9th International Conference on Tangible, Embedded, and Embodied Interaction (TEI '15). ACM, New York, NY, USA, 355--358. https://doi.org/10.1145/2677199.2680585
39. Manuel Prätorius, Dimitar Valkov, Ulrich Burgbacher, and Klaus Hinrichs. 2014. DigiTap: An Eyes-free VR/AR Symbolic Input Device. In Proceedings of the 20th ACM Symposium on Virtual Reality Software and Technology (VRST '14). ACM, New York, NY, USA, 9--18. https://doi.org/10.1145/2671015.2671029
40. Jun Rekimoto. 2001. GestureWrist and GesturePad: Unobtrusive Wearable Interaction Devices. In Proceedings of the International Symposium on Wearable Computers (ISWC' 01).
41. T. Scott Saponas, Desney S. Tan, Dan Morris, Ravin Balakrishnan, Jim Turner, and James A. Landay. 2009. Enabling Always-available Input with Muscle-computer Interfaces. In Proceedings of the 22nd Annual ACM Symposium on User Interface Software and Technology (UIST '09). ACM, New York, NY, USA, 167--176. https://doi.org/10.1145/1622176. 1622208
42. Munehiko Sato, Ivan Poupyrev, and Chris Harrison. 2012. Touché: Enhancing Touch Interaction on Humans, Screens, Liquids, and Everyday Objects. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '12). ACM, New York, NY, USA, 483--492. https://doi.org/10.1145/2207676.2207743
43. Jie Song, Gábor Sörös, Fabrizio Pece, Sean Ryan Fanello, Shahram Izadi, Cem Keskin, and Otmar Hilliges. 2014. In-air Gestures Around Unmodified Mobile Devices. In Proceedings of the 27th Annual ACM Symposium on User Interface Software and Technology (UIST '14). ACM, New York, NY, USA, 319--329. https://doi.org/10.1145/2642918.2647373
44. Srinath Sridhar, Anders Markussen, Antti Oulasvirta, Christian Theobalt, and Sebastian Boring. 2017. WatchSense: On- and Above-Skin Input Sensing Through a Wearable Depth Sensor. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (CHI '17). ACM, New York, NY, USA, 3891--3902. https://doi.org/10.1145/3025453.3026005
45. Ryosuke Takada, Wei Lin, Toshiyuki Ando, Buntarou Shizuki, and Shin Takahashi. 2017. A Technique for Touch Force Sensing using a Waterproof Device's Built-in Barometer. In Proceedings of the 2017 CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA '17). ACM, New York, NY, USA, 2140--2146. https://doi.org/10.1145/3027063.3053130
46. Jonathan Taylor, Lucas Bordeaux, Thomas Cashman, Bob Corish, Cem Keskin, Toby Sharp, Eduardo Soto, David Sweeney, Julien Valentin, Benjamin Luff, Arran Topalian, Erroll Wood, Sameh Khamis, Pushmeet Kohli, Shahram Izadi, Richard Banks, Andrew Fitzgibbon, and Jamie Shotton. 2016. Efficient and Precise Interactive Hand Tracking Through Joint, Continuous Optimization of Pose and Correspondences. ACM Transactions on Graphics 35, 4, Article 143 (July 2016), 12 pages. https://doi.org/10.1145/2897824.2925965
47. Qinglong Wang, Xiangshi Ren, and Xiaoying Sun. 2016. EV-Pen: an Electrovibration Haptic Feedback Pen for Touchscreens. In SIGGRAPH ASIA 2016 Emerging Technologies (SA '16). ACM, New York, NY, USA, Article 8, 2 pages. https://doi.org/10.1145/2988240.2988241
48. Hiroki Watanabe, Tsutomu Terada, and Masahiko Tsukamoto. 2017. Gesture Recognition Method utilizing Ultrasonic Active Acoustic Sensing. Journal of Information Processing 25 (2017), 331--340. https://doi.org/10.2197/ipsjjip.25.331
49. Martin Weigel, Aditya Shekhar Nittala, Alex Olwal, and Jürgen Steimle. 2017. SkinMarks: Enabling Interactions on Body Landmarks Using Conformal Skin Electronics. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (CHI '17). ACM, New York, NY, USA, 3095--3105. https://doi.org/10.1145/3025453.3025704
50. Eric Whitmire, Mohit Jain, Divye Jain, Greg Nelson, Ravi Karkar, Shwetak Patel, and Mayank Goel. 2017. DigiTouch: Reconfigurable Thumb-to-Finger Input and Text Entry on Head-mounted Displays. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 1, 3, Article 113 (Sept. 2017), 21 pages. https://doi.org/10.1145/3130978
51. Pui Chung Wong, Kening Zhu, and Hongbo Fu. 2018. FingerT9: Leveraging Thumb-to-finger Interaction for Same-side-hand Text Entry on Smartwatches. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (CHI '18). ACM, New York, NY, USA, Article 178, 10 pages. https://doi.org/10.1145/3173574.3173752
52. Tomohiro Yokota and Tomoko Hashida. 2016. Hand Gesture and On-body Touch Recognition by Active Acoustic Sensing throughout the Human Body. In Proceedings of the 29th Annual Symposium on User Interface Software and Technology (UIST '16 Adjunct). ACM, New York, NY, USA, 113--115. https://doi.org/10.1145/2984751.2985721
53. Sugiura Yuta, Nakamura Fumihiko, Kawai Wataru, Kikuchi Takashi, and Sugimoto Maki. 2017. Behind The Palm: Hand Gesture Recognition through Measuring Skin Deformation on Back of Hand by using Optical Sensors. In 56th Annual Conference of the Society of Instrument and Control Engineers of Japan (SICE) (SICE '17). 1082--1087. https://doi.org/10.23919/SICE.2017.8105457
54. Cheng Zhang, Xiaoxuan Wang, Anandghan Waghmare, Sumeet Jain, Thomas Ploetz, Omer T. Inan, Thad E. Starner, and Gregory D. Abowd. 2017. FingOrbits: Interaction with Wearables using Synchronized Thumb Movements. In Proceedings of the 2017 ACM International Symposium on Wearable Computers (ISWC '17). ACM, New York, NY, USA, 62--65. https://doi.org/10.1145/3123021.3123041
55. Cheng Zhang, Qiuyue Xue, Anandghan Waghmare, Ruichen Meng, Sumeet Jain, Yizeng Han, Xinyu Li, Kenneth Cunefare, Thomas Ploetz, Thad Starner, Omer Inan, and Gregory D. Abowd. 2018. FingerPing: Recognizing Fine-grained Hand Poses using Active Acoustic On-body Sensing. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (CHI '18). ACM, New York, NY, USA, Article 437, 10 pages. https://doi.org/10.1145/3173574.3174011
56. Yang Zhang and Chris Harrison. 2015. Tomo: Wearable, Low-Cost Electrical Impedance Tomography for Hand Gesture Recognition. In Proceedings of the 28th Annual ACM Symposium on User Interface Software and; Technology (UIST '15). ACM, New York, NY, USA, 167--173. https://doi.org/10.1145/2807442.2807480
57. Yang Zhang, Robert Xiao, and Chris Harrison. 2016. Advancing Hand Gesture Recognition with High Resolution Electrical Impedance Tomography. In Proceedings of the 29th Annual Symposium on User Interface Software and Technology (UIST '16). ACM, New York, NY, USA, 843--850. https://doi.org/10.1145/2984511.2984574
58. Yang Zhang, Junhan Zhou, Gierad Laput, and Chris Harrison. 2016. SkinTrack: Using the Body As an Electrical Waveguide for Continuous Finger Tracking on the Skin. In Proceedings of the 34th SIGCHI Conference on Human Factors in Computing Systems (CHI '16). ACM, New York, NY, USA, 1491--1503. https://doi.org/10.1145/2858036.2858082

Index Terms

• Human-centered computing

  • Human computer interaction (HCI)

    • Interaction techniques

      • Gestural input

  • Ubiquitous and mobile computing

    • Ubiquitous and mobile devices