WiFinger: talk to your smart devices with finger-grained gesture

ABSTRACT

In recent literatures, WiFi signals have been widely used to "sense" people's locations and activities. Researchers have exploited the characteristics of wireless signals to "hear" people's talk and "see" keystrokes by human users. Inspired by the excellent work of relevant scholars, we turn to explore the field of human-computer interaction using finger-grained gestures under WiFi environment. In this paper, we present Wi-Finger - the first solution using ubiquitous wireless signals to achieve number text input in WiFi devices. We implement a prototype of WiFinger on a commercial Wi-Fi infrastructure. Our scheme is based on the key intuition that while performing a certain gesture, the fingers of a user move in a unique formation and direction and thus generate a unique pattern in the time series of Channel State Information (CSI) values. WiFinger is deigned to recognize a set of finger-grained gestures, which are further used to realize continuous text input in off-the-shelf WiFi devices. As the results show, WiFinger achieves up to 90.4% average classification accuracy for recognizing 9 digits finger-grained gestures from American Sign Language (ASL), and its average accuracy for single individual number text input in desktop reaches 82.67% within 90 digits.

References

1. Ali, K., Liu, A. X., Wang, W., and Shahzad, M. Keystroke recognition using wifi signals. In Proc of ACM MobiCom (2015), 90--102.
2. Wang, G., Zou, Y., Zhou, Z., k. wu, and Ni, L. We can hear you with wi-fi! In Proc of ACM MobiCom (2014), 593--604.
3. Gupta, S., Morris, D., Patel, S., and Tan, D. Soundwave: using the doppler effect to sense gestures. In Proc of the SIGCHI Conference on Human Factors in Computing Systems (2012), 1911--1914.
4. Leap Motion. https://www.leapmotion.com.
5. Microsoft Kinect. http://www.roborealm.com/help/MicrosoftKinect.php.
6. Pu, Q., Gupta, S., Gollakota, S., and Patel, S. Whole-home gesture recognition using wireless signals. In Proc of ACM MobiCom (2013), 27--38.
7. Adib, F., and Katabi, D. See through walls with wifi! In Proc of ACM SIGCOMM (2013), 75--86.
8. Melgarejo, P., Zhang, X., Ramanathan, P., and Chu, D. Leveraging directional antenna capabilities for fine-grained gesture recognition. In Proc of ACM UbiComp (2014), 541--551.
9. Halperin, D., Hu, W., Sheth, A., and Wetherall, D. Tool release: Gathering 802.11n traces with channel state information. ACM SIGCOMM Computer Communication Review 41, 1 (2011), 53--53.
10. Sturman, D. J., and Zeltzer, D. A survey of glove-based input. IEEE Computer Graphics and Applications 14, 1 (1994), 30--39.
11. Dipietro, L., Sabatini, A. M., and Dario, P. A survey of glove-based systems and their applications. IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews) 38, 4 (2008), 461--482.
12. Finger Gesture. http://www.lifeprint.com/dictionary.htm.
13. Scholz, M., Riedel, T., Hock, M., and Beigl, M. Device-free and device-bound activity recognition using radio signal strength. In Proc of the 4th Augmented Human International Conference, ACM (2013), 100--107.
14. Tarzia, S. P., Dick, R. P., Dinda, P. A., and Memik, G. Sonar-based measurement of user presence and attention. In Proc of ACM UbiComp (2009), 89--92.
15. Zhao, Y., Patwari, N., Phillips, J. M., and Venkatasubramanian, S. Radio tomographic imaging and tracking of stationary and moving people via kernel distance. In Proc of the 12th international conference on Information processing in sensor networks (2013), 229--240.
16. Starner, T., and Pentland, A. Real-time american sign language recognition from video using hidden markov models. In Motion-Based Recognition. Springer, 1997, 227--243.
17. Juan, P. W., Kölsch, M., Stern, H., and Edan, Y. Vision-based hand-gesture applications. Communications of the ACM 54, 2 (2011), 60--71.
18. Rehg, J. M., and Kanade, T. Visual tracking of high dof articulated structures: an application to human hand tracking. In Computer Vision ECCV'94. Springer, 1994, 35--46.
19. Ketabdar, H., Roshandel, M., and Kamer, A. Y. Towards using embedded magnetic field sensor for around mobile device 3d interaction. In Proc of the 12th international conference on Human computer interaction with mobile devices and services (2010), 153--156.
20. Ketabdar, H., Moghadam, P., Naderi, B., and Roshandel, M. Magnetic signatures in air for mobile devices. In Proc of the 14th international conference on Human-computer interaction with mobile devices and services companion (2012), 185--188.
21. Agrawal, S., Constandache, I., Gaonkar, S., Roy, C. R., Caves, K., and DeRuyter, F. Using mobile phones to write in air. In Proc of ACM Mobisys (2011), 15--28.
22. Park, T., Lee, J., Hwang, I., Yoo, C., Nachman, L., and Song, J. E-gesture: a collaborative architecture for energy-efficient gesture recognition with hand-worn sensor and mobile devices. In Proc of ACM SenSys (2011), 260--273.
23. Sigg, S., Shi, S., Buesching, F., Ji, Y., and Wolf, L. Leveraging rf-channel fluctuation for activity recognition: Active and passive systems, continuous and rssi-based signal features. In Proc of International Conference on Advances in Mobile Computing & Multimedia (2013), 43.
24. Sigg, S., Scholz, M., Shi, S., Ji, Y., and Beigl, M. Rf-sensing of activities from non-cooperative subjects in device-free recognition systems using ambient and local signals. Mobile Computing, IEEE Transactions on 13, 4 (2014), 907--920.
25. Abdelnasser, H., Youssef, M., and Harras, K. A. Wigest: A ubiquitous wifi-based gesture recognition system. In Computer Communications (INFOCOM), 2015 IEEE Conference on (2015), 1472--1480.
26. Han, C., Wu, K., Wang, Y., and Ni, L. M. Wifall: Device-free fall detection by wireless networks. In Proc of IEEE INFOCOM (2014), 271--279.
27. Wang, Y., Liu, J., Chen, Y., Gruteser, M., Yang, J., and Liu, H. E-eyes: device-free location-oriented activity identification using fine-grained wifi signatures. In Proc of ACM MobiCom (2014), 617--628.
28. Zhou, Z., Yang, Z., Wu, C., Shangguan, L., and Liu, Y. Towards omnidirectional passive human detection. In Proc of IEEE INFOCOM (2013), 3057--3065.
29. Xi, W., Zhao, J., Li, X.-Y., Zhao, K., Tang, S., Liu, X., and Jiang, Z. Electronic frog eye: Counting crowd using wifi. In Proc of IEEE INFOCOM (2014), 361--369.
30. Nandakumar, R., Kellogg, B., and Gollakota, S. Wi-fi gesture recognition on existing devices. CoRR abs/1411.5394 (2014).
31. Sen, S., Lee, J., Kim, K.-H., and Congdon, P. Avoiding multipath to revive inbuilding wifi localization. In Proc of the 11th annual international conference on Mobile systems, applications, and services (2013), 249--262.
32. Xiao, J., Wu, K., Yi, Y., and Ni, L. M. Fifs: Fine-grained indoor fingerprinting system. In Computer Communications and Networks (ICCCN), 2012 21st International Conference on (2012), 1--7.
33. Yang, Z., Zhou, Z., and Liu, Y. From rssi to csi: Indoor localization via channel response. ACM Computing Surveys (CSUR) 46, 2 (2013), 25.
34. Kellogg, B., Talla, V., and Gollakota, S. Bringing gesture recognition to all devices. In 11th USENIX Symposium on Networked Systems Design and Implementation (NSDI 14) (2014), 303--316.
35. Lyonnet, B., Ioana, C., and Amin, M. G. Human gait classification using microdoppler time-frequency signal representations. In Radar Conference, 2010 IEEE (2010), 915--919.
36. Adib, F., Kabelac, Z., Katabi, D., and Miller, R. C. 3d tracking via body radio reflections. In 11th USENIX Symposium on Networked Systems Design and Implementation (NSDI 14) (2014), 317--329.
37. Chen, B., Yenamandra, V., and Srinivasan, K. Tracking keystrokes using wireless signals. In Proc of the 13th Annual International Conference on Mobile Systems, Applications, and Services (2015), 31--44.
38. Halperin, D., Hu, W., Sheth, A., and Wetherall, D. Two antennas are better than one: A measurement study of 802.11 n.
39. Davies, L., and Gather, U. The identification of multiple outliers. Journal of the American Statistical Association 88, 423 (1993), 782--792.
40. Wang, W., Liu, A. X., Shahzad, M., Ling, K., and Lu, S. Understanding and modeling of wifi signal based human activity recognition. In Proc of ACM MobiCom (2015), 65--76.
41. Mallat, S. G. A theory for multiresolution signal decomposition: the wavelet representation. Pattern Analysis and Machine Intelligence, IEEE Transactions on 11, 7 (1989), 674--693.
42. Müller, M. Dynamic time warping. Information retrieval for music and motion (2007), 69--84.
43. Halperin, D., Hu, W., Sheth, A., and Wetherall, D. Predictable 802.11 packet delivery from wireless channel measurements. In Proc of ACM SIGCOMM (2010), 159--170.
44. Committee, I. C. S. L. M. S., et al. Wireless LAN medium access control (MAC) and physical layer (PHY) specifications. IEEE Std, 1997.
45. Perahia, E., and Stacey, R. Next Generation Wireless LANS: 802.11 n and 802.11 ac. Cambridge university press, 2013.
46. Quegan, S. Spotlight synthetic aperture radar: Signal processing algorithms. International Journal for Numerical Methods in Engineering 83, 11 (2010), 1498--1517.
47. Baldauf, M., Dustdar, S., and Rosenberg, F. A survey on context-aware systems. International Journal of Ad Hoc and Ubiquitous Computing 2, 4 (2007), 263--277.

Index Terms

• Human-centered computing

  • Human computer interaction (HCI)