Digits: freehand 3D interactions anywhere using a wrist-worn gloveless sensor

ABSTRACT

Digits is a wrist-worn sensor that recovers the full 3D pose of the user's hand. This enables a variety of freehand interactions on the move. The system targets mobile settings, and is specifically designed to be low-power and easily reproducible using only off-the-shelf hardware. The electronics are self-contained on the user's wrist, but optically image the entirety of the user's hand. This data is processed using a new pipeline that robustly samples key parts of the hand, such as the tips and lower regions of each finger. These sparse samples are fed into new kinematic models that leverage the biomechanical constraints of the hand to recover the 3D pose of the user's hand. The proposed system works without the need for full instrumentation of the hand (for example using data gloves), additional sensors in the environment, or depth cameras which are currently prohibitive for mobile scenarios due to power and form-factor considerations. We demonstrate the utility of Digits for a variety of application scenarios, including 3D spatial interaction with mobile devices, eyes-free interaction on-the-move, and gaming. We conclude with a quantitative and qualitative evaluation of our system, and discussion of strengths, limitations and future work.

References

1. Ashbrook, D. L. Enabling mobile. Phd thesis, Georgia Institute of Technology, 2010.
2. Avidan, S., and A. Shamir. Seam carving for content-aware image resizing. SIGGRAPH Comput. Graph. 26, 3 (July 2007), 10.
3. Bailly, G. et al. ShoeSense: A New Perspective on Hand Gestures and Wearable Applications. In ACM SIGCHI (2012), 1239--1248.
4. Becker, J. C., and N. V. Thakor. A study of the range of motion of human fingers with application to anthropomorphic designs. IEEE Trans. Biomed. Eng. 35, 2 (1988), 110--117.
5. Buxton, B. Multi-touch Systems that I have Known and Loved. Tech. rep., Microsoft Research, 2007.
6. Dipietro, L., A. M. Sabatini, and P. Dario. Evaluation of an instrumented glove for hand-movement acquisition. Int. J. Rehabil. Res. 40, 2 (2003), 179--189.
7. Dipietro, L., A. M. Sabatini, and P. Dario. A Survey of Glove-Based Systems and Their Applications. IEEE Trans. Syst. Man Cybern. Part C Appl. Rev. 38, 4 (2008), 461--482.
8. Erol, A. et al. Vision-based hand pose estimation: A review. Comput. Vision Image Understanding 108, 1--2 (Oct. 2007), 52--73.
9. Gustafson, S., D. Bierwirth, and P. Baudisch. Imaginary interfaces. In ACM UIST (2010), 3--12.
10. Harrison, C., H. Benko, and A. D. Wilson. OmniTouch: Wearable Multitouch Interaction Everywhere. In ACM UIST, ACM (2011), 441--450.
11. Harrison, C., D. Tan, and D. Morris. Skinput: Appropriating the Body as an Input Surface. ACM SIGCHI 3, 8 (2010), 453--462.
12. Hilliges, O. et al. HoloDesk: Direct 3D Interactions with a Situated See-Through Display. In ACM SIGCHI (2012), 2421--2430.
13. Howard, B., and S. Howard. Lightglove: Wrist-worn virtual typing and pointing. In IEEE ISWC (2001), 172--173.
14. Izadi, S. et al. C-Slate: A Multi-Touch and Object Recognition System for Remote Collaboration using Horizontal Surfaces. In IEEE TABLETOP (2007), 3--10.
15. Izadi, S. et al. KinectFusion: Real-time 3D Reconstruction and Interaction Using a Moving Depth Camera. In ACM UIST (2011), 559--568.
16. Kamper, D. G., T. George Hornby, and W. Z. Rymer. Extrinsic flexor muscles generate concurrent flexion of all three finger joints. J. Biomech. 35, 12 (2002), 1581--1589.
17. Kim, J. et al. The Gesture Watch: A Wireless Contact-free Gesture based Wrist Interface. In IEEE ISWC (2007), 1--8.
18. Lee, S. C., B. Li, and T. Starner. AirTouch: Synchronizing In-air Hand Gesture and On-body Tactile Feedback. In IEEE ISWC (2011), 3--10.
19. Lee, T., and T. Hollerer. Handy AR: Markerless Inspection of Augmented Reality Objects Using Fingertip Tracking. In IEEE ISWC (2007), 1--8.
20. Malik, S. Real-time hand tracking and finger tracking for interaction. Tech. rep., University of Toronto, 2003.
21. Marschner, S. R. et al. Image-Based BRDF Measurement Including Human Skin. In Eurographics Rendering Workshop, vol. 5 (1999), 1--15.
22. Mayol-Cuevas, W., B. Tordoff, and D. Murray. On the Choice and Placement of Wearable Vision Sensors. IEEE Trans. Syst. Man Cybern. Part A Syst. Humans 39, 2 (Mar. 2009), 414--425.
23. Metcalf, C. et al. Validation and Application of a Computational Model for Wrist and Hand Movements using Surface Markers. IEEE Trans. Biomed. Eng. 55, 3 (2008), 1199--1210.
24. Mistry, P., and P. Maes. SixthSense: a wearable gestural interface. In ACM SIGGRAPH ASIA Sketches, K. Anjyo, Ed., ACM (2009), 60558.
25. Molyneaux, D. et al. Interactive Environment-Aware Handheld Projectors for Pervasive Computing Spaces. In Pervasive Computing (2012), 197--215.
26. Nakatsuma, K., and H. Shinoda. Touch interface on back of the hand. In ACM SIGGRAPH Emerging Technologies (2011), 4503.
27. Oikonomidis, I., N. Kyriazis, and A. A. Argyros. Full DOF tracking of a hand interacting with an object by modeling occlusions and physical constraints. In IEEE ICCV (Nov. 2011), 2088--2095.
28. Pamplona, V. The image-based data glove. In Symposium on Virtual and Augmented Reality (2008), 204--211.
29. Prados, E., and O. Faugeras. Shape From Shading. In Handbook of Mathematical Models in Computer Vision, N. Paragios, Y. Chen, and O. Faugeras, Eds. Springer, 2006, 375--388.
30. Rekimoto, J. GestureWrist and GesturePad: unobtrusive wearable interaction devices. In IEEE ISWC, vol. 5 (2001), 21--27.
31. Romero, J., H. Kjellstrom, and D. Kragic. Hands in action: real-time 3D reconstruction of hands in interaction with objects. In IEEE ICRA (2010), 458--463.
32. Saponas, T. S. et al. Enabling always-available input with muscle-computer interfaces. In ACM UIST, no. 38 (2009), 167--176.
33. Starner, T. et al. The gesture pendant: a self-illuminating, wearable, infrared computer vision system for home automation control and medical monitoring. In IEEE ISWC (2000), 87--94.
34. Starner, T., J. Weaver, and A. Pentland. A wearable computer based American sign language recognizer. In IEEE ISWC, vol. 1 of ISWC '97 (1997), 130--137.
35. Vardy, A. The wristcam as input device. In IEEE ISWC (1999), 199--202.
36. Villar, N. et al. Mouse 2.0. In ACM UIST (New York, New York, USA, Oct. 2009), 33--42.
37. Wang, R. 6D hands: markerless hand-tracking for computer aided design. In ACM UIST (2011), 549--558.
38. Wise, S. et al. Evaluation of a fiber optic glove for semi-automated goniometric measurements. Int. J. Rehabil. Res. 27, 4 (1990), 411--424.
39. Zhang, Z. A flexible new technique for camera calibration. IEEE Trans. Pattern Anal. Mach. Intell. 22, 11 (2000), 1330--1334.

Index Terms

• Human-centered computing

  • Human computer interaction (HCI)