RetroDepth: 3D silhouette sensing for high-precision input on and above physical surfaces

ABSTRACT

We present RetroDepth, a new vision-based system for accurately sensing the 3D silhouettes of hands, styluses, and other objects, as they interact on and above physical surfaces. Our setup is simple, cheap, and easily reproducible, comprising of two infrared cameras, diffuse infrared LEDs, and any off-the-shelf retro-reflective material. The retro-reflector aids image segmentation, creating a strong contrast between the surface and any object in proximity. A new highly efficient stereo matching algorithm precisely estimates the 3D contours of interacting objects and the retro-reflective surfaces. A novel pipeline enables 3D finger, hand and object tracking, as well as gesture recognition, purely using these 3D contours. We demonstrate high-precision sensing, allowing robust disambiguation between a finger or stylus touching, pressing or interacting above the surface. This allows many interactive scenarios that seamlessly mix together freehand 3D interactions with touch, pressure and stylus input. As shown, these rich modalities of input are enabled on and above any retro-reflective surface, including custom "physical widgets" fabricated by users. We compare our system with Kinect and Leap Motion, and conclude with limitations and future work.

References

1. 3Gear Systems Inc. http://threegear.com/, 2013.
2. Agarwal, A., Izadi, S., Chandraker, M., and Blake, A. High precision multi-touch sensing on surfaces using cameras. In Tabletop'07, 197--200.
3. Akaoka, E., Ginn, T., and Vertegaal, R. Displayobjects: prototyping functional physical interfaces on 3d styrofoam, paper or cardboard models. In TEI'10, ACM (2010), 49--56.
4. Annett, M., Grossman, T., Wigdor, D., and Fitzmaurice, G. Medusa: a proximity-aware multi-touch tabletop. In UIST'11, ACM (2011), 337--346.
5. Benko, H., Jota, R., and Wilson, A. Miragetable: freehand interaction on a projected augmented reality tabletop. In CHI'12, ACM (2012), 199--208.
6. Bleyer, M., Rhemann, C., and Rother, C. Patchmatch stereo - stereo matching with slanted support windows. In British Machine Vision Conference (2011).
7. Breiman, L. Random Forests. Machine Learning 45, 1 (Oct. 2001), 5--32.
8. Brown, M. Z., Burschka, D., and Hager, G. D. Advances in computational stereo. PAMI 25, 8 (2003), 993--1008.
9. Davis, J. W., and Bobick, A. F. Sideshow: A silhouettebased interactive dual-screen environment. Tech. rep., MIT, 1998.
10. Dippon, A., and Klinker, G. Kinecttouch: accuracy test for a very low-cost 2.5 d multitouch tracking system. In ITS'11, ACM (2011), 49--52.
11. Haptix. http://www.haptixtouch.com/, 2013.
12. Haubner, N., Schwanecke, U., Dörner, R., Lehmann, S., and Luderschmidt, J. Detecting interaction above digital tabletops using a single depth camera. Machine Vision and Applications (2013), 1--13.
13. Hilliges, O., Izadi, S., Wilson, A. D., Hodges, S., Garcia-Mendoza, A., and Butz, A. Interactions in the air: adding further depth to interactive tabletops. In UIST'09, ACM (2009), 139--148.
14. Hilliges, O., Kim, D., Izadi, S., Weiss, M., and Wilson, A. Holodesk: direct 3d interactions with a situated seethrough display. In CHI'12, ACM (2012), 2421--2430.
15. Hirsch, M., Lanman, D., Holtzman, H., and Raskar, R. Bidi screen: a thin, depth-sensing lcd for 3d interaction using light fields. In TOG, vol. 28, ACM (2009), 159.
16. Izadi, S., Agarwal, A., Criminisi, A., Winn, J., Blake, A., and Fitzgibbon, A. C-slate: a multi-touch and object recognition system for remote collaboration using horizontal surfaces. In Tabletop'07, IEEE (2007), 3--10.
17. Izadi, S., Hodges, S., Butler, A., West, D., Rrustemi, A., Molloy, M., and Buxton,W. Thinsight: a thin form-factor interactive surface technology. CACM 52, 12, 90--98.
18. Izadi, S., Hodges, S., Taylor, S., Rosenfeld, D., Villar, N., Butler, A., and Westhues, J. Going beyond the display: a surface technology with an electronically switchable diffuser. In UIST'08, ACM (2008), 269--278.
19. Keskin, C., Kirac, F., Kara, Y. E., and Akarun, L. Hand Pose Estimation and Hand Shape Classification Using Multi-layered Randomized Decision Forests. In ECCV'12 (2012).
20. Koike, H., Sato, Y., and Kobayashi, Y. Integrating paper and digital information on enhanceddesk: a method for realtime finger tracking on an augmented desk system. TOCHI 8, 4 (2001), 307--322.
21. Krueger, M. W. Artificial reality II, vol. 10. Addison-Wesley Reading (Ma), 1991.
22. Lanman, D., and Taubin, G. Build your own 3d scanner: 3d photography for beginners. In ACM SIGGRAPH 2009 Courses, ACM (2009), 8.
23. Lee, J., Park, K. S., and Hahn, M. The 3d sensor table for bare hand tracking and posture recognition. In Advances in Multimedia Modeling. Springer, 2006, 138--146.
24. Liu, Y., Weibel, N., and Hollan, J. Interactive space: A framework for prototyping multi-touch interaction on and above the desktop. In CHI'13, vol. 13 (2013).
25. Malik, S., and Laszlo, J. Visual touchpad: a two-handed gestural input device. In ICMI'04, 289--296.
26. Marquardt, N., Jota, R., Greenberg, S., and Jorge, J. A. The continuous interaction space: interaction techniques unifying touch and gesture on and above a digital surface. In INTERACT'11. Springer, 2011, 461--476.
27. Marroquim, R., Kraus, M., and Cavalcanti, P. R. Efficient point-based rendering using image reconstruction. In SPBG (2007), 101--108.
28. Matsushita, N., and Rekimoto, J. Holowall: designing a finger, hand, body, and object sensitive wall. In UIST'97, ACM (1997), 209--210.
29. Moeller, J., Kerne, A., and Moeller, J. Zerotouch: An optical multi-touch and free-air interaction architecture. In CH'12, ACM (2012), 2165--2174.
30. Paradiso, J. A. Several sensor approaches that retrofit large surfaces for interactivity. In UbiComp 2002 (2002).
31. Remondino, F., and Stoppa, D. Tof range-imaging cameras. Springer, 2013.
32. Scharstein, D., and Szeliski, R. A taxonomy and evaluation of dense two-frame stereo correspondence algorithms. In IJCV (2002).
33. Shotton, J., Fitzgibbon, A., Cook, M., Sharp, T., Finocchio, M., Moore, R., Kipman, A., and Blake, A. Realtime Human Pose Recognition in Parts from Single Depth Images. In CVPR (2011).
34. Strecha, C., and Van Gool, L. PDE-based multi-view depth estimation. In Proc. 3DIMPVT (2002), 416--425.
35. Subramanian, S., Aliakseyeu, D., and Lucero, A. Multilayer interaction for digital tables. In UIST'06, ACM (2006), 269--272.
36. Takeoka, Y., Miyaki, T., and Rekimoto, J. Z-touch: an infrastructure for 3d gesture interaction in the proximity of tabletop surfaces. In ITS'10, ACM (2010), 91--94.
37. Ullmer, B., and Ishii, H. The metadesk: models and prototypes for tangible user interfaces. In UIST'97, ACM (1997), 223--232.
38. Wang, R., Paris, S., and Popović, J. 6d hands: markerless hand-tracking for computer aided design. In UIST'11, ACM (2011), 549--558.
39. Wellner, P. Interacting with paper on the digitaldesk. CACM 36, 7 (1993), 87--96.
40. Wesche, G., and Seidel, H.-P. Freedrawer: a free-form sketching system on the responsive workbench. In VRST, ACM (2001), 167--174.
41. Wilson, A. D. Playanywhere: a compact interactive tabletop projection-vision system. In UIST'05, ACM (2005), 83--92.
42. Wilson, A. D. Robust computer vision-based detection of pinching for one and two-handed gesture input. In UIST'06, ACM (2006), 255--258.
43. Wilson, A. D. Depth-sensing video cameras for 3d tangible tabletop interaction. In Tabletop'07, IEEE (2007), 201--204.
44. Wilson, A. D. Using a depth camera as a touch sensor. In ITS'10, ACM (2010), 69--72.

Index Terms

• Human-centered computing

  • Human computer interaction (HCI)