KinectFusion: real-time 3D reconstruction and interaction using a moving depth camera

ABSTRACT

KinectFusion enables a user holding and moving a standard Kinect camera to rapidly create detailed 3D reconstructions of an indoor scene. Only the depth data from Kinect is used to track the 3D pose of the sensor and reconstruct, geometrically precise, 3D models of the physical scene in real-time. The capabilities of KinectFusion, as well as the novel GPU-based pipeline are described in full. Uses of the core system for low-cost handheld scanning, and geometry-aware augmented reality and physics-based interactions are shown. Novel extensions to the core GPU pipeline demonstrate object segmentation and user interaction directly in front of the sensor, without degrading camera tracking or reconstruction. These extensions are used to enable real-time multi-touch interactions anywhere, allowing any planar or non-planar reconstructed physical surface to be appropriated for touch.

References

1. P. J. Besl and N. D. McKay. A method for registration of 3D shapes. IEEE Trans. Pattern Anal. Mach. Intell., 14:239--256, February 1992.
2. X. Cao and R. Balakrishnan. Interacting with dynamically defined information spaces using a handheld projector and a pen. In UIST, pages 225--234, 2006.
3. Y. Chen and G. Medioni. Object modeling by registration of multiple range images. Image and Vision Computing (IVC), 10(3):145--155, 1992.
4. Y. Cui et al. 3d shape scanning with a time-of-flight camera. In Computer Vision and Pattern Recognition (CVPR), pages 1173 --1180, June 2010.
5. B. Curless and M. Levoy. A volumetric method for building complex models from range images. ACM Trans. Graph., 1996.
6. S. Farsiu et al. Fast and robust multiframe super resolution. IEEE Transactions on Image Processing, 13(10):1327--1344, 2004.
7. J. Frahm et al. Building Rome on a cloudless day. In Proc. Europ. Conf. on Computer Vision (ECCV), 2010.
8. B. Freedman, A. Shpunt, M. Machline, and Y. Arieli. Depth Mapping Using Projected Patterns. Patent Application, 10 2008. WO 2008/120217 A2.
9. S. L. Grand. Broad-phase collision detection with CUDA. In GPU Gems 3. Addison-Wesley, 2007.
10. T. Harada. Real-time rigid body simulation on gpus. In GPU Gems 3. Addison-Wesley Professional, 2007.
11. R. Hartley and A. Zisserman. Multiple View Geometry in Computer Vision. Cambridge University Press, second edition, 2004.
12. P. Henry et al. RGB-D mapping: Using depth cameras for dense 3D modeling of indoor environments. In Proc. of the Int. Symposium on Experimental Robotics (ISER), 2010.
13. B. Huhle et al. Fusion of range and color images for denoising and resolution enhancement with a non-local filter. Computer Vision and Image Understanding, 114(12):1336--1345, 2010.
14. M. Kazhdan, M. Bolitho, and H. Hoppe. Poisson surface reconstruction. In Proc. of the Eurographics Symposium on Geometry Processing, 2006.
15. G. Klein and D. W. Murray. Parallel tracking and mapping for small AR workspaces. In ISMAR, 2007.
16. M. Levoy et al. The digital Michelangelo Project: 3D scanning of large statues. ACM Trans. Graph., 2000.
17. K. Low. Linear least-squares optimization for point-to-plane icp surface registration. Technical report, TR04-004, University of North Carolina, 2004.
18. P. Merrell et al. Real-time visibility-based fusion of depth maps. In Proc. of the Int. Conf. on Computer Vision (ICCV), 2007.
19. R. A. Newcombe and A. J. Davison. Live dense reconstruction with a single moving camera. In Proc. of the IEEE (CVPR), 2010.
20. R. A. Newcombe, S. Lovegrove, and A. J. Davison. Dense tracking and mapping in real-time. In Proc. of the Int. Conf. on Computer Vision (ICCV), 2011.
21. R. A. Newcombe et al. Real-Time Dense Surface Mapping and Tracking with Kinect. In ISMAR, 2011.
22. S. Osher and R. Fedkiw. Level Set Methods and Dynamic Implicit Surfaces. Springer, 2002.
23. S. Rusinkiewicz, O. Hall-Holt, and M. Levoy. Real-time 3D model acquisition. ACM Trans. Graph., 2002.
24. S. Rusinkiewicz and M. Levoy. Efficient variants of the ICP algorithm. 3D Digital Imaging and Modeling, Int. Conf. on, 0:145, 2001.
25. S. Thrun. Robotic mapping: A survey. In Exploring Artificial Intelligence in the New Millenium. 2002.
26. D. Vlasic et al. Dynamic shape capture using multi-view photometric stereo. ACM Trans. Graph., 28(5), 2009.
27. D. Wagner, T. Langlotz, and D. Schmalstieg. Robust and unobtrusive marker tracking on mobile phones. In ISMAR, pages 121--124, 2008.
28. T. Weise, T. Wismer, B. Leibe, and L. V. Gool. In-hand scanning with online loop closure. In IEEE Int. Workshop on 3-D Digital Imaging and Modeling, 2009.
29. K. Zhou, M. Gong, X. Huang, and B. Guo. Data-parallel octrees for surface reconstruction. IEEE Trans. on Visualization and Computer Graphics, 17, 2011.

Index Terms

• Computing methodologies

  • Artificial intelligence

    • Computer vision

      • Image and video acquisition

        • 3D imaging

  • Computer graphics

    • Animation