Multi-sensor data fusion for hand tracking using Kinect and Leap Motion

ABSTRACT

Often presented as competing products on the market of low cost 3D sensors, the Kinect™ and the Leap Motion™ (LM) can actually be complementary in some scenario. We promote, in this paper, the fusion of data acquired by both LM and Kinect sensors to improve hand tracking performances. The sensor fusion is applied to an existing augmented reality system targeting the treatment of phantom limb pain (PLP) in upper limb amputees. With the Kinect we acquire 3D images of the patient in real-time. These images are post-processed to apply a mirror effect along the sagittal plane of the body, before being displayed back to the patient in 3D, giving him the illusion that he has two arms. The patient uses the virtual reconstructed arm to perform given tasks involving interactions with virtual objects. Thanks to the plasticity of the brain, the restored visual feedback of the missing arm allows, in some cases, to reduce the pain intensity. The Leap Motion brings to the system the ability to perform accurate motion tracking of the hand, including the fingers. By registering the position and orientation of the LM in the frame of reference of the Kinect, we make our system able to accurately detect interactions of the hand and the fingers with virtual objects, which will greatly improve the user experience. We also show that the sensor fusion nicely extends the tracking domain by supplying finger positions even when the Kinect sensor fails to acquire the depth values for the hand.

References

1. B. Penelle, D. Mouraux, E. Brassinne, T. Tuna, A. Nonclercq, and N. Warzée. 3D augmented reality applied to the treatment of neuropathic pain. In Proc. 9th Intl Conf. on Disability, Virtual Reality and Assoc. Technologies, pages 61--68, Laval, France, September 2012. P M Sharkey, E Klinger (Eds).
2. P. J. Besl and N. D. McKay. A method for registration of 3-D shapes. IEEE Trans. Pattern Anal. Mach. Intell., 14(2):239--256, February 1992.
3. G. R. Bradski and A. Kaehler. Learning OpenCV, 1st Edition. O'Reilly Media, Inc., first edition, 2008.
4. H. Flor. Phantom-limb pain: characteristics, causes, and treatment. The Lancet Neurology, 1(3):182--189, 2002.
5. K. Khoshelham and S. O. Elberink. Accuracy and resolution of Kinect depth data for indoor mapping applications. Sensors, 12(12):1437--1454, February 2012.
6. G. L. Moseley. Using visual illusion to reduce at-level neuropathic pain in paraplegia. PAIN, 130(3):294--298, 2007.
7. I. Oikonomidis, N. Kyriazis, and A. Argyros. Efficient model-based 3D tracking of hand articulations using Kinect. In Proceedings of the British Machine Vision Conference, pages 101.1--101.11. BMVA Press, 2011.
8. V. S. Ramachandran and E. L. Altschuler. The use of visual feedback, in particular mirror visual feedback, in restoring brain function. Brain, 132(7):1693--1710, 2009.
9. F. Weichert, D. Bachmann, B. Rudak, and D. Fisseler. Analysis of the accuracy and robustness of the Leap Motion controller. Sensors, 13(5):6380--6393, May 2013.

Index Terms

• Computing methodologies

  • Computer graphics

    • Graphics systems and interfaces

• Human-centered computing

  • Human computer interaction (HCI)

    • Interaction devices

      • Graphics input devices