Embodied hands: modeling and capturing hands and bodies together

Abstract

Humans move their hands and bodies together to communicate and solve tasks. Capturing and replicating such coordinated activity is critical for virtual characters that behave realistically. Surprisingly, most methods treat the 3D modeling and tracking of bodies and hands separately. Here we formulate a model of hands and bodies interacting together and fit it to full-body 4D sequences. When scanning or capturing the full body in 3D, hands are small and often partially occluded, making their shape and pose hard to recover. To cope with low-resolution, occlusion, and noise, we develop a new model called MANO (hand Model with Articulated and Non-rigid defOrmations). MANO is learned from around 1000 high-resolution 3D scans of hands of 31 subjects in a wide variety of hand poses. The model is realistic, low-dimensional, captures non-rigid shape changes with pose, is compatible with standard graphics packages, and can fit any human hand. MANO provides a compact mapping from hand poses to pose blend shape corrections and a linear manifold of pose synergies. We attach MANO to a standard parameterized 3D body shape model (SMPL), resulting in a fully articulated body and hand model (SMPL+H). We illustrate SMPL+H by fitting complex, natural, activities of subjects captured with a 4D scanner. The fitting is fully automatic and results in full body models that move naturally with detailed hand motions and a realism not seen before in full body performance capture. The models and data are freely available for research purposes at http://mano.is.tue.mpg.de.

References

1. 2000. CMU Graphics Lab Motion Capture Database. http://mocap.cs.cmu.edu. (2000).
2. 3dMDbody 2017. 3dMDbody. http://www.3dmd.com/3dmd-systems/4d/3dmddynamic-body. (2017). Accessed: 01/09/2017.
3. 3dMDhand 2017. 3dMDhand. http://www.3dmd.com/3dmdhandfoot-system. (2017). Accessed: 01/09/2017.
4. Brett Allen, Brian Curless, and Zoran Popović. 2003. The space of human body shapes: Reconstruction and parameterization from range scans. ACM Transactions on Graphics, (Proc. SIGGRAPH) 22, 3 (2003), 587--594.
5. Brett Allen, Brian Curless, Zoran Popović, and Aaron Hertzmann. 2006. Learning a Correlated Model of Identity and Pose-dependent Body Shape Variation for Real-time Synthesis. In ACM SIGGRAPH/Eurographics Symposium on Computer Animation (SCA '06). Eurographics Association, 147--156.
6. Dragomir Anguelov, Praveen Srinivasan, Daphne Koller, Sebastian Thrun, Jim Rodgers, and James Davis. 2005. SCAPE: Shape completion and animation of people. ACM Transactions on Graphics (TOG) 24, 3 (2005), 408--416.
7. Vassilis Athitsos and Stan Sclaroff. 2003. Estimating 3D Hand Pose from a Cluttered Image. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 432--439.
8. Luca Ballan, Aparna Taneja, Juergen Gall, Luc Van Gool, and Marc Pollefeys. 2012. Motion Capture of Hands in Action using Discriminative Salient Points. In European Conference on Computer Vision (ECCV). 640--653.
9. Sven Bambach, Stefan Lee, David Crandall, and Chen Yu. 2015. Lending A Hand: Detecting Hands and Recognizing Activities in Complex Egocentric Interactions. In International Conference on Computer Vision (ICCV). 1949--1957.
10. Federica Bogo, Michael J. Black, Matthew Loper, and Javier Romero. 2015. Detailed Full-Body Reconstructions of Moving People from Monocular RGB-D Sequences. In International Conference on Computer Vision (ICCV). 2300--2308.
11. Federica Bogo, Angjoo Kanazawa, Christoph Lassner, Peter Gehler, Javier Romero, and Michael J. Black. 2016. Keep it SMPL: Automatic Estimation of 3D Human Pose and Shape from a Single Image. In European Conference on Computer Vision --- ECCV 2016.
12. Matthieu Bray, Esther Koller-Meier, and Luc Van Gool. 2007. Smart Particle Filtering for High-Dimensional Tracking. Computer Vision and Image Understanding (CVIU) 106, 1 (2007), 116--129.
13. Richard G. Brereton. 2015. The Mahalanobis distance and its relationship to principal component scores. Journal of Chemometrics 29, 3 (2015), 143--145.
14. Yinpeng Chen, Zicheng Liu, and Zhengyou Zhang. 2013. Tensor-Based Human Body Modeling. In IEEE Conf. on Computer Vision and Pattern Recognition (CVPR). 105--112.
15. Martin De La Gorce, David J. Fleet, and Nikos Paragios. 2011. Model-Based 3D Hand Pose Estimation from Monocular Video. IEEE Transactions on Pattern Analysis and Machine Intelligence (PAMI) 33, 9 (2011), 1793--1805.
16. Laura Dipietro, Angelo M Sabatini, and Paolo Dario. 2008. A survey of glove-based systems and their applications. Transactions on Systems, Man, and Cybernetics 38, 4 (2008), 461--482.
17. Alexey Dosovitskiy, Jost Tobias Springenberg, and Thomas Brox. 2015. Learning to generate chairs with convolutional neural networks. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 1538--1546.
18. George ElKoura and Karan Singh. 2003. Handrix: animating the human hand. In Proceedings of the 2003 ACM SIGGRAPH/Eurographics symposium on Computer animation. Eurographics Association, 110--119.
19. Ali Erol, George Bebis, Mircea Nicolescu, Richard D. Boyle, and Xander Twombly. 2007. Vision-based hand pose estimation: A review. Computer Vision and Image Understanding (CVIU) 108, 1--2 (2007), 52--73.
20. Thomas Feix, Javier Romero, Heinz-Bodo Schmiedmayer, Aaron M. Dollar, and Danica Kragic. 2016. The GRASP Taxonomy of Human Grasp Types. IEEE Transactions on Human-Machine Systems 46, 1 (2016), 66--77.
21. Oren Freifeld and Michael J. Black. 2012. Lie Bodies: A Manifold Representation of 3D Human Shape. In European Conf. on Computer Vision (ECCV) (Part I, LNCS 7572), A. Fitzgibbon et al. (Eds.) (Ed.). Springer-Verlag, 1--14.
22. Stuart Geman and Donald E. McClure. 1987. Statistical methods for tomographic image reconstruction. In Proceedings of the 46th Session of the International Statistical Institute, Bulletin of the ISI, Vol. 52.
23. Henning Hamer, Juergen Gall, Thibaut Weise, and Luc Van Gool. 2010. An Object-Dependent Hand Pose Prior from Sparse Training Data. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 671--678.
24. Henning Hamer, Konrad Schindler, Esther Koller-Meier, and Luc Van Gool. 2009. Tracking a hand manipulating an object. In International Conference on Computer Vision (ICCV). 1475--1482.
25. Nils Hasler, Carsten Stoll, Martin Sunkel, Bodo Rosenhahn, and Hans-Peter Seidel. 2009. A statistical model of human pose and body shape. Computer Graphics Forum 28, 2 (2009), 337--346.
26. Nils Hasler, Thorsten Thormählen, Bodo Rosenhahn, and Hans-Peter Seidel. 2010. Learning Skeletons for Shape and Pose. In Proceedings of the 2010 ACM SIGGRAPH Symposium on Interactive 3D Graphics and Games (I3D '10). ACM, New York, NY, USA, 23--30.
27. Tony Heap and David Hogg. 1996. Towards 3D hand tracking using a deformable model. In IEEE International Conference on Automatic Face and Gesture Recognition (FG). 140--145.
28. David A. Hirshberg, Matthew Loper, Eric Rachlin, and Michael J. Black. 2012. Coregistration: Simultaneous alignment and modeling of articulated 3D shape. In European Conf. on Computer Vision (ECCV). 242--255.
29. Catalin Ionescu, Dragos Papava, Vlad Olaru, and Cristian Sminchisescu. 2014. Human3.6M: Large Scale Datasets and Predictive Methods for 3D Human Sensing in Natural Environments. IEEE Transactions on Pattern Analysis and Machine Intelligence 36, 7 (jul 2014), 1325--1339.
30. Sophie Jörg, Jessica Hodgins, and Alla Safonova. 2012. Data-driven finger motion synthesis for gesturing characters. ACM Transactions on Graphics (TOG) 31, 6 (2012), 189.
31. Sameh Khamis, Jonathan Taylor, Jamie Shotton, Cem Keskin, Shahram Izadi, and Andrew Fitzgibbon. 2015. Learning an Efficient Model of Hand Shape Variation From Depth Images. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 2540--2548.
32. David Kim, Otmar Hilliges, Shahram Izadi, Alex D. Butler, Jiawen Chen, Iason Oikonomidis, and Patrick Olivier. 2012. Digits: Freehand 3D interactions anywhere using a wrist-worn gloveless sensor. In ACM Symposium on User Interface Software and Technology (UIST). 167--176.
33. Leap 2017. Leap Motion. https://www.leapmotion.com. (2017). Accessed: 01/09/2017.
34. John P. Lewis, Matt Cordner, and Nickson Fong. 2000. Pose Space Deformation: A Unified Approach to Shape Interpolation and Skeleton-driven Deformation. In ACM Transactions on Graphics (SIGGRAPH). 165--172.
35. Charles Loop. 1987. Smooth Subdivision Surfaces Based on Triangles. Department of Mathematics, University of Utah.
36. Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael J. Black. 2015. SMPL: A Skinned Multi-Person Linear Model. ACM Transactions on Graphics, (Proc. SIGGRAPH Asia) 34, 6 (Oct. 2015), 248:1--248:16.
37. Matthew M. Loper and Michael J. Black. 2014. OpenDR: An Approximate Differentiable Renderer. In Computer Vision - ECCV 2014 (Lecture Notes in Computer Science), Vol. 8695. Springer, 154--169.
38. John MacCormick and Michael Isard. 2000. Partitioned sampling, articulated objects, and interface-quality hand tracking. In European Conference on Computer Vision (ECCV). 3--19.
39. MANO. web. http://mano.is.tue.mpg.de. (web). Accessed: 2017-09-01.
40. Stan Melax, Leonid Keselman, and Sterling Orsten. 2013. Dynamics Based 3D Skeletal Hand Tracking. In Graphics Interface. 63--70.
41. Alex Mohr and Michael Gleicher. 2003. Building efficient, accurate character skins from examples. ACM Transactions on Graphics (TOG) 22, 3 (2003), 562--568.
42. Jorge Nocedal and Stephen Wright. 2006. Numerical optimization. Springer Science.
43. Markus Oberweger, Paul Wohlhart, and Vincent Lepetit. 2015. Training a Feedback Loop for Hand Pose Estimation. In International Conference on Computer Vision (ICCV). 3316--3324.
44. Iason Oikonomidis, Nikolaos Kyriazis, and Antonis A. Argyros. 2011a. Efficient model-based 3D tracking of hand articulations using Kinect. In British Machine Vision Conference (BMVC). 101.1--101.11.
45. Iason Oikonomidis, Nikolaos Kyriazis, and Antonis A. Argyros. 2011b. Full DOFtracking of a hand interacting with an object by modeling occlusions and physical constraints. In International Conference on Computer Vision (ICCV). 2088--2095.
46. Iasonas Oikonomidis, Nikolaos Kyriazis, and Antonis A. Argyros. 2012. Tracking the articulated motion of two strongly interacting hands. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 1862--1869.
47. Iason Oikonomidis, Manolis I.A. Lourakis, and Antonis A. Argyros. 2014. Evolutionary Quasi-random Search for Hand Articulations Tracking. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 3422--3429.
48. Edwin Olson and Pratik Agarwal. 2013. Inference on Networks of Mixtures for Robust Robot Mapping. International Journal of Robotics Research (IJRR) 32, 7 (2013), 826--840.
49. Wilder Penfield and Edwin Boldrey. 1937. Somatic motor and sensory representation in the cerebral cortex of man as studied by electrical stimulation. Brain 60, 4 (1937), 389.
50. Gerard Pons-Moll, Javier Romero, Naureen Mahmood, and Michael J. Black. 2015. Dyna: A Model of Dynamic Human Shape in Motion. ACM Transactions on Graphics, (Proc. SIGGRAPH) 34, 4 (July 2015), 120:1--120:14.
51. Chen Qian, Xiao Sun, Yichen Wei, Xiaoou Tang, and Jian Sun. 2014. Realtime and Robust Hand Tracking from Depth. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 1106--1113.
52. James M. Rehg and Takeo Kanade. 1994. Visual tracking of high dof articulated structures: an application to human hand tracking. In European Conference on Computer Vision (ECCV). 35--46.
53. Kathleen M. Robinette, Sherri Blackwell, Hein Daanen, Mark Boehmer, Scott Fleming, Tina Brill, David Hoeferlin, and Dennis Burnsides. 2002. Civilian American and European Surface Anthropometry Resource (CAESAR) Final Report. Technical Report AFRL-HE-WP-TR-2002-0169. US Air Force Research Laboratory.
54. Gregory Rogez, James S. Supančič III, and Deva Ramanan. 2015. First-Person Pose Recognition Using Egocentric Workspaces. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 4325--4333.
55. Javier Romero, Hedvig Kjellström, Carl H. Ek, and Danica Kragic. 2013. Non-parametric hand pose estimation with object context. Image and Vision Computing 31, 8 (2013), 555 -- 564.
56. Marco Santello, Martha Flanders, and John F. Soechting. 1998. Postural Hand Synergies for Tool Use. Journal of Neuroscience 18, 23 (1998), 10105--10115.
57. Tanner Schmidt, Richard Newcombe, and Dieter Fox. 2014. DART: Dense Articulated Real-Time Tracking. In Robotics: Science and Systems (RSS).
58. Matthias Schröder, Jonathan Maycock, Helge Ritter, and Mario Botsch. 2014. Real-time hand tracking using synergistic inverse kinematics. In Robotics and Automation (ICRA), 2014 IEEE International Conference on. 5447--5454.
59. Hyewon Seo, Frederic Cordier, and Nadia Magnenat-Thalmann. 2003. Synthesizing Animatable Body Models with Parameterized Shape Modifications. In ACM SIGGRAPH/Eurographics Symposium on Computer Animation (SCA '03). 120--125.
60. Toby Sharp, Cem Keskin, Duncan Robertson, Jonathan Taylor, Jamie Shotton, David Kim, Christoph Rhemann, Ido Leichter, Alon Vinnikov, Yichen Wei, Daniel Freedman, Pushmeet Kohli, Eyal Krupka, Andrew Fitzgibbon, and Shahram Izadi. 2015. Accurate, Robust, and Flexible Real-time Hand Tracking. In ACM Conference on Human Factors in Computing Systems (CHI). 3633--3642.
61. Tomas Simon, Hanbyul Joo, Iain Matthews, and Yaser Sheikh. 2017. Hand Keypoint Detection in Single Images using Multiview Bootstrapping. CVPR (2017).
62. Srinath Sridhar, Franziska Mueller, Antti Oulasvirta, and Christian Theobalt. 2015. Fast and Robust Hand Tracking Using Detection-Guided Optimization. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 3213--3221.
63. Srinath Sridhar, Franziska Mueller, Michael Zollhoefer, Dan Casas, Antti Oulasvirta, and Christian Theobalt. 2016. Real-time Joint Tracking of a Hand Manipulating an Object from RGB-D Input. In European Conference on Computer Vision (ECCV). 294--310.
64. Srinath Sridhar, Antti Oulasvirta, and Christian Theobalt. 2013. Interactive Markerless Articulated Hand Motion Tracking using RGB and Depth Data. In International Conference on Computer Vision (ICCV). 2456--2463.
65. Björn Stenger, Paulo R.S. Mendonça, and Roberto Cipolla. 2001. Model-based 3D tracking of an articulated hand. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 310--315.
66. Erik B. Sudderth, Michael I. Mandel, William T. Freeman, and Alan S. Willsky. 2004. Visual Hand Tracking Using Nonparametric Belief Propagation. In Workshop on Generative Model Based Vision. 189--189.
67. James S. Supančič III, Grégory Rogez, Yi Yang, Jamie Shotton, and Deva Ramanan. 2015. Depth-based hand pose estimation: data, methods, and challenges. In International Conference on Computer Vision (ICCV). 1868--1876.
68. Andrea Tagliasacchi, Sofien Bouaziz, Mark Pauly, and Hao Li. 2016. Modern techniques and applications for real-time non-rigid registration. In SIGGRAPH ASIA 2016 Courses.
69. Andrea Tagliasacchi, Matthias Schröder, Anastasia Tkach, Sofien Bouaziz, Mario Botsch, and Mark Pauly. 2015. Robust Articulated-ICP for Real-Time Hand Tracking. Eurographics Symposium on Geometry Processing (SGP) 34, 5 (2015), 101--114.
70. David Joseph Tan, Thomas Cashman, Jonathan Taylor, Andrew Fitzgibbon, Daniel Tarlow, Sameh Khamis, Shahram Izadi, and Jamie Shotton. 2016. Fits Like a Glove: Rapid and Reliable Hand Shape Personalization. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 5610--5619.
71. Danhang Tang, Hyung Jin Chang, Alykhan Tejani, and Tae-Kyun Kim. 2017. Latent Regression Forest: Structured Estimation of 3D Hand Poses. IEEE Transactions on Pattern Analysis and Machine Intelligence 39, 7 (July 2017), 1374--1387.
72. Jonathan Taylor, Lucas Bordeaux, Thomas Cashman, Bob Corish, Cem Keskin, Eduardo Soto, David Sweeney, Julien Valentin, Benjamin Luff, Arran Topalian, Erroll Wood, Sameh Khamis, Pushmeet Kohli, Toby Sharp, Shahram Izadi, Richard Banks, Andrew Fitzgibbon, and Jamie Shotton. 2016. Efficient and Precise Interactive Hand Tracking through Joint, Continuous Optimization of Pose and Correspondences. ACM Transactions on Graphics (SIGGRAPH) 35, 4 (2016), 143:1--143:12.
73. Jonathan Taylor, Richard Stebbing, Varun Ramakrishna, Cem Keskin, Jamie Shotton, Shahram Izadi, Aaron Hertzmann, and Andrew Fitzgibbon. 2014. User-Specific Hand Modeling from Monocular Depth Sequences. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 644--651.
74. Arasanathan Thayananthan, Björn Stenger, Philip H.S. Torr, and Roberto Cipolla. 2003. Shape context and chamfer matching in cluttered scenes. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 127--133.
75. Anastasia Tkach, Mark Pauly, and Andrea Tagliasacchi. 2016. Sphere-meshes for real-time hand modeling and tracking. ACM Transactions on Graphics (TOG) 35, 6 (2016).
76. Emo Todorov and Zoubin Ghahramani. 2004. Analysis of the synergies underlying complex hand manipulation. In The 26th Annual International Conference of the IEEE Engineering in Medicine and Biology Society, Vol. 2. 4637--4640.
77. Jonathan Tompson, Murphy Stein, Yann Lecun, and Ken Perlin. 2014. Real-Time Continuous Pose Recovery of Human Hands Using Convolutional Networks. ACM Transactions on Graphics (TOG) 33, 5 (2014), 169:1--169:10.
78. Dimitrios Tzionas, Luca Ballan, Abhilash Srikantha, Pablo Aponte, Marc Pollefeys, and Juergen Gall. 2016. Capturing Hands in Action using Discriminative Salient Points and Physics Simulation. International Journal of Computer Vision (IJCV) 118, 2 (2016), 172--193.
79. Dimitrios Tzionas and Juergen Gall. 2015. 3D Object Reconstruction from Hand-Object Interactions. In International Conference on Computer Vision (ICCV). 729--737.
80. Gül Varol, Javier Romero, Xavier Martin, Naureen Mahmood, Michael J. Black, Ivan Laptev, and Cordelia Schmid. 2017. Learning from Synthetic Humans. In CVPR.
81. Robert Y. Wang and Jovan Popović. 2009. Real-time Hand-tracking with a Color Glove. ACM Transactions on Graphics (TOG) 28, 3 (2009), 63:1--63:8.
82. Yangang Wang, Jianyuan Min, Jianjie Zhang, Yebin Liu, Feng Xu, Qionghai Dai, and Jinxiang Chai. 2013. Video-based hand manipulation capture through composite motion control. ACM Transactions on Graphics (TOG) 32, 4 (2013), 43:1--43:14.
83. Ying Wu, John Y. Lin, and Thomas S. Huang. 2001. Capturing natural hand articulation. In International Conference on Computer Vision (ICCV). 426--432.
84. Chi Xu, Ashwin Nanjappa, Xiaowei Zhang, and Li Cheng. 2016. Estimate Hand Poses Efficiently from Single Depth Images. International Journal of Computer Vision (IJCV) 116, 1 (2016), 21--45.
85. Mao Ye, Qing Zhang, Liang Wang, Jiejie Zhu, Ruigang Yang, and Juergen Gall. 2013. A Survey on Human Motion Analysis from Depth Data. In Time-of-Flight and Depth Imaging. Sensors, Algorithms, and Applications. 149--187.
86. Wenping Zhao, Jinxiang Chai, and Ying-Qing Xu. 2012. Combining marker-based mocap and RGB-D camera for acquiring high-fidelity hand motion data. In ACM SIGGRAPH/Eurographics Symposium on Computer Animation. 33--42.

Index Terms

• Computing methodologies

  • Artificial intelligence

    • Computer vision

      • Image and video acquisition

        • Motion capture