ClothCap: seamless 4D clothing capture and retargeting

Abstract

Designing and simulating realistic clothing is challenging. Previous methods addressing the capture of clothing from 3D scans have been limited to single garments and simple motions, lack detail, or require specialized texture patterns. Here we address the problem of capturing regular clothing on fully dressed people in motion. People typically wear multiple pieces of clothing at a time. To estimate the shape of such clothing, track it over time, and render it believably, each garment must be segmented from the others and the body. Our ClothCap approach uses a new multi-part 3D model of clothed bodies, automatically segments each piece of clothing, estimates the minimally clothed body shape and pose under the clothing, and tracks the 3D deformations of the clothing over time. We estimate the garments and their motion from 4D scans; that is, high-resolution 3D scans of the subject in motion at 60 fps. ClothCap is able to capture a clothed person in motion, extract their clothing, and retarget the clothing to new body shapes; this provides a step towards virtual try-on.

References

1. A. Balan and M. J. Black. 2008. The naked truth: Estimating body shape under clothing,. In European Conf. on Computer Vision, ECCV (LNCS), Vol. 5304. Springer-Verlag, Marseilles, France, 15--29.
2. Kiran S. Bhat, Christopher D. Twigg, Jessica K. Hodgins, Pradeep K. Khosla, Zoran Popović, and Steven M. Seitz. 2003. Estimating Cloth Simulation Parameters from Video. In Proceedings of the 2003 ACM SIGGRAPH/Eurographics Symposium on Computer Animation (SCA '03). Eurographics Association, Aire-la-Ville, Switzerland, Switzerland, 37--51. http://dl.acm.org/citation.cfm?id=846276.846282
3. Federica Bogo, Javier Romero, Gerard Pons-Moll, and Michael J. Black. 2017. Dynamic FAUST: Registering Human Bodies in Motion. In IEEE Conf. on Computer Vision and Pattern Recognition (CVPR).
4. K.L. Bouman, B. Xiao, P. Bataglia, and W.T. Freeman. 2013. Estimating the Material Properties of Fabrics from Videos. In International Conference in Computer Vision (ICCV). 1984--1991.
5. Derek Bradley, Tiberiu Popa, Alla Sheffer, Wolfgang Heidrich, and Tamy Boubekeur. 2008. Markerless Garment Capture. ACM Trans. Graphics (Proc. SIGGRAPH) 27, 3 (2008), 99.
6. Rémi Brouet, Alla Sheffer, Laurence Boissieux, and Marie-Paule Cani. 2012. Design Preserving Garment Transfer. ACM Transactions on Graphics (Aug. 2012). http://hal.inria.fr/hal-00695903
7. Dan Casas, Marco Volino, John Collomosse, and Adrian Hilton. 2014. 4d video textures for interactive character appearance. Computer Graphics Forum 33, 2 (2014), 371--380.
8. Xiaowu Chen, Bin Zhou, Feixiang Lu, Lin Wang, Lang Bi, and Ping Tan. 2015. Garment Modeling with a Depth Camera. ACM Trans. Graph. 34, 6, Article 203 (Oct. 2015), 12 pages.
9. Alvaro Collet, Ming Chuang, Pat Sweeney, Don Gillett, Dennis Evseev, David Calabrese, Hugues Hoppe, Adam Kirk, and Steve Sullivan. 2015. High-Quality Streamable Free-Viewpoint Video. ACM Transactions on Graphics (SIGGRAPH) 34, 4 (2015).
10. Radek Danecek, Endri Dibra, A. Cengiz Öztireli, Remo Ziegler, and Markus Gross. 2017. DeepGarment : 3D Garment Shape Estimation from a Single Image. Computer Graphics Forum 36(2), Proceedings of the 38th Annual Conference of the European Association for Computer Graphics (Eurographics) (2017).
11. Matthias Dantone, Juergen Gall, Christian Leistner, and Luc Van Gool. 2014. Body parts dependent joint regressors for human pose estimation in still images. IEEE Transactions on Pattern Analysis and Machine Intelligence 36, 11 (2014), 2131--2143.
12. Edilson de Aguiar, Leonid Sigal, Adrien Treuille, and Jessica K. Hodgins. 2010. Stable Spaces for Real-time Clothing. ACM Trans. Graph. 29, 4, Article 106 (July 2010), 9 pages.
13. Edilson de Aguiar, Carsten Stoll, Christian Theobalt, Naveed Ahmed, Hans-Peter Seidel, and Sebastian Thrun. 2008. Performance Capture from Sparse Multi-view Video. ACM Trans. Graph. 27, 3, Article 98 (Aug. 2008), 10 pages.
14. Mingsong Dou, Sameh Khamis, Yury Degtyarev, Philip Davidson, Sean Ryan Fanello, Adarsh Kowdle, Sergio Orts Escolano, Christoph Rhemann, David Kim, Jonathan Taylor, and others. 2016. Fusion4d: Real-time performance capture of challenging scenes. ACM Transactions on Graphics (TOG) 35, 4 (2016), 114.
15. Juergen Gall, Carsten Stoll, Edilson De Aguiar, Christian Theobalt, Bodo Rosenhahn, and Hans-Peter Seidel. 2009. Motion capture using joint skeleton tracking and surface estimation. In Computer Vision and Pattern Recognition, 2009. CVPR 2009. IEEE Conference on. IEEE, 1746--1753.
16. Russell Gillette, Craig Peters, Nicholas Vining, Essex Edwards, and Alla Sheffer. 2015. Real-Time Dynamic Wrinkling of Coarse Animated Cloth. In Proc. Symposium on Computer Animation.
17. Rony Goldenthal, David Harmon, Raanan Fattal, Michel Bercovier, and Eitan Grinspun. 2007. Efficient Simulation of Inextensible Cloth. ACM Transactions on Graphics (Proceedings of SIGGRAPH 2007) 26, 3 (2007).
18. P. Guan, L. Reiss, D. Hirshberg, A. Weiss, and M. J. Black. 2012. DRAPE: DRessing Any PErson. ACM Trans. on Graphics (Proc. SIGGRAPH) 31, 4 (July 2012), 35:1--35:10.
19. Nils Hasler, Carsten Stoll, Bodo Rosenhahn, Thorsten ThormÃd'hlen, and Hans-Peter Seidel. 2009. Estimating body shape of dressed humans. Computers & Graphics 33, 3 (2009), 211 -- 216. {IEEE} International Conference on Shape Modelling and Applications 2009.
20. Anna Hilsmann and Peter Eisert. 2009. Tracking and Retexturing Cloth for Real-Time Virtual Clothing Applications. In Proceedings of the 4th International Conference on Computer Vision/Computer Graphics CollaborationTechniques (MIRAGE '09). Springer-Verlag, Berlin, Heidelberg, 94--105.
21. Peng Huang, Margara Tejera, John Collomosse, and Adrian Hilton. 2015. Hybrid skeletal-surface motion graphs for character animation from 4d performance capture. ACM Transactions on Graphics (TOG) 34, 2 (2015), 17.
22. Matthias Innmann, Michael Zollhöfer, Matthias Nießner, Christian Theobalt, and Marc Stamminger. 2016. VolumeDeform: Real-time Volumetric Non-rigid Reconstruction. In Proceedings of European Conference on Computer Vision (ECCV). 17.
23. Arjun Jain, Thorsten Thormählen, Hans-Peter Seidel, and Christian Theobalt. 2010. MovieReshape: Tracking and Reshaping of Humans in Videos. ACM Trans. Graph. 29, 6, Article 148 (Dec. 2010), 10 pages.
24. Sam Johnson and Mark Everingham. 2010. Clustered Pose and Nonlinear Appearance Models for Human Pose Estimation. In Proceedings of the British Machine Vision Conference.
25. Ladislav Kavan, Dan Gerszewski, Adam W. Bargteil, and Peter-Pike Sloan. 2011. Physics-inspired Upsampling for Cloth Simulation in Games. ACM Trans. Graph. 30, 4, Article 93 (July 2011), 10 pages.
26. Doyub Kim, Woojong Koh, Rahul Narain, Kayvon Fatahalian, Adrien Treuille, and James F. O'Brien. 2013. Near-exhaustive Precomputation of Secondary Cloth Effects. ACM Transactions on Graphics 32, 4 (July 2013), 87:1--7. http://graphics.berkeley.edu/papers/Kim-NEP-2013-07/ Proceedings of ACM SIGGRAPH 2013, Anaheim.
27. Meekyoung Kim, Gerard Pons-Moll, Sergi Pujades, Sungbae Bang, Jinwwok Kim, Michael Black, and Sung-Hee Lee. 2017. Data-Driven Physics for Human Soft Tissue Animation. ACM Transactions on Graphics, (Proc. SIGGRAPH) 36, 4 (2017).
28. Christoph Lassner, Javier Romero, Martin Kiefel, Federica Bogo, Michael J. Black, and Peter V. Gehler. 2017. Unite the People: Closing the Loop Between 3D and 2D Human Representations. In IEEE Conf. on Computer Vision and Pattern Recognition (CVPR). http://files.is.tuebingen.mpg.de/classner/up
29. X. Liang, C. Xu, X. Shen, J. Yang, J. Tang, L. Lin, and S. Yan. 2015. Human Parsing with Contextualized Convolutional Neural Network. In Int. Conf. Comp. Vis. (ICCV).
30. Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael J. Black. 2015. SMPL: A Skinned Multi-Person Linear Model. ACM Trans. Graphics (Proc. SIGGRAPH Asia) 34, 6 (Oct. 2015), 248:1--248:16.
31. E. Miguel, D. Bradley, B. Thomaszewski, B. Bickel, W. Matusik, M. A. Otaduy, and S. Marschner. 2012. Data-Driven Estimation of Cloth Simulation Models. Comput. Graph. Forum 31, 2pt2 (May 2012), 519--528.
32. Alexandros Neophytou and Adrian Hilton. 2014. A layered model of human body and garment deformation. In 2014 2nd International Conference on 3D Vision, Vol. 1. IEEE, 171--178.
33. Richard A Newcombe, Dieter Fox, and Steven M Seitz. 2015. Dynamicfusion: Reconstruction and tracking of non-rigid scenes in real-time. In Proceedings of the IEEE conference on computer vision and pattern recognition. 343--352.
34. Gerard Pons-Moll, Javier Romero, Naureen Mahmood, and Michael J. Black. 2015a. Dyna: A Model of Dynamic Human Shape in Motion. ACM Transactions on Graphics, (Proc. SIGGRAPH) 34, 4 (July 2015), 120:1--120:14.
35. Gerard Pons-Moll, Jonathan Taylor, Jamie Shotton, Aaron Hertzmann, and Andrew Fitzgibbon. 2015b. Metric Regression Forests for Correspondence Estimation. International Journal of Computer Vision (2015), 1--13.
36. Tiberiu Popa, Qingnan Zhou, Derek Bradley, Vladislav Kraevoy, Hongbo Fu, Alla Sheffer, and Wolfgang Heidrich. 2009. Wrinkling Captured Garments Using Space-Time Data-Driven Deformation. Computer Graphics Forum (Proc. Eurographics) 28, 2 (2009), 427--435.
37. Nadia Robertini, Edilson De Aguiar, Thomas Helten, and Christian Theobalt. 2014. Efficient Multi-view Performance Capture of Fine-Scale Surface Detail. In Proceedings of the 2014 2Nd International Conference on 3D Vision - Volume 01 (3DV '14). IEEE Computer Society, Washington, DC, USA, 5--12.
38. K. Robinette, S. Blackwell, H. Daanen, M. Boehmer, S. Fleming, T. Brill, D. Hoeferlin, and D. Burnsides. 2002. Civilian American and European Surface Anthropometry Resource (CAESAR) Final Report. Technical Report AFRL-HE-WP-TR-2002-0169. US Air Force Research Laboratory.
39. Lorenz Rogge, Felix Klose, Michael Stengel, Martin Eisemann, and Marcus Magnor. 2014. Garment Replacement in Monocular Video Sequences. ACM Transactions on Graphics 34, 1 (Nov. 2014), 6:1--6:10.
40. Bodo Rosenhahn, Uwe Kersting, Katie Powell, Reinhard Klette, Gisela Klette, and Hans-Peter Seidel. 2007. A system for articulated tracking incorporating a clothing model. Machine Vision and Applications 18, 1 (2007), 25--40.
41. M. Sekine, K. Sugita, F. Perbet, B. Stenger, and M. Nishiyama. 2014. Virtual Fitting by Single-Shot Body Shape Estimation. In Int. Conf. on 3D Body Scanning Technologies. 406--413.
42. Leonid Sigal, Moshe Mahler, Spencer Diaz, Kyna McIntosh, Elizabeth Carter, Timothy Richards, and Jessica Hodgins. 2015. A Perceptual Control Space for Garment Simulation. ACM Trans. Graph. 34, 4, Article 117 (July 2015), 10 pages.
43. Olga Sorkine. 2006. Differential Representations for Mesh Processing. Computer Graphics Forum 25, 4 (2006), 789--807.
44. Olga Sorkine, Daniel Cohen-Or, Yaron Lipman, Marc Alexa, Christian Rössl, and Hans-Peter Seidel. 2004. Laplacian Surface Editing. In Proceedings of the EUROGRAPHICS/ACM SIGGRAPH Symposium on Geometry Processing. ACM Press, 179--188.
45. Carsten Stoll, Juergen Gall, Edilson de Aguiar, Sebastian Thrun, and Christian Theobalt. 2010. Video-based Reconstruction of Animatable Human Characters. ACM Trans. Graph. 29, 6, Article 139 (Dec. 2010), 10 pages.
46. Stephan Streuber, M. Alejandra Quiros-Ramirez, Matthew Q. Hill, Carina A. Hahn, Silvia Zuffi, Alice OâĂŹToole, and Michael J. Black. 2016. Body Talk: Crowdshaping Realistic 3D Avatars with Words. ACM Trans. Graph. (Proc. SIGGRAPH) 35, 4 (July 2016), 54:1--54:14.
47. D. Sun, E. Sudderth, and M. J. Black. 2010. Layered image motion with explicit occlusions, temporal consistency, and depth ordering. In Advances in Neural Information Processing Systems 23 (NIPS). MIT Press, 2226--2234.
48. M. Tejera, D. Casas, and A. Hilton. 2013. Animation Control of Surface Motion Capture. Cybernetics, IEEE Transactions on 43, 6 (Dec 2013), 1532--1545.
49. Daniel Vlasic, Ilya Baran, Wojciech Matusik, and Jovan Popović. 2008. Articulated mesh animation from multi-view silhouettes. ACM Transactions on Graphics (TOG) 27, 3 (2008), 97.
50. Timo von Marcard, Gerard Pons-Moll, and Bodo Rosenhahn. 2016. Human Pose Estimation from Video and IMUs. Transactions on Pattern Analysis and Machine Intelligence PAMI (Jan. 2016).
51. Timo von Marcard, Bodo Rosenhahn, Michael Black, and Gerard Pons-Moll. 2017. Sparse Inertial Poser: Automatic 3D Human Pose Estimation from Sparse IMUs. Computer Graphics Forum 36(2), Proceedings of the 38th Annual Conference of the European Association for Computer Graphics (Eurographics) (2017).
52. Huamin Wang, Florian Hecht, Ravi Ramamoorthi, and James F. O'Brien. 2010. Example-Based Wrinkle Synthesis for Clothing Animation. ACM Transactions on Graphics 29, 4 (July 2010), 107:1--8. http://graphics.berkeley.edu/papers/Wang-EBW-2010-07/ Proceedings of ACM SIGGRAPH 2010, Los Angles, CA.
53. Huamin Wang, James F. O'Brien, and Ravi Ramamoorthi. 2011. Data-Driven Elastic Models for Cloth: Modeling and Measurement. ACM Transactions on Graphics, Proc. SIGGRAPH 30, 4 (July 2011), 71:1--11.
54. Ruizhe Wang, Lingyu Wei, Etienne Vouga, Qixing Huang, Duygu Ceylan, Gerard Medioni, and Hao Li. 2016. Capturing Dynamic Textured Surfaces of Moving Targets. In Proceedings of the European Conference on Computer Vision (ECCV).
55. Ryan White, Keenan Crane, and D. A. Forsyth. 2007. Capturing and Animating Occluded Cloth. ACM Trans. Graph. 26, 3, Article 34 (July 2007).
56. Chenglei Wu, Kiran Varanasi, and Christian Theobalt. 2012. Full Body Performance Capture Under Uncontrolled and Varying Illumination: A Shading-based Approach. In Proceedings of the 12th European Conference on Computer Vision - Volume Part IV (ECCV'12). Springer-Verlag, Berlin, Heidelberg, 757--770.
57. Stefanie Wuhrer, Leonid Pishchulin, Alan Brunton, Chang Shu, and Jochen Lang. 2014. Estimation of human body shape and posture under clothing. Computer Vision and Image Understanding 127 (2014), 31--42.
58. Feng Xu, Yebin Liu, Carsten Stoll, James Tompkin, Gaurav Bharaj, Qionghai Dai, Hans-Peter Seidel, Jan Kautz, and Christian Theobalt. 2011. Video-based Characters: Creating New Human Performances from a Multi-view Video Database. ACM Trans. Graph. 30, 4, Article 32 (July 2011), 10 pages.
59. Jinlong Yang, Jean-Sébastien Franco, Franck Hétroy-Wheeler, and Stefanie Wuhrer. 2016. Estimation of Human Body Shape in Motion with Wide Clothing. In European Conference on Computer Vision 2016. Amsterdam, Netherlands. https://hal.inria.fr/hal-01344795
60. Chao Zhang, Sergi Pujades, Michael Black, and Gerard Pons-Moll. 2017. Detailed, accurate, human shape estimation from clothed 3D scan sequences. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
61. Bin Zhou, Xiaowu Chen, Qiang Fu, Kan Guo, and Ping Tan. 2013. Garment Modeling from a Single Image. Computer Graphics Forum (2013).

Index Terms

• Computing methodologies

  • Computer graphics

    • Animation

    • Shape modeling

      • Mesh models