High-quality streamable free-viewpoint video

Abstract

We present the first end-to-end solution to create high-quality free-viewpoint video encoded as a compact data stream. Our system records performances using a dense set of RGB and IR video cameras, generates dynamic textured surfaces, and compresses these to a streamable 3D video format. Four technical advances contribute to high fidelity and robustness: multimodal multi-view stereo fusing RGB, IR, and silhouette information; adaptive meshing guided by automatic detection of perceptually salient areas; mesh tracking to create temporally coherent subsequences; and encoding of tracked textured meshes as an MPEG video stream. Quantitative experiments demonstrate geometric accuracy, texture fidelity, and encoding efficiency. We release several datasets with calibrated inputs and processed results to foster future research.

References

1. 4D View Solutions, 2007. http://www.4dviews.com.
2. Ahmed, N., Theobalt, C., Dobrev, P., and Seidel, H. 2008. Robust fusion of dynamic shape and normal capture for high-quality reconstruction of time-varying geometry. In Proc. CVPR.
3. Ahmed, N., Theobalt, C., Rossl, C., Thrun, S., and Seidel, H. 2008. Dense correspondence finding for parameterization-free animation reconstruction from video. In Proc. CVPR.
4. Alexa, M., Behr, J., Cohen-Or, D., Fleishman, S., Levin, D., and Silva, C. T. 2001. Point set surfaces. In Proc. Conf. on Visualization.
5. Aspert, N., Santa-cruz, D., and Ebrahimi, T. 2002. MESH: Measuring errors between surfaces using the Hausdorff distance. In Proc. ICME.
6. Bleyer, M., Rhemann, C., and Rother, C. 2011. PatchMatch stereo - stereo matching with slanted support windows. In Proc. BMVC.
7. Bojsen-Hansen, M., Li, H., and Wojtan, C. 2012. Tracking surfaces with evolving topology. ACM Trans. Graph. 31, 4.
8. Borshukov, G., Piponi, D., Larsen, O., Lewis, J. P., and Tempelaar-Lietz, C. 2005. Universal capture -- Image-based facial animation for "The Matrix Reloaded". In ACM SIGGRAPH Courses.
9. Budd, C., Huang, P., Klaudiny, M., and Hilton, A. 2013. Global non-rigid alignment of surface sequences. Int. J. Comput. Vision 102, 1--3.
10. Campbell, N. D. F., Vogiatzis, G., Hernandez, C., and Cipolla, R. 2008. Using multiple hypotheses to improve depth-maps for multi-view stereo. In Proc. ECCV.
11. Carranza, J., Theobalt, C., Magnor, M. A., and Seidel, H.-P. 2003. Free-viewpoint video of human actors. ACM Trans. Graph. 22, 3.
12. Casas, D., Volino, M., Collomosse, J., and Hilton, A. 2014. 4D video textures for interactive character appearance. Comput. Graph. Forum 33, 2.
13. Chuang, M., Luo, L., Brown, B., Rusinkiewicz, S., and Kazhdan, M. 2009. Estimating the Laplace-Beltrami operator by restricting 3D functions. Symposium on Geometry Processing.
14. de Aguiar, E., Stoll, C., Theobalt, C., Ahmed, N., Seidel, H.-P., and Thrun, S. 2008. Performance capture from sparse multi-view video. ACM Trans. Graph. 27, 3.
15. DoubleMe, 2014. https://www.doubleme.me.
16. Erickson, J., and Whittlesey, K. 2005. Greedy optimal homotopy and homology generators. In Proc. ACM-SIAM Symposium on Discrete algorithms.
17. Franco, J., Lapierre, M., and Boyer, E. 2006. Visual shapes of silhouette sets. In Proc. Intl. Symp. 3D Data Processing, Visualization and Transmission.
18. FreeD, 2014. http://replay-technologies.com.
19. Furukawa, Y., and Ponce, J. 2010. Accurate, dense, and robust multiview stereopsis. IEEE PAMI 32, 8.
20. Gal, R., Wexler, Y., Ofek, E., Hoppe, H., and Cohen-Or, D. 2010. Seamless montage for texturing models. Comput. Graph. Forum 29, 2.
21. Gall, J., Stoll, C., Aguiar, E. D., Theobalt, C., Rosenhahn, B., and peter Seidel, H. 2009. Motion capture using joint skeleton tracking and surface estimation. In Proc. CVPR.
22. Garland, M., and Heckbert, P. S. 1997. Surface simplification using quadric error metrics. In ACM SIGGRAPH.
23. Goesele, M., Curless, B., and Seitz, S. M. 2006. Multi-view stereo revisited. In Proc. CVPR.
24. Goldluecke, B., and Magnor, M. 2004. Space-time isosurface evolution for temporally coherent 3D reconstruction. In Proc. CVPR.
25. Golomb, S. 1966. Run-length encodings (corresp.). IEEE Transactions on Information Theory 12, 3.
26. Guennebaud, G., Jacob, B., et al., 2010. Eigen v3. http://eigen.tuxfamily.org.
27. Guskov, I., and Wood, Z. J. 2001. Topological noise removal. In Proc. Graphics Interface.
28. Hernandez, C., and Schmitt, F. 2004. Silhouette and stereo fusion for 3D object modeling. Computer Vision and Image Understanding 96, 3.
29. Hiep, V. H., Keriven, R., Labatut, P., and Pons, J.-P. 2009. Towards high-resolution large-scale multi-view stereo. In Proc. CVPR.
30. Hu, X., and Mordohai, P. 2012. A quantitative evaluation of confidence measures for stereo vision. IEEE PAMI 34, 11.
31. Huang, C.-H., Boyer, E., Navab, N., and Ilic, S. 2014. Human shape and pose tracking using keyframes. In Proc. CVPR.
32. ISO/IEC 23009-1, 2014. Information technology -- dynamic adaptive streaming over HTTP (DASH) -- Part 1: Media presentation description and segment formats.
33. Kanade, T., Rander, P., and Narayanan, P. J. 1997. Virtualized reality: Constructing virtual worlds from real scenes. IEEE Multimedia 4, 1.
34. Kazhdan, M., and Hoppe, H. 2013. Screened Poisson surface reconstruction. ACM Trans. Graph. 32, 3.
35. Kazhdan, M., Bolitho, M., and Hoppe, H. 2006. Poisson surface reconstruction. In Symposium on Geometry Processing.
36. Klaudiny, M., Budd, C., and Hilton, A. 2012. Towards optimal non-rigid surface tracking. In Proc. ECCV.
37. Labatut, P., Pons, J.-P., and Keriven, R. 2007. Efficient multi-view reconstruction of large-scale scenes using interest points, delaunay triangulation and graph cuts. In Proc. ICCV.
38. Lee, C. H., Varshney, A., and Jacobs, D. W. 2005. Mesh saliency. ACM Trans. Graph. 24, 3.
39. Lempitsky, V. S., and Ivanov, D. V. 2007. Seamless mosaicing of image-based texture maps. In Proc. CVPR.
40. Letouzey, A., and Boyer, E. 2012. Progressive shape models. In Proc. CVPR.
41. Li, H., Adams, B., Guibas, L. J., and Pauly, M. 2009. Robust single-view geometry and motion reconstruction. ACM Trans. Graph. 28, 5.
42. Lindstrom, P., and Turk, G. 2000. Image-driven simplification. ACM Trans. Graph. 19, 3.
43. Liu, Y., Dai, Q., and Xu, W. 2010. A point-cloud-based multiview stereo algorithm for free-viewpoint video. IEEE TVCG.
44. Matusik, W., Buehler, C., Raskar, R., Gortler, S. J., and McMillan, L. 2000. Image-based visual hulls. In ACM SIGGRAPH.
45. Microsoft, 2011. UVAtlas. http://uvatlas.codeplex.com.
46. Moezzi, S., Tai, L.-C., and Gerard, P. 1997. Virtual view generation for 3D digital video. IEEE Multimedia 4, 1.
47. Narayanan, P., Rander, P., and Kanade, T. 1998. Constructing virtual worlds using dense stereo. In Proc. ICCV.
48. Shan, Q., Curless, B., Furukawa, Y., Hernandez, C., and Seitz, S. M. 2014. Occluding contours for multi-view stereo. In Proc. ECCV.
49. Sinha, S. N., and Pollefeys, M. 2005. Multi-view reconstruction using photo-consistency and exact silhouette constraints: a maximum-flow formulation. In Proc. ICCV.
50. Song, P., Wu, X., and Wang, M. Y. 2010. Volumetric stereo and silhouette fusion for image-based modeling. The Visual Computer 26, 12.
51. Starck, J., and Hilton, A. 2007. Surface capture for performance-based animation. IEEE Computer Graphics and Application 27, 6.
52. Sumner, R. W., Schmid, J., and Pauly, M. 2007. Embedded deformation for shape manipulation. ACM Trans. Graph. 26, 3.
53. Vasa, L., and Skala, V. 2007. CoDDyaC: Connectivity Driven Dynamic Mesh Compression. In Proc. 3DTV.
54. Vlasic, D., Baran, I., Matusik, W., and Popovic, J. 2008. Articulated mesh animation from multiview silhouettes. ACM Trans. Graph. 27, 3.
55. Vlasic, D., Peers, P., Baran, I., Debevec, P., Popović, J., Rusinkiewicz, S., and Matusik, W. 2009. Dynamic shape capture using multi-view photometric stereo. ACM Trans. Graph. 28, 5.
56. Volino, M., Casas, D., Collomosse, J. P., and Hilton, A. 2014. Optimal representation of multiple view video. In Proc. BMVC.
57. Wand, M., Adams, B., Ovsjanikov, M., Berner, A., Bokeloh, M., Jenke, P., Guibas, L., Seidel, H.-P., and Schilling, A. 2009. Efficient reconstruction of nonrigid shape and motion from real-time 3D scanner data. ACM Trans. Graph. 28, 2.
58. Wang, Z., Bovik, A. C., Sheikh, H. R., and Simoncelli, E. P. 2004. Image quality assessment: From error visibility to structural similarity. IEEE Trans. Image Proc. 13, 4.
59. Wood, Z., Hoppe, H., Desbrun, M., and Schröder, P. 2004. Removing excess topology from isosurfaces. ACM Trans. Graph. 23, 2.
60. Wu, C., Varanasi, K., Liu, Y., Seidel, H.-P., and Theobalt, C. 2011. Shading--based dynamic shape refinement from multi-view video under general illumination. In Proc. ICCV.
61. Ye, G., Liu, Y., Deng, Y., Hasler, N., Ji, X., Dai, Q., and Theobalt, C. 2013. Free-viewpoint video of human actors using multiple handheld Kinects. IEEE Trans. on System, Man & Cybernetics 43, 5.
62. Yu, F., Luo, H., Lu, Z., and Wang, P. 2010. 3D mesh compression. Three-Dimensional Model Analysis and Processing.
63. Zhou, Q.-Y., and Koltun, V. 2014. Color map optimization for 3D reconstruction with consumer depth cameras. ACM Trans. Graph. 33, 4.
64. Zitnick, C. L., Kang, S. B., Uyttendaele, M., Winder, S., and Szeliski, R. 2004. High-quality video view interpolation using a layered representation. ACM Trans. Graph. 23, 3.
65. Zollhöfer, M., Niessner, M., Izadi, S., Rehmann, C., Zach, C., Fisher, M., Wu, C., Fitzgibbon, A., Loop, C., Theobalt, C., and Stamminger, M. 2014. Real-time non-rigid reconstruction using an RGB-D camera. ACM Trans. Graph. 33, 4.

Index Terms

• Computing methodologies

  • Computer graphics