Reflectance modeling by neural texture synthesis

Abstract

We extend parametric texture synthesis to capture rich, spatially varying parametric reflectance models from a single image. Our input is a single head-lit flash image of a mostly flat, mostly stationary (textured) surface, and the output is a tile of SVBRDF parameters that reproduce the appearance of the material. No user intervention is required. Our key insight is to make use of a recent, powerful texture descriptor based on deep convolutional neural network statistics for "softly" comparing the model prediction and the examplars without requiring an explicit point-to-point correspondence between them. This is in contrast to traditional reflectance capture that requires pointwise constraints between inputs and outputs under varying viewing and lighting conditions. Seen through this lens, our method is an indirect algorithm for fitting photorealistic SVBRDFs. The problem is severely ill-posed and non-convex. To guide the optimizer towards desirable solutions, we introduce a soft Fourier-domain prior for encouraging spatial stationarity of the reflectance parameters and their correlations, and a complementary preconditioning technique that enables efficient exploration of such solutions by L-BFGS, a standard non-linear numerical optimizer.

References

1. Aittala, M., Weyrich, T., and Lehtinen, J. 2015. Two-shot SVBRDF capture for stationary materials. ACM Trans. Graph. 34, 4, 110:1--110:13.
2. Barron, J., and Malik, J. 2015. Shape, illumination, and reflectance from shading. IEEE TPAMI (to appear).
3. Boivin, S., and Gagalowicz, A. 2001. Image-based rendering of diffuse, specular and glossy surfaces from a single image. In Proc. ACM SIGGRAPH, 107--116.
4. Clark, R., 2010. Crazybump. http://www.crazybump.com, Last access: 16 Jan 2016.
5. Debevec, P., Hawkins, T., Tchou, C., Duiker, H.-P., Sarokin, W., and Sagar, M. 2000. Acquiring the reflectance field of a human face. In Proc. ACM SIGGRAPH, 145--156.
6. Dong, Y., Wang, J., Tong, X., Snyder, J., Lan, Y., Ben-Ezra, M., and Guo, B. 2010. Manifold bootstrapping for SVBRDF capture. ACM Trans. Graph. 29, 4, 98:1--98:10.
7. Dong, Y., Tong, X., Pellacini, F., and Guo, B. 2011. Appgen: interactive material modeling from a single image. ACM Trans. Graph. 30, 6, 146:1--146:10.
8. Efros, A. A., and Freeman, W. T. 2001. Image quilting for texture synthesis and transfer. In Proc. ACM SIGGRAPH, 341--346.
9. Efros, A. A., and Leung, T. K. 1999. Texture synthesis by non-parametric sampling. In Proc. International Conference on Computer Vision (ICCV '99), vol. 2, 1033--1038.
10. Galerne, B., Gousseau, Y., and Morel, J.-M. 2010. Random phase textures: Theory and synthesis. IEEE Transactions in Image Processing 20, 1, 257--267.
11. Gatys, L. A., Ecker, A. S., and Bethge, M. 2015. A neural algorithm of artistic style. CoRR abs/1508.06576.
12. Gatys, L. A., Ecker, A. S., and Bethge, M. 2015. Texture synthesis using convolutional neural networks. In Advances in Neural Information Processing Systems 28.
13. Graham, P., Tunwattanapong, B., Busch, J., Yu, X., Jones, A., Debevec, P., and Ghosh, A. 2013. Measurement-Based Synthesis of Facial Microgeometry. Computer Graphics Forum 32, 2pt3, 335--344.
14. Heeger, D. J., and Bergen, J. R. 1995. Pyramid-based texture analysis/synthesis. In Proc. ACM SIGGRAPH, 229--238.
15. Hertzmann, A., Jacobs, C. E., Oliver, N., Curless, B., and Salesin, D. H. 2001. Image analogies. In Proc. ACM SIGGRAPH, 327--340.
16. Huang, L., Yang, Y., Deng, Y., and Yu, Y 2015. Densebox: Unifying landmark localization with end to end object detection. CoRR abs/1509.04874.
17. Kaspar, A., Neubert, B., Lischinski, D., Pauly, M., and Kopf, J. 2015. Self Tuning Texture Optimization. Computer Graphics Forum 34, 2.
18. Kwatra, V., Essa, I., Bobick, A., and Kwatra, N. 2005. Texture optimization for example-based synthesis. ACM Trans. Graph. 24, 3, 795--802.
19. LeCun, Y., Bottou, L., Bengio, Y., and Haffner, P. 1998. Gradient-based learning applied to document recognition. In Proc. IEEE, 2278--2324.
20. Lensch, H. P. A., Kautz, J., Goesele, M., Heidrich, W., and Seidel, H.-P. 2003. Image-based reconstruction of spatial appearance and geometric detail. ACM Trans. Graph. 22, 2, 234--257.
21. Ngan, A., Durand, F., and Matusik, W. 2005. Experimental analysis of BRDF models. In Proc. Eurographics Symposium on Rendering, 117--226.
22. Portilla, J., and Simoncelli, E. P. 2000. A parametric texture model based on joint statistics of complex wavelet coefficients. Int. J. Comput. Vision 40, 1, 49--70.
23. Rabin, J., Peyré, G., Delon, J., and Bernot, M. 2011. Wasserstein barycenter and its application to texture mixing. In Proc. Scale Space and Variational Methods in Computer Vision (SSVM), vol. 6667, 435--446.
24. Razavian, A. S., Azizpour, H., Sullivan, J., and Carls-son, S. 2014. CNN features off-the-shelf: An astounding baseline for recognition. In Proc. CVPR.
25. Simonyan, K., and Zisserman, A. 2014. Very deep convolutional networks for large-scale image recognition. CoRR abs/1409.1556.
26. Super, B., and Bovik, A. 1995. Shape from texture using local spectral moments. IEEE TPAMI 17, 4.
27. Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., and Rabinovich, A. 2015. Going deeper with convolutions. In Proc. CVPR.
28. Tong, X., Zhang, J., Liu, L., Wang, X., Guo, B., and Shum, H.-Y. 2002. Synthesis of bidirectional texture functions on arbitrary surfaces. ACM Trans. Graph. 21, 3, 665--672.
29. Vedaldi, A., and Lenc, K. 2014. MatConvNet - convolutional neural networks for MATLAB. CoRR abs/1412.4564.
30. Wang, J., Zhao, S., Tong, X., Snyder, J., and Guo, B. 2008. Modeling anisotropic surface reflectance with example-based microfacet synthesis. ACM Trans. Graph. 27, 3, 41:1--41:9.
31. Wang, C.-P., Snavely, N., and Marschner, S. 2011. Estimating dual-scale properties of glossy surfaces from step-edge lighting. ACM Trans. Graph. 30, 6, 172:1--172:12.
32. Wei, L.-Y., Han, J., Zhou, K., Bao, H., Guo, B., and Shum, H.-Y. 2008. Inverse texture synthesis. ACM Trans. Graph. 27, 3, 52:1--52:9.
33. Weyrich, T., Matusik, W., Pfister, H., Bickel, B., Donner, C., Tu, C., McAndless, J., Lee, J., Ngan, A., Jensen, H. W., and Gross, M. 2006. Analysis of human faces using a measurement-based skin reflectance model. ACM Trans. Graph. 25, 3, 1013--1024.
34. Weyrich, T., Lawrence, J., Lensch, H., Rusinkiewicz, S., and Zickler, T. 2009. Principles of appearance acquisition and representation. Foundations and Trends in Computer Graphics and Vision 4, 2, 75--191.
35. Zickler, T., Ramamoorthi, R., Enrique, S., and Belhumeur, P. N. 2006. Reflectance sharing: predicting appearance from a sparse set of images of a known shape. IEEE TPAMI 28, 8, 1287--1302.

Index Terms

• Computing methodologies

  • Computer graphics

    • Image manipulation

      • Texturing

    • Rendering

      • Reflectance modeling

  • Machine learning

    • Machine learning approaches

      • Neural networks