Acquiring the reflectance field of a human face

ABSTRACT

We present a method to acquire the reflectance field of a human face and use these measurements to render the face under arbitrary changes in lighting and viewpoint. We first acquire images of the face from a small set of viewpoints under a dense sampling of incident illumination directions using a light stage. We then construct a reflectance function image for each observed image pixel from its values over the space of illumination directions. From the reflectance functions, we can directly generate images of the face from the original viewpoints in any form of sampled or computed illumination. To change the viewpoint, we use a model of skin reflectance to estimate the appearance of the reflectance functions for novel viewpoints. We demonstrate the technique with synthetic renderings of a person's face under novel illumination and viewpoints.

References

1. 1.ADELSON, E. H., AND BERGEN, J. R. Computational Models of Visual Processing. MIT Press, Cambridge, Mass., 1991, ch. 1. The Plenoptic Function and the Elements of Early Vision.
2. 2.BEIER, Y., AND NEELY, S. Feature-based image metamorphosis. Computer Graphics (Proceedings of SIGGRAPH 92) 26, 2 (July 1992), 35-42.
3. 3.BLANZ, V., AND VETTER, T. A morphable model for the synthesis of 3d faces. Proceedings of SIGGRAPH 99 (August 1999), 187-194.
4. 4.BREGLER, C., COVELL, M., AND SLANEY, M. Video rewrite: Driving visual speech with audio. Proceedings of SIGGRAPH 97 (August 1997), 353-360.
5. 5.BUSBRIDGE, I. W. The Mathematics of Radiative Transfer. Cambridge University Press, Bristol, UK, 1960.
6. 6.COOK, R. L., AND TORRANCE, K. E. A reflectance model for computer graphics. Computer Graphics (Proceedings of SIGGRAPH 81) 15, 3 (August 1981), 307-316.
7. 7.DANA, K. J., GINNEKEN, B., NAYAR, S. K., AND KOENDERINK, J.J. Reflectance and texture of real-world surfaces. In Proc. IEEE Conf. on Comp. Vision and Patt. Recog. (1997), pp. 151-157.
8. 8.DEBEVEC, P. Rendering synthetic objects into real scenes: Bridging traditional and image-based graphics with global illumination and high dynamic range photography. In SIGGRAPH 98 (July 1998).
9. 9.DEBEVEC, P. E., AND MALIK, J. Recovering high dynamic range radiance maps from photographs. In SIGGRAPH 97 (August 1997), pp. 369-378.
10. 10.DEBEVEC, P. E., YU, Y., AND BORSHUKOV, G. D. Efficient viewdependent image-based rendering with projective texture-mapping. In 9th Eurographics workshop on Rendering (June 1998), pp. 105-116.
11. 11.FUA, P., AND MICCIO, C. From regular images to animated heads: A least squares approach. In ECCV98 (1998).
12. 12.GERSHUN, A. Svetovoe Pole (the Light Field, in English). Journal of Mathematics and Physics XVIII (1939), 51--151.
13. 13.GORTLER, S. J., GRZESZCZUK, R., SZELISKI, R., AND COHEN, M. F. The Lumigraph. In SIGGRAPH 96 (1996), pp. 43-54.
14. 14.GUENTER, B., GRIMM, C., WOOD, D., MALVAR, H., AND PIGHIN, F. Making faces. Proceedings of SIGGRAPH 98 (July 1998), 55-66.
15. 15.HAEBERLI, P. Synthetic lighting for photography. Available at http://www.sgi.com/grafica/synth/index.html, January 1992.
16. 16.HANRAHAN, P., AND KRUEGER, W. Reflection from layered surfaces due to subsurface scattering. Proceedings of SIGGRAPH 93 (August 1993), 165-174.
17. 17.KARNER, K. F., MAYER, H., AND GERVAUTZ, M. An image based measurement system for anisotropic reflection. In EUROGRAPHICS Annual Conference Proceedings (1996).
18. 18.KAUTZ, J., AND McCOOL, M. D. Interactive rendering with arbitrary BRDFs using separable approximations. Eurographics Rendering Workshop 1999 (June 1999).
19. 19.LAFORTUNE, E. P. F., FOG, S.-C., TORRANCE, K. E., AND GREENBERG, D. P. Non-linear approximation of reflectance functions. Proceedings of SIGGRAPH 97 (August 1997), 117-126.
20. 20.LEE, Y., TERZOPOULOS, D., AND WATERS, K. Realistic modeling for facial animation. Proceedings of SIGGRAPH 95 (August 1995), 55-62.
21. 21.LEVOY, M., AND HANRAHAN, P. Light field rendering. In SIG- GRAPH 96 (1996), pp. 31-42.
22. 22.MARSCHNER, S. R., WESTIN, S. H., LAFORTUNE, E. P. F., TOR- RANCE, K. E., AND GREENBERG, D. P. Image-based BRDF measurement including human skin. Eurographics Rendering Workshop 1999 (June 1999).
23. 23.MILLER, G. S. P., RUBIN, S., AND PONCELEON, D. Lazy decompression of surface light fields for precomputed global illumination. Eurographics Rendering Workshop 1998 (June 1998), 281-292.
24. 24.NAYAR, S., FANG, X., AND BOULT, T. Separation of reflection components using color and polarization. IJCV 21, 3 (February 1997), 163-186.
25. 25.NICODEMUS, F. E., RICHMOND, J. C., HSIA, J. J., GINSBERG, I. W., AND LIMPERIS, T. Geometric considerations and nomenclature for reflectance.
26. 26.NIMEROFF, J. S., SIMONCELLI, E., AND DORSEY, J. Efficient rerendering of naturally illuminated environments. Fifth Eurographics Workshop on Rendering (June 1994), 359-373.
27. 27.OREN, M., AND NAYAR, S. K. Generalization of Lambert's reflectance model. Proceedings of SIGGRAPH 94 (July 1994), 239- 246.
28. 28.PARKE, F. I. Computer generated animation of faces. Proc. ACM annual conf. (August 1972).
29. 29.PIGHIN, F., HECKER, J., LISCHINSKI, D., SZELISKI, R., AND SALESIN, D. U. Synthesizing realistic facial expressions from photographs. Proceedings of SIGGRAPH 98 (July 1998), 75-84.
30. 30.SAGAR, M. A., BULLIVANT, D., MALLINSON, G. D., HUNTER, P. J., AND HUNTER, I. W. A virtual environment and model of the eye for surgical simulation. Proceedings of SIGGRAPH 94 (July 1994), 205-213.
31. 31.SATO, Y., AND IKEUCHI, K. Temporal-color space analysis of reflection. JOSA-A 11, 11 (November 1994), 2990-3002.
32. 32.SATO, Y., WHEELER, M. D., AND IKEUCHI, K. Object shape and reflectance modeling from observation. In SIGGRAPH 97 (1997), pp. 379-387.
33. 33.SMITH, B., AND ROWE, L. Compressed domain processing of JPEG- encoded images. Real-Time Imaging 2, 2 (1996), 3-17.
34. 34.TERZOPOULOS, D., AND WATERS, K. Physically-based facial modelling, analysis, and animation. Journal of Visualization and Computer Animation 1, 2 (August 1990), 73-80.
35. 35.TORRANCE, K. E., AND SPARROW, g. M. Theory for off-specular reflection from roughened surfaces. Journal of Optical Society of America 57, 9 (1967).
36. 36.VAN GEMERT, M. F. C., JACQUES, S. L., STERENBERG, H. J. C. M., AND STAR, W.M. Skin optics. IEEE Transactions on Biomedical Engineering 36, 12 (December 1989), 1146-1154.
37. 37.WARD, G. J. Measuring and modeling anisotropic reflection. In SIG- GRAPH 92 (July 1992), pp. 265-272.
38. 38.WILLIAMS, L. Performance-driven facial animation. Computer Graphics (Proceedings of SIGGRAPH 90) 24, 4 (August 1990), 235- 242.
39. 39.WONG, T.-T., HERO, P.-A., OR, S.-H., AND NO, W.-Y. Imagebased rendering with controllable illumination. Eurographics Rendering Workshop 1997 (June 1997), 13-22.
40. 40.Wu, Y., THALMANN, N. M., AND THALMANN, D. A dynamic wrinkle model in facial animation and skin aging. Journal of Visualization and Computer Animation 6, 4 (October 1995), 195-206.
41. 41.Yu, Y., DEBEVEC, P., MALIK, J., AND HAWKINS, T. Inverse global illumination: Recovering reflectance models of real scenes from photographs. Proceedings of SIGGRAPH 99 (August 1999), 215-224.
42. 42.ZONGKER, D. E., WERNER, D. M., CURLESS, B., AND SALESIN, D.H. Environment matting and compositing. Proceedings of SIG- GRAPH 99 (August 1999), 205-214.

Index Terms

• Computing methodologies

  • Artificial intelligence

    • Computer vision

      • Computer vision problems

      • Computer vision tasks

        • Scene understanding

  • Computer graphics

    • Image manipulation

      • Texturing

    • Rendering

      • Reflectance modeling

• Human-centered computing

  • Human computer interaction (HCI)