A morphable model for the synthesis of 3D faces

ABSTRACT

In this paper, a new technique for modeling textured 3D faces is introduced. 3D faces can either be generated automatically from one or more photographs, or modeled directly through an intuitive user interface. Users are assisted in two key problems of computer aided face modeling. First, new face images or new 3D face models can be registered automatically by computing dense one-to-one correspondence to an internal face model. Second, the approach regulates the naturalness of modeled faces avoiding faces with an "unlikely" appearance.

Starting from an example set of 3D face models, we derive a morphable face model by transforming the shape and texture of the examples into a vector space representation. New faces and expressions can be modeled by forming linear combinations of the prototypes. Shape and texture constraints derived from the statistics of our example faces are used to guide manual modeling or automated matching algorithms.

We show 3D face reconstructions from single images and their applications for photo-realistic image manipulations. We also demonstrate face manipulations according to complex parameters such as gender, fullness of a face or its distinctiveness.

References

1. 1.T. Akimoto, Y. Suenaga, and R.S. Wallace. Automatic creation of 3D facial models. IEEE Computer Graphics and Applications, 13 (3): 16-22, 1993.
2. 2.J.R. Bergen and R. Hingorani. Hierarchical motion-based frame rate conversion. Technical report, David Sarnoff Research Center Princeton NJ 08540, 1990.
3. 3.R Bergeron and R Lachapelle. Controlling facial expressions and body movements. In Advanced Computer Animation, SIGGRAPH '85 Tutorials, volume 2, pages 61-79, New York, 1985. ACM.
4. 4.D. Beymer and T. Poggio. Image representation for visual learning. Science, 272:1905-1909, 1996.
5. 5.D. Beymer, A. Shashua, and T. Poggio. Example-based image analysis and synthesis. A.I. Memo No. 1431, Artificial Intelligence Laboratory, Massachusetts Institute of Technology, 1993.
6. 6.S.E. Brennan. The caricature generator. Leonardo, 18:170-178, 1985.
7. 7.RJ. Burt and E.H. Adelson. Merging images through pattern decomposition. In Applications of Digital Image Processing VIII, number 575, pages 173-181. SPIE The International Society for Optical Engeneering, 1985.
8. 8.C.S. Choi, T. Okazaki, H. Harashima, and T. Takebe. A system of analyzing and synthesizing facial images. In Proc. IEEE Int. Symposium of Circuit and Syatems (ISCAS91), pages 2665-2668, 1991.
9. 9.T.F. Cootes, G.J. Edwards, and C.J. Taylor. Active appearance models. In Burkhardt and Neumann, editors, Computer Vision -ECCV'98 Vol. H, Freiburg, Germany, 1998. Springer, Lecture Notes in Computer Science 1407.
10. 10.D. DeCarlos, D. Metaxas, and M. Stone. An anthropometric lace model using variational techniques. In Computer Graphics Proceedings SIGGRAPH'98, pages 67-74, 1998.
11. 11.S. DiPaola. Extending the range of facial types. Journal of Visualization and Computer Animation, 2(4): 129-131, 1991.
12. 12.G.J. Edwards, A. Lanitis, C.J. Taylor, and T.F. Cootes. Modelling the variability in face images. In P~vc. of the 2nd Int. Conf. on Automatic Face and Gesture Recognition, IEEE Comp. Soc. Press, Los Alamitos, CA, 1996.
13. 13.L.G. Farkas. Anthropometry of the Head and Face. RavenPress, New York, 1994.
14. 14.B. Guenter, C. Grimm, D. Wolf, H. Malvar, and F. Pighin. Making faces. In Computer Graphics Proceedings SIGGRAPH'98, pages 55-66, 1998.
15. 15.I.T. Jollife. Principal Component Analysis. Springer-Verlag, New York, 1986.
16. 16.M. Jones and T. Poggio. Multidimensional morphable models: A framework for representing and matching object classes. In Proceedings of the Sixth International Conference on Computer Vision, Bombay, India, 1998.
17. 17.R. M. Koch, M. H. Gross, and A. A. Bosshard. Emotion editing using finite elements. In Proceedings of the Eurographics '98, COMPUTER GRAPHICS Forum, Vol. 17, No. 3, pages C295-C302, Lisbon, Portugal, 1998.
18. 18.A. Lanitis, C.J. Taylor, and T.F. Cootes. Automatic interpretation and coding of face images using flexible models. IEEE Transactions on Pattern Analysis and Machine Intelligence, 19(7):743-756, 1997.
19. 19.Y.C. Lee, D. Terzopoulos, and Keith Waters. Constructing physics-based facial models of individuals. Visual Computer, Proceedings of Graphics Interface '93:1-8, 1993.
20. 20.Y.C. Lee, D. Terzopoulos, and Keith Waters. Realistic modeling for facial animation. In SIGGRAPH '95 Conference P1vceedings, pages 55-62, Los Angels, 1995. ACM.
21. 21.J. R Lewis. Algorithms for solid noise synthesis. In SIGGRAPH '89 Conference P~vceedings, pages 263-270. ACM, 1989.
22. 22.N. Magneneat-Thalmann, H. Minh, M. Angelis, and D. Thalmann. Design, transformation and animation of human faces. Visual Computer, 5:32-39, 1989.
23. 23.L. Moccozet and N. Magnenat-Thalmann. Dirichlet free-form deformation and their application to hand simulation. In Computer Animation'97, 1997.
24. 24.F.I. Parke and K. Waters. Computer Facial Animation. AKPeters, Wellesley, Massachusetts, 1996.
25. 25.EI. Parke. Computer generated animation of faces. In ACM National Conference. ACM, November 1972.
26. 26.F.I. Parke. A Parametric Model of Human Faces. PhD thesis, University of Utah, Salt Lake City, 1974.
27. 27.M. Petrow, A. Talapov, T. Robertson, A. Lebedev, A. Zhilyaev, and L. Polonskiy. Optical 3D digitizer: Bringing life to virtual world. IEEE Computer Graphics and Applications, 18(3):28-37, 1998.
28. 28.F. Pighin, J. Hecker, D. Lischinski, Szeliski R, and D. Salesin. Synthesizing realistic facial expressions from photographs. In Computer Graphics Proceedings SIGGRAPH'98, pages 75-84, 1998.
29. 29.S. Platt and N. Badler. Animating facial expression. Computer Graphics, 15(3):245-252, 1981.
30. 30.G. Sannier and N. Magnenat-Thalmann. A user-friendly texture-fitting methodology tor virtual humans. In Computer Graphics International'97, 1997.
31. 31.L. Sirovich and M. Kirby. Low-dimensional procedure tor the characterization of human laces. Journal of the Optical Society of America A, 4:519-554, 1987.
32. 32.D. Terzopoulos and Keith Waters. Physically-based facial modeling, analysis, and animation. Visualization and Computer Animation, 1:73-80, 1990.
33. 33.Demetri Terzopoulos and Hong Qin. Dynamic NURBS with geometric constraints to interactive sculpting. ACM Transactions on Graphics, 13 (2): 103-136, April 1994.
34. 34.J.T. Todd, S. M. Leonard, R. E. Shaw, and J. B. Pittenger. The perception of human growth. Scientific American, 1242:106-114, 1980.
35. 35.T. Vetter and V. Blanz. Estimating coloured 3d lace models from single images: An example based approach. In Burkhardt and Neumann, editors, Computer Vision - ECCV'98 Vol. H, Freiburg, Germany, 1998. Springer, Lecture Notes in Computer Science 1407.
36. 36.T. Vetter, M. J. Jones, and T. Poggio. A bootstrapping algorithm tor learning linear models of object classes. In IEEE Conference on Computer Vision and Pattern Recognition - CVPR'97, Puerto Rico, USA, 1997. IEEE Computer Society Press.
37. 37.T. Vetter and T. Poggio. Linear object classes and image synthesis from a single example image. IEEE Transactions on Pattern Analysis and Machine Intelligence, 19(7):733-742, 1997.
38. 38.Keith Waters. A muscle model for animating three-dimensional facial expression. Computer Graphics, 22(4):17-24, 1987.

Index Terms

• Computing methodologies

  • Artificial intelligence

    • Computer vision

      • Image and video acquisition

        • 3D imaging

  • Computer graphics

    • Animation