Image analysis and graphics synthesis can be achieved with learning techniques using directly image examples without physically-based, 3D models. In our technique: -- the mapping from novel images to a vector of ``pose'''' and ``expression'''' parameters can be learned from a small set of example images using a function approximation technique that we call an {\it analysis network}; -- the inverse mapping from input ``pose'''' and ``expression'''' parameters to output images can be synthesized from a small set of example images and used to produce new images using a similar {\it synthesis network}. The techniques described here have several applications in computer graphics, special effects, interactive multimedia and very low bandwidth teleconferencing.
Cited By
- Blanz V and Vetter T A Morphable Model For The Synthesis Of 3D Faces Seminal Graphics Papers: Pushing the Boundaries, Volume 2, (157-164)
- Buck I, Finkelstein A, Jacobs C, Klein A, Salesiny D, Seims J, Szeliski R and Toyama K Performance-driven hand-drawn animation ACM SIGGRAPH 2006 Courses, (25-es)
- Park J and Lee S Stepwise reconstruction of high-resolution facial image based on interpolated morphable face model Proceedings of the 5th international conference on Audio- and Video-Based Biometric Person Authentication, (102-111)
- Park J and Lee S Resolution enhancement of facial image based on top-down learning First ACM SIGMM international workshop on Video surveillance, (59-64)
- Hwang B, Park J and Lee S Synthesis of high-resolution facial image based on top-down learning Proceedings of the 4th international conference on Audio- and video-based biometric person authentication, (377-385)
- Hwang B and Lee S (2003). Reconstruction of Partially Damaged Face Images Based on a Morphable Face Model, IEEE Transactions on Pattern Analysis and Machine Intelligence, 25:3, (365-372), Online publication date: 1-Mar-2003.
- Ezzat T, Geiger G and Poggio T Trainable videorealistic speech animation Proceedings of the 29th annual conference on Computer graphics and interactive techniques, (388-398)
- Shelton C (2019). Morphable Surface Models, International Journal of Computer Vision, 38:1, (75-91), Online publication date: 30-Jun-2000.
- Ezzat T and Poggio T (2019). Visual Speech Synthesis by Morphing Visemes, International Journal of Computer Vision, 38:1, (45-57), Online publication date: 30-Jun-2000.
- Poggio T and Verri A (2019). Introduction, International Journal of Computer Vision, 38:1, (5-7), Online publication date: 30-Jun-2000.
- Giese M and Poggio T (2019). Morphable Models for the Analysis and Synthesis of Complex Motion Patterns, International Journal of Computer Vision, 38:1, (59-73), Online publication date: 30-Jun-2000.
- Buck I, Finkelstein A, Jacobs C, Klein A, Salesin D, Seims J, Szeliski R and Toyama K Performance-driven hand-drawn animation Proceedings of the 1st international symposium on Non-photorealistic animation and rendering, (101-108)
- Donato G, Bartlett M, Hager J, Ekman P and Sejnowski T (1999). Classifying Facial Actions, IEEE Transactions on Pattern Analysis and Machine Intelligence, 21:10, (974-989), Online publication date: 1-Oct-1999.
- Blanz V and Vetter T A morphable model for the synthesis of 3D faces Proceedings of the 26th annual conference on Computer graphics and interactive techniques, (187-194)
- Pollard S and Hayes S View synthesis by edge transfer with application to the generation of immersive video objects Proceedings of the ACM symposium on Virtual reality software and technology, (91-98)
- Avidan S and Shashua A (1998). Novel View Synthesis by Cascading Trilinear Tensors, IEEE Transactions on Visualization and Computer Graphics, 4:4, (293-306), Online publication date: 1-Oct-1998.
- Jones M and Poggio T (1998). Multidimensional Morphable Models, International Journal of Computer Vision, 29:2, (107-131), Online publication date: 29-Aug-1998.
- Vetter T (1998). Synthesis of Novel Views from a Single Face Image, International Journal of Computer Vision, 28:2, (103-116), Online publication date: 1-Jun-1998.
- Black M and Yacoob Y (1997). Recognizing Facial Expressions in Image Sequences Using Local Parameterized Models of Image Motion, International Journal of Computer Vision, 25:1, (23-48), Online publication date: 1-Oct-1997.
- Avidan S, Evgeniou T, Shashua A and Poggio T Image-based view synthesis by combining trilinear tensors and learning techniques Proceedings of the ACM symposium on Virtual reality software and technology, (103-110)
- Shashua A and Navab N (1996). Relative Affine Structure, IEEE Transactions on Pattern Analysis and Machine Intelligence, 18:9, (873-883), Online publication date: 1-Sep-1996.
- Yacoob Y and Davis L (1996). Recognizing Human Facial Expressions From Long Image Sequences Using Optical Flow, IEEE Transactions on Pattern Analysis and Machine Intelligence, 18:6, (636-642), Online publication date: 1-Jun-1996.
Recommendations
Image-Based Texture Analysis for Realistic Image Synthesis
SIBGRAPI '00: Proceedings of the 13th Brazilian Symposium on Computer Graphics and Image ProcessingWe present a method to measure reflectance and texture of surfaces in a one step process. For later use in digital image synthesis, it is mandatory to separate the gathered intensity values into these two parts to eliminate highlighting artifacts from ...
Example-based synthesis of three-dimensional clouds from photographs
CGI '17: Proceedings of the Computer Graphics International ConferenceIn this paper, we propose a modeling method of clouds from a single photograph. To synthesize realistic images of outdoor scenes, modeling of 3D clouds is necessary. However, synthesizing realistic clouds is still a difficult task. To address this ...
Photometric Image-Based Rendering for Virtual Lighting Image Synthesis
IWAR '99: Proceedings of the 2nd IEEE and ACM International Workshop on Augmented RealityA concept named Photometric Image-Based Rendering (PIBR) is introduced for a seamless augmented reality. The PIBR is defined as Image-Based Rendering which covers appearance changes caused by the lighting condition changes, while Geometric Image-Based ...