Video textures

ABSTRACT

This paper introduces a new type of medium, called a video texture, which has qualities somewhere between those of a photograph and a video. A video texture provides a continuous infinitely varying stream of images. While the individual frames of a video texture may be repeated from time to time, the video sequence as a whole is never repeated exactly. Video textures can be used in place of digital photos to infuse a static image with dynamic qualities and explicit actions. We present techniques for analyzing a video clip to extract its structure, and for synthesizing a new, similar looking video of arbitrary length. We combine video textures with view morphing techniques to obtain 3D video textures. We also introduce video-based animation, in which the synthesis of video textures can be guided by a user through high-level interactive controls. Applications of video textures and their extensions include the display of dynamic scenes on web pages, the creation of dynamic backdrops for special effects and games, and the interactive control of video-based animation.

References

1. 1.Z. Bar-Joseph. Statistical learning of multi-dimensional textures. Master's thesis, The Hebrew University of Jerusalem, June 1999.
2. 2.D. Beymer. Feature correspondence by interleaving shape and texture computations. In IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'96), pages 921-928, San Francisco, California, June 1996.
3. 3.C. Bregler, M. Covell, and M. Slaney. Video rewrite: Driving visual speech with audio. Computer Graphics (SIG-GRAPH'97), pages 353-360, August 1997.
4. 4.P. J. Burt and E. H. Adelson. A multiresolution spline with applications to image mosaics. ACM Transactions on Graphics, 2(4):217-236, October 1983.
5. 5.S. E. Chen. QuickTime VR - an image-based approach to virtual environment navigation. Computer Graphics (SIG- GRAPH'95), pages 29-38, August 1995.
6. 6.J. De Bonet. Multiresolution sampling procedure for analysis and synthesis of texture images. Computer Graphics (SIG- GRAPH'97), pages 361-368, August 1997.
7. 7.P. Debevec et al., editors. Image-Based Modeling, Rendering, and Lighting, SIGGRAPH'99 Course 39, August 1999.
8. 8.P. E. Debevec, C. J. Taylor, and J. Malik. Modeling and rendering architecture from photographs: A hybrid geometry- and image-based approach. Computer Graphics (SIGGRAPH'96), pages 11-20, August 1996.
9. 9.A. A. Efros and T. K. Leung. Texture synthesis by nonparametric sampling. In Seventh International Conference on Computer Vision (ICCV'99), pages 1033-1038, Kerkyra, Greece, September 1999.
10. 10.A. Finkelstein, C. E. Jacobs, and D. H. Salesin. Multiresolution video. Proceedings of SIGGRAPH 96, pages 281-290, August 1996. ISBN 0-201-94800-1. Held in New Orleans, Louisiana.
11. 11.W. T. Freeman, E. H. Adelson, and D. J. Heeger. Motion without movement. Computer Graphics (Proceedings of SIG- GRAPH 91), 25(4):27-30, July 1991.
12. 12.D. J. Heeger and J. R. Bergen. Pyramid-based texture analysis/synthesis. Proceedings of SIGGRAPH 95, pages 229-238, August 1995.
13. 13.Workshop on Image-Based Modeling and Rendering, Stanford University, March 1998. http://graphics.stanford.edu/workshops/ibr98/.
14. 14.C. E. Jacobs, A. Finkelstein, and D. H. Salesin. Fast multiresolution image querying. Proceedings of SIGGRAPH 95, pages 277-286, August 1995.
15. 15.L. P. Kaelbling, M. L. Littman, and A. W. Moore. Reinforcement learning: A survey. Journal of Artificial Intelligence Research, 4, 1996.
16. 16.T. Kanade, P. W. Rander, and P. J. Narayanan. Virtualized reality: constructing virtual worlds from real scenes. IEEE MultiMedia Magazine, 1(1):34-47, Jan-March 1997.
17. 17.M. Levoy and P. Hanrahan. Light field rendering. In Computer Graphics Proceedings, Annual Conference Series, pages 31- 42, Proc. SIGGRAPH'96 (New Orleans), August 1996. ACM SIGGRAPH.
18. 18.L. McMillan and G. Bishop. Plenoptic modeling: An image-based rendering system. Computer Graphics (SIG- GRAPH'95), pages 39-46, August 1995.
19. 19.S. Moezzi et al. Reality modeling and visualization from multiple video sequences. IEEE Computer Graphics and Applications, 16(6):58-63, November 1996.
20. 20.S. A. Niyogi and E. H. Adelson. Analyzing and recognizing walking figures in xyt. In IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'94), pages 469-474, Seattle, Washington, June 1994.
21. 21.F. Pighin, J. Hecker, D. Lischinski, D. H. Salesin, and R. Szeliski. Synthesizing realistic facial expressions from photographs. In Computer Graphics (SIGGRAPH'98) Proceedings, pages 75-84, Orlando, July 1998. ACM SIGGRAPH.
22. 22.R. Polana and R. C. Nelson. Detection and recognition of periodic, nonrigid motion. International Journal of Computer Vision, 23(3):261-282, 1997.
23. 23.S. Pollard et al. View synthesis by trinocular edge matching and transfer. In British Machine Vision Conference (BMVC98), Southampton, England, September 1998.
24. 24.S. M. Seitz and C. M. Dyer. View morphing. In Computer Graphics Proceedings, Annual Conference Series, pages 21- 30, Proc. SIGGRAPH'96 (New Orleans), August 1996. ACM SIGGRAPH.
25. 25.S. M. Seitz and C. R. Dyer. View invariant analysis of cyclic motion. International Journal of Computer Vision, 25(3):231- 251, December 1997.
26. 26.J. Shade, S. Gortler, L.-W. He, and R. Szeliski. Layered depth images. In Computer Graphics (SIGGRAPH'98) Proceedings, pages 231-242, Orlando, July 1998. ACM SIGGRAPH.
27. 27.J. Shade, D. Lischinski, D. Salesin, T. DeRose, and J. Snyder. Hierarchical images caching for accelerated walkthroughs of complex environments. In Computer Graphics (SIG-GRAPH'96) Proceedings, pages 75-82, Proc. SIGGRAPH'96 (New Orleans), August 1996. ACM SIGGRAPH.
28. 28.B. Shaw. Light of other days. In Other Days, Other Eyes. Ace Books, NewYork, 1972. (also published in Analog and various sci-fi anthologies).
29. 29.H.-Y. Shum and R. Szeliski. Construction of panoramic mosaics with global and local alignment. International Journal of Computer Vision, 36(2):101-130, February 2000.
30. 30.A. R. Smith and J. F. Blinn. Blue screen matting. In Computer Graphics Proceedings, Annual Conference Series, pages 259- 268, Proc. SIGGRAPH'96 (NewOrleans), August 1996. ACM SIGGRAPH.
31. 31.R. Szeliski and H.-Y. Shum. Creating full view panoramic image mosaics and texture-mapped models. In Computer Graphics (SIGGRAPH'97) Proceedings, pages 251-258, Los Angees, August 1997. ACM SIGGRAPH.
32. 32.J. Torborg and J. T. Kajiya. Talisman: Commodity realtime 3D graphics for the PC. In Computer Graphics Proceedings, Annual Conference Series, pages 353-363, Proc. SIGGRAPH'96 (New Orleans), August 1996. ACM SIGGRAPH.

Index Terms

• Computing methodologies

  • Artificial intelligence

    • Computer vision

      • Computer vision problems

        • Image segmentation

        • Video segmentation

      • Computer vision representations

        • Appearance and texture representations

        • Shape representations

      • Computer vision tasks

        • Video summarization

      • Image and video acquisition

  • Computer graphics

    • Animation

    • Image manipulation

      • Texturing

    • Rendering

  • Modeling and simulation

    • Simulation types and techniques

      • Simulation by animation