Image-space modal bases for plausible manipulation of objects in video

Abstract

We present algorithms for extracting an image-space representation of object structure from video and using it to synthesize physically plausible animations of objects responding to new, previously unseen forces. Our representation of structure is derived from an image-space analysis of modal object deformation: projections of an object's resonant modes are recovered from the temporal spectra of optical flow in a video, and used as a basis for the image-space simulation of object dynamics. We describe how to extract this basis from video, and show that it can be used to create physically-plausible animations of objects without any knowledge of scene geometry or material properties.

References

1. Bathe, K.-J. 2006. Finite element procedures. Klaus-Jurgen Bathe.
2. Brincker, R., Ventura, C., and Andersen, P. 2003. Why output-only modal testing is a desirable tool for a wide range of practical applications. In Proc. Of the International Modal Analysis Conference (IMAC) XXI, paper, vol. 265.
3. Chen, J. G., Wadhwa, N., Cha, Y.-J., Durand, F., Freeman, W. T., and Buyukozturk, O. 2015. Modal identification of simple structures with high-speed video using motion magnification. Journal of Sound and Vibration 345, 58--71.
4. Chuang, Y.-Y., Goldman, D. B., Zheng, K. C., Curless, B., Salesin, D. H., and Szeliski, R. 2005. Animating pictures with stochastic motion textures. ACM Trans. Graph. 24, 3 (July), 853--860.
5. Davis, A., Rubinstein, M., Wadhwa, N., Mysore, G., Durand, F., and Freeman, W. T. 2014. The visual microphone: Passive recovery of sound from video. ACM Transactions on Graphics (Proc. SIGGRAPH) 33, 4, 79:1--79:10.
6. Davis, A., Bouman, K. L., Chen, J. G., Rubinstein, M., Durand, F., and Freeman, W. T. 2015. Visual vibrometry: Estimating material properties from small motion in video. The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (June).
7. De Roeck, G., Peeters, B., and Ren, W.-X. 2000. Benchmark study on system identification through ambient vibration measurements. In Proceedings of IMAC-XVIII, the 18th International Modal Analysis Conference, San Antonio, Texas, 1106--1112.
8. Doretto, G., Chiuso, A., Wu, Y., and Soatto, S. 2003. Dynamic textures. International Journal of Computer Vision 51, 2, 91--109.
9. Helfrick, M. N., Niezrecki, C., Avitabile, P., and Schmidt, T. 2011. 3d digital image correlation methods for full-field vibration measurement. Mechanical Systems and Signal Processing 25, 3, 917--927.
10. Huang, J., Tong, Y., Zhou, K., Bao, H., and Desbrun, M. 2011. Interactive shape interpolation through controllable dynamic deformation. Visualization and Computer Graphics, IEEE Transactions on 17, 7, 983--992.
11. James, D. L., and Pai, D. K. 2002. Dyrt: dynamic response textures for real time deformation simulation with graphics hardware. ACM Transactions on Graphics (TOG) 21, 3, 582--585.
12. James, D. L., and Pai, D. K. 2003. Multiresolution green's function methods for interactive simulation of large-scale elastostatic objects. ACM Transactions on Graphics (TOG) 22, 1, 47--82.
13. Li, S., Huang, J., de Goes, F., Jin, X., Bao, H., and Desbrun, M. 2014. Space-time editing of elastic motion through material optimization and reduction. ACM Transactions on Graphics 33, 4, Art--No.
14. Pai, D. K., Doel, K. V. D., James, D. L., Lang, J., Lloyd, J. E., Richmond, J. L., and Yau, S. H. 2001. Scanning physical interaction behavior of 3d objects. In Proceedings of the 28th Annual Conference on Computer Graphics and Interactive Techniques, ACM, New York, NY, USA, SIGGRAPH '01, 87--96.
15. Pentland, A., and Sclaroff, S. 1991. Closed-form solutions for physically based shape modeling and recognition. 715--729.
16. Pentland, A., and Williams, J. 1989. Good vibrations: Modal dynamics for graphics and animation, vol. 23. ACM.
17. Schödl, A., Szeliski, R., Salesin, D. H., and Essa, I. 2000. Video textures. In Proceedings of the 27th Annual Conference on Computer Graphics and Interactive Techniques, ACM Press/Addison-Wesley Publishing Co., New York, NY, USA, SIGGRAPH '00, 489--498.
18. Shabana, A. A. 1991. Theory of vibration, vol. 2. Springer.
19. Simoncelli, E. P., Freeman, W. T., Adelson, E. H., and Heeger, D. J. 1992. Shiftable multi-scale transforms. IEEE Trans. Info. Theory 2, 38, 587--607.
20. Stam, J., 1996. Stochastic dynamics: Simulating the effects of turbulence on flexible structures.
21. Sun, M., Jepson, A. D., and Fiume, E. 2003. Video input driven animation (vida). In Proceedings of the Ninth IEEE International Conference on Computer Vision - Volume 2, IEEE Computer Society, Washington, DC, USA, ICCV '03, 96--.
22. Szummer, M., and Picard, R. W. 1996. Temporal texture modeling. In IEEE Intl. Conf. Image Processing, vol. 3, 823--826.
23. Tao, H., and Huang, T. S. 1998. Connected vibrations: A modal analysis approach for non-rigid motion tracking. In CVPR, IEEE Computer Society, 735--740.
24. Wadhwa, N., Rubinstein, M., Durand, F., and Freeman, W. T. 2013. Phase-based video motion processing. ACM Trans. Graph. (Proceedings SIGGRAPH 2013) 32, 4.
25. Wadhwa, N., Rubinstein, M., Durand, F., and Freeman, W. T. 2014. Riesz pyramid for fast phase-based video magnification. In Computational Photography (ICCP), 2014 IEEE International Conference on, IEEE.

Index Terms

• Computing methodologies

  • Artificial intelligence

    • Computer vision

      • Computer vision problems

        • Video segmentation

      • Computer vision tasks

        • Scene understanding