Eulerian video magnification and analysis

Abstract

The world is filled with important, but visually subtle signals. A person's pulse, the breathing of an infant, the sag and sway of a bridge---these all create visual patterns, which are too difficult to see with the naked eye. We present Eulerian Video Magnification, a computational technique for visualizing subtle color and motion variations in ordinary videos by making the variations larger. It is a microscope for small changes that are hard or impossible for us to see by ourselves. In addition, these small changes can be quantitatively analyzed and used to recover sounds from vibrations in distant objects, characterize material properties, and remotely measure a person's pulse.

References

1. Amir-Khalili, A., Peyrat, J.-M., Abinahed, J., Al-Alao, O., Al-Ansari, A., Hamarneh, G., Abugharbieh, R. Auto localization and segmentation of occluded vessels in robot-assisted partial nephrectomy. In MICCAI 2014 (2014). Springer, 407--414.
2. Chen, J.G., Wadhwa, N., Cha, Y.-J., Durand, F., Freeman, W.T., Buyukozturk, O. Structural modal identification through high speed camera video: Motion magnification. In Topics in Modal Analysis I. Volume 7 (2014). Springer, 191--197.
3. Chen, J.G., Wadhwa, N., Cha, Y.-J., Durand, F., Freeman, W.T., Buyukozturk, O. Modal identification of simple structures with high-speed video using motion magnification. Journal of Sound and Vibration 345 (2015), 58--71.
4. Chen, J.G., Wadhwa, N., Durand, F., Freeman, W.T., Buyukozturk, O. Developments with motion magnification for structural modal identification through camera video. In Dynamics of Civil Structures. Volume 2 (2015). Springer, 49--57.
5. Davis, A., Bouman, K.L., Chen, J.G., Rubinstein, M., Durand, F., Freeman, W.T. Visual vibrometry: Estimating material properties from small motions in video. In CVPR 2015 (2015), 5335--5343.
6. Davis, A., Chen, J.G., Durand, F. Imagespace modal bases for plausible manipulation of objects in video. ACM Trans. Graph. 34, 6 (2015), 239.
7. Davis, A., Rubinstein, M., Wadhwa, N., Mysore, G.J., Durand, F., Freeman, W.T. The visual microphone: Passive recovery of sound from video. ACM Trans. Graph. 33, 4 (2014), 79.
8. Elgharib, M.A., Hefeeda, M., Durand, F., Freeman, W.T. Video magnification in presence of large motions. In CVPR 2015 (2015), 4119--4127.
9. Fleet, D.J., Jepson, A.D. Computation of component image velocity from local phase information. Int. J. Comput. Vision 5, 1 (1990), 77--104.
10. Fuchs, M., Chen, T., Wang, O., Raskar, R., Seidel, H.-P., Lensch, H.P. Real-time temporal shaping of high-speed video streams. Comput. Graph. 34, 5 (2010), 575--584.
11. Gautama, T., Van Hulle, M.M. A phase-based approach to the estimation of the optical flow field using spatial filtering. IEEE Trans. Neural Netw. 13, 5 (2002), 1127--1136.
12. Horn, B.K., Schunck, B.G. Determining optical flow. In 1981 Technical Symposium East (1981). International Society for Optics and Photonics, 319--331.
13. Liu, C., Torralba, A., Freeman, W.T., Durand, F., Adelson, E.H. Motion magnification. ACM Trans. Graph. 24, 3 (2005), 519--526.
14. Lucas, B.D., Kanade, T., et al. An iterative image registration technique with an application to stereo vision. In IJCAI. Volume 81 (1981), 674--679.
15. McLeod, A.J., Baxter, J.S., de Ribaupierre, S., Peters, T.M. Motion magnification for endoscopic surgery. In SPIE Medical Imaging (2014), 90360C--90360C.
16. Poh, M.-Z., McDuff, D.J., Picard, R.W. Non-contact, automated cardiac pulse measurements using video imaging and blind source separation. Opt. Express 18, 10 (2010), 10762--10774.
17. Rubinstein, M. Analysis and visualization of temporal variations in video. PhD thesis, Massachusetts Institute of Technology, Cambridge, MA (2014).
18. Sellon, J.B., Farrahi, S., Ghaffari, R., Freeman, D.M. Longitudinal spread of mechanical excitation through tectorial membrane traveling waves. Proc. Natl. Acad. Sci. 112, 42 (2015), 12968--12973.
19. Simoncelli, E.P., Freeman, W.T. The steerable pyramid: A flexible architecture for multi-scale derivative computation. In ICIP 1995 (1995), 3444.
20. Simoncelli, E.P., Freeman, W.T., Adelson, E.H., Heeger, D.J. Shiftable multi-scale transforms. IEEE Trans. Info. Theory 2, 38 (1992), 587--607.
21. Wachel, J., Morton, S.J., Atkins, K.E. Piping vibration analysis. In Proceedings of the 19th Turbomachinery Symposium. (1990).
22. Wadhwa, N., Rubinstein, M., Durand, F., Freeman, W.T. Phase-based video motion processing. ACM Trans. Graph. 32, 4 (2013), 80.
23. Wadhwa, N., Rubinstein, M., Durand, F., Freeman, W.T. Riesz pyramids for fast phase-based video magnification. In ICCP2014 (2014). IEEE, 1--10.
24. Wu, H.-Y., Rubinstein, M., Shih, E., Guttag, J. V., Durand, F., Freeman, W.T. Eulerian video magnification for revealing subtle changes in the world. ACM Trans. Graph. 31, 4 (2012), 65.

Index Terms

• Computing methodologies

  • Artificial intelligence

    • Computer vision

      • Computer vision problems

        • Video segmentation

      • Computer vision tasks

        • Scene understanding