Bundled camera paths for video stabilization

Abstract

We present a novel video stabilization method which models camera motion with a bundle of (multiple) camera paths. The proposed model is based on a mesh-based, spatially-variant motion representation and an adaptive, space-time path optimization. Our motion representation allows us to fundamentally handle parallax and rolling shutter effects while it does not require long feature trajectories or sparse 3D reconstruction. We introduce the 'as-similar-as-possible' idea to make motion estimation more robust. Our space-time path smoothing adaptively adjusts smoothness strength by considering discontinuities, cropping size and geometrical distortion in a unified optimization framework. The evaluation on a large variety of consumer videos demonstrates the merits of our method.

References

1. Baker, S., Bennett, E. P., Kang, S. B., and Szeliski, R. 2010. Removing rolling shutter wobble. In Proc. CVPR.
2. Bay, H., Ess, A., Tuytelaars, T., and Van Gool, L. 2008. Speeded-up robust features (surf). Comput. Vis. Image Underst. 110, 3, 346--359.
3. Bronshtein, I. N., and Semendyayev, K. A. 1997. Handbook of Mathematics. Springer-Verlag, New York, NY, USA.
4. Brox, T., Bruhn, A., Papenberg, N., and Weickert, J. 2004. High accuracy optical flow estimation based on a theory for warping. In Proc. ECCV.
5. Buehler, C., Bosse, M., and McMillan, L. 2001. Non-metric image-based rendering for video stabilization. In Proc. CVPR.
6. Chen, B.-Y., Lee, K.-Y., Huang, W.-T., and Lin, J.-S. 2008. Capturing intention-based full-frame video stabilization. Computer Graphics Forum 27, 7, 1805--1814.
7. Cho, S., Wang, J., and Lee, S. 2012. Video deblurring for hand-held cameras using patch-based synthesis. ACM Trans. Graph. (Proc. of SIGGRAPH) 31, 4.
8. Fischler, M. A., and Bolles, R. C. 1981. Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography. Commun. ACM 24, 6, 381--395.
9. Forssén, P.-E., and Ringaby, E. 2010. Rectifying rolling shutter video from hand-held devices. In CVPR.
10. Gao, J., Kim, S. J., and Brown, M. S. 2011. Constructing image panoramas using dual-homography warping. In Proc. CVPR.
11. Gleicher, M. L., and Liu, F. 2007. Re-cinematography: Improving the camera dynamics of casual video. In Proc. of ACM Multimedia.
12. Goldstein, A., and Fattal, R. 2012. Video stabilization using epipolar geometry. ACM Trans. Graph. (TOG) 31, 5, 126:1--126:10.
13. Grundmann, M., Kwatra, V., and Essa, I. 2011. Auto-directed video stabilization with robust l1 optimal camera paths. In Proc. CVPR.
14. Grundmann, M., Kwatra, V., Castro, D., and Essa, I. 2012. Calibration-free rolling shutter removal. In Proc. ICCP.
15. Hartley, R., and Zisserman, A. 2003. Multiple View Geometry in Computer Vision, 2 ed. Cambridge University Press, New York, NY, USA.
16. Igarashi, T., Moscovich, T., and Hughes, J. F. 2005. As-rigid-as-possible shape manipulation. ACM Trans. Graph. (Proc. of SIGGRAPH) 24, 3, 1134--1141.
17. Karpenko, A., Jacobs, D., Baek, J., and Levoy, M. 2011. Digital video stabilization and rolling shutter correction using gyroscopes. In Stanford CS Tech Report.
18. Lee, K.-Y., Chuang, Y.-Y., Chen, B.-Y., and Ouhyoung, M. 2009. Video stabilization using robust feature trajectories. In Proc. ICCV.
19. Liang, C.-K., Chang, L.-W., and Chen, H. H. 2008. Analysis and compensation of rolling shutter effect. In IEEE Trans. on Image Processing.
20. Lin, W.-Y., Liu, S., Matsushita, Y., Ng, T.-T., and Cheong, L.-F. 2011. Smoothly varying affine stitching. In Proc. CVPR.
21. Liu, F., Gleicher, M., Jin, H., and Agarwala, A. 2009. Content-preserving warps for 3d video stabilization. ACM Trans. Graph. (Proc. of SIGGRAPH) 28.
22. Liu, F., Gleicher, M., Wang, J., Jin, H., and Agarwala, A. 2011. Subspace video stabilization. ACM Trans. Graph. 30.
23. Liu, S., Wang, Y., Yuan, L., Bu, J., Tan, P., and Sun, J. 2012. Video stabilization with a depth camera. In Proc. CVPR.
24. Lucas, B. D., and Kanade, T. 1981. An iterative image registration technique with an application to stereo vision. In Proc. of the International Joint Conference on Artificial Intelligence (IJCAI), 674--679.
25. Matsushita, Y., Ofek, E., Ge, W., Tang, X., and Shum, H.-Y. 2006. Full-frame video stabilization with motion inpainting. IEEE Trans. Pattern Anal. Mach. Intell. 28, 1150--1163.
26. Morimoto, C., and Chellappa, R. 1998. Evaluation of image stabilization algorithms. In Proc. of IEEE International Conference on Acoustics, Speech and Signal Processing, 2789--2792.
27. Nakamura, J. 2005. Image Sensors and Signal Processing for Digital Still Cameras. CRC Press, Inc.
28. Nir, T., Bruckstein, A. M., and Kimmel, R. 2008. Over-parameterized variational optical flow. Int. J. Comput. Vision (IJCV) 76, 2, 205--216.
29. Schaefer, S., McPhail, T., and Warren, J. 2006. Image deformation using moving least squares. ACM Trans. Graph. (Proc. of SIGGRAPH) 25, 3, 533--540.
30. Shum, H.-Y., and Szeliski, R. 2000. Construction of panoramic image mosaics with global and local alignment. Int. J. Comput. Vision (IJCV) 36, 2, 101--130.
31. Smith, B. M., Zhang, L., Jin, H., and Agarwala, A. 2009. Light field video stabilization. In Proc. ICCV.
32. Szeliski, R. 1996. Motion estimation with quadtree splines. IEEE Trans. Pattern Anal. Mach. Intell. 18, 12, 1199--1210.
33. Tomasi, C., and Manduchi, R. 1998. Bilateral filtering for gray and color images. In Proc. ICCV, 839--846.

Index Terms

• Computing methodologies

  • Artificial intelligence

    • Computer vision

  • Computer graphics

    • Image manipulation