Ellipsoidal path connections for time-gated rendering

Abstract

During the last decade, we have been witnessing the continued development of new time-of-flight imaging devices, and their increased use in numerous and varied applications. However, physics-based rendering techniques that can accurately simulate these devices are still lacking: while existing algorithms are adequate for certain tasks, such as simulating transient cameras, they are very inefficient for simulating time-gated cameras because of the large number of wasted path samples. We take steps towards addressing these deficiencies, by introducing a procedure for efficiently sampling paths with a predetermined length, and incorporating it within rendering frameworks tailored towards simulating time-gated imaging. We use our open-source implementation of the above to empirically demonstrate improved rendering performance in a variety of applications, including simulating proximity sensors, imaging through occlusions, depth-selective cameras, transient imaging in dynamic scenes, and non-line-of-sight imaging.

References

1. 2019 FESCA-100 Femtosecond streak camera. https://www.hamamatsu.com/us/en/product/photometry-systems/streak-camera/fesca-100-femtosecond-streak-camera/index.html. Accessed: 01-12-2019.
2. 2019 Laser Range-Gated Imaging for Imaging at Long Ranges and Through Obscurants. http://www.sensorsinc.com/applications/military/laser-range-gating. Accessed: 01-12-2019.
3. Marco Ament, Christoph Bergmann, and Daniel Weiskopf. 2014. Refractive radiative transfer equation. ACM Transactions on Graphics (TOG) 33, 2 (2014), 17.
4. Luke Anderson, Tzu-Mao Li, Jaakko Lehtinen, and Frédo Durand. 2017. Aether: An embedded domain specific sampling language for Monte Carlo rendering. ACM Transactions on Graphics (TOG) 36, 4 (2017), 99.
5. Victor Arellano, Diego Gutierrez, and Adrian Jarabo. 2017. Fast back-projection for non-line of sight reconstruction. Optics Express 25, 10 (2017), 11574--11583.
6. Ian M Baker, Stuart S Duncan, and Jeremy W Copley. 2004. A low-noise laser-gated imaging system for long-range target identification. In Infrared Technology and Applications XXX, Vol. 5406. International Society for Optics and Photonics, 133--145.
7. Mauro Buttafava, Jessica Zeman, Alberto Tosi, Kevin Eliceiri, and Andreas Velten. 2015. Non-line-of-sight imaging using a time-gated single photon avalanche diode. Optics express 23, 16 (2015), 20997--21011.
8. Lucrezia Cester, Ashley Lyons, Maria Braidotti, and Daniele Faccio. 2019. Time-of-Flight Imaging at 10 ps Resolution with an ICCD Camera. Sensors 19, 1 (2019), 180.
9. Susan Chan, Ryan E Warburton, Genevieve Gariepy, Jonathan Leach, and Daniele Faccio. 2017. Non-line-of-sight tracking of people at long range. Optics express 25, 9 (2017), 10109--10117.
10. Subrahmanyan Chandrasekhar. 1960. Radiative transfer. Dover Publication.
11. BB Das, KM Yoo, and RR Alfano. 1993. Ultrafast time-gated imaging in thick tissues: a step toward optical mammography. Optics letters 18, 13 (1993), 1092--1094.
12. Ofer David, Norman S Kopeika, and Boaz Weizer. 2006. Range gated active night vision system for automobiles. Applied optics 45, 28 (2006), 7248--7254.
13. Philip Dutre, Philippe Bekaert, and Kavita Bala. 2006. Advanced global illumination. AK Peters/CRC Press.
14. Daniele Faccio and Andreas Velten. 2018. A trillion frames per second: the techniques and applications of light-in-flight photography. Reports on Progress in Physics 81, 10 (2018), 105901.
15. Genevieve Gariepy, Nikola Krstajić, Robert Henderson, Chunyong Li, Robert R Thomson, Gerald S Buller, Barmak Heshmat, Ramesh Raskar, Jonathan Leach, and Daniele Faccio. 2015. Single-photon sensitive light-in-fight imaging. Nature communications 6 (2015), 6021.
16. Ioannis Gkioulekas, Anat Levin, Frédo Durand, and Todd Zickler. 2015. Micron-scale light transport decomposition using interferometry. ACM Transactions on Graphics (ToG) 34, 4 (2015), 37.
17. Ioannis Gkioulekas, Anat Levin, and Todd Zickler. 2016. An Evaluation of Computational Imaging Techniques for Heterogeneous Inverse Scattering. European Conference on Computer Vision (2016).
18. Yoav Grauer and Ezri Sonn. 2015. Active gated imaging for automotive safety applications. In Video Surveillance and Transportation Imaging Applications 2015, Vol. 9407. International Society for Optics and Photonics, 94070F.
19. Tobias Gruber, Frank Julca-Aguilar, Mario Bijelic, Werner Ritter, Klaus Dietmayer, and Felix Heide. 2019. Gated2Depth: Real-time Dense Lidar from Gated Images. arXiv preprint arXiv.1902.04997 (2019).
20. MohitGupta, Shree K Nayar, Matthias B Hullin, and Jaime Martin. 2015. Phasor imaging: A generalization of correlation-based time-of-flight imaging. ACM Transactions on Graphics (ToG) 34, 5 (2015), 156.
21. Mohit Gupta, Andreas Velten, Shree K Nayar, and Eric Breitbach. 2018. What are optimal coding functions for time-of-flight imaging? ACM Transactions on Graphics (TOG) 37, 2 (2018), 13.
22. Toshiya Hachisuka, Anton S Kaplanyan, and Carsten Dachsbacher. 2014. Multiplexed metropolis light transport. ACM Transactions on Graphics (TOG) 33, 4 (2014), 100.
23. Toshiya Hachisuka, Shinji Ogaki, and Henrik Wann Jensen. 2008. Progressive photon mapping. In ACM Transactions on Graphics (TOG), Vol. 27. ACM, 130.
24. Felix Heide, Matthias B Hullin, James Gregson, and Wolfgang Heidrich. 2013. Low-budget transient imaging using photonic mixer devices. ACM Transactions on Graphics (ToG) 32, 4 (2013), 45.
25. Felix Heide, Matthew O'Toole, Kai Zhang, David Lindell, Steven Diamond, and Gordon Wetzstein. 2019. Non-line-of-sight Imaging with Partial Occluders and Surface Normals. ACM Transactions on Graphics (ToG) (2019).
26. Felix Heide, Lei Xiao, Wolfgang Heidrich, and Matthias B Hullin. 2014. Diffuse mirrors: 3D reconstruction from diffuse indirect illumination using inexpensive time-of-flight sensors. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 3222--3229.
27. Wenzel Jakob. 2010. Mitsuba renderer. http://www.mitsuba-renderer.org.
28. Wenzel Jakob and Steve Marschner. 2012. Manifold exploration: A markov chain monte carlo technique for rendering scenes with difficult specular transport. ACM Transactions on Graphics (2012).
29. Adrian Jarabo. 2012. Femto-photography: Visualizing light in motion. Master's thesis, Universidad de Zaragoza (2012).
30. Adrian Jarabo, Julio Marco, Adolfo Muñoz, Raul Buisan, Wojciech Jarosz, and Diego Gutierrez. 2014. A framework for transient rendering. ACM Transactions on Graphics (ToG) 33, 6 (2014), 177.
31. Adrian Jarabo, Belen Masia, Julio Marco, and Diego Gutierrez. 2017. Recent advances in transient imaging: A computer graphics and vision perspective. Visual Informatics 1, 1 (2017), 65--79.
32. Henrik Wann Jensen. 2001. Realistic image synthesis using photon mapping. AK Peters/CRC Press.
33. Ahmed Kirmani, Tyler Hutchison, James Davis, and Ramesh Raskar. 2009. Looking around the corner using transient imaging. In 2009 IEEE 12th International Conference on Computer Vision. IEEE, 159--166.
34. Eric P Lafortune and Yves D Willems. 1996. Rendering participating media with bidirectional path tracing. In Rendering Techniques' 96. Springer, 91--100.
35. Joseph R Lakowicz, Henryk Szmacinski, Kazimierz Nowaczyk, Klaus W Berndt, and Michael Johnson. 1992. Fluorescence lifetime imaging. Analytical biochemistry 202, 2 (1992), 316--330.
36. Martin Laurenzis and Andreas Velten. 2014. Nonline-of-sight laser gated viewing of scattered photons. Optical Engineering 53, 2 (2014), 023102.
37. Tzu-Mao Li, Miika Aittala, Frédo Durand, and Jaakko Lehtinen. 2018. Differentiable monte carlo ray tracing through edge sampling. In SIGGRAPH Asia 2018 Technical Papers. ACM, 222.
38. Jingyu Lin, Yebin Liu, Matthias B Hullin, and Qionghai Dai. 2014. Fourier analysis on transient imaging with a multifrequency time-of-flight camera. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 3230--3237.
39. Julio Marco, Ibón Guillén, Wojciech Jarosz, Diego Gutierrez, and Adrian Jarabo. 2018. Progressive Transient Photon Beams. arXiv preprint arXiv:1805.09562 (2018).
40. Julio Marco, Quercus Hernandez, Adolfo Muñoz, Yue Dong, Adrian Jarabo, Min H Kim, Xin Tong, and Diego Gutierrez. 2017a. DeepToF: off-the-shelf real-time correction of multipath interference in time-of-flight imaging. ACM Transactions on Graphics (TOG) 36, 6 (2017), 219.
41. Julio Marco, Wojciech Jarosz, Diego Gutierrez, and Adrian Jarabo. 2017b. Transient photon beams. In ACM SIGGRAPH 2017 Posters. ACM, 52.
42. Stephen Robert Marschner and Donald P Greenberg. 1998. Inverse rendering for computer graphics. Cornell University.
43. Nikhil Naik, Shuang Zhao, Andreas Velten, Ramesh Raskar, and Kavita Bala. 2011. Single View Reflectance Capture Using Multiplexed Scattering and Time-of-flight Imaging. ACM Trans. Graph. 30, 6 (2011), 171:1--171:10.
44. Matthew O'Toole, Felix Heide, Lei Xiao, Matthias B Hullin, Wolfgang Heidrich, and Kiriakos N Kutulakos. 2014. Temporal frequency probing for 5D transient analysis of global light transport. ACM Transactions on Graphics (ToG) 33, 4 (2014), 87.
45. Matthew O'Toole, David B Lindell, and Gordon Wetzstein. 2018. Confocal non-line-of-sight imaging based on the light-cone transform. Nature 555, 7696 (2018), 338.
46. Adithya Pediredla, Mauro Buttafava, Alberto Tosi, Oliver Cossairt, and Ashok Veeraraghavan. 2017a. Reconstructing rooms using photon echoes: A plane based model and reconstruction algorithm for looking around the corner. In 2017 IEEE International Conference on Computational Photography (ICCP). IEEE, 1--12.
47. Adithya Pediredla, Akshat Dave, and Ashok Veeraraghavan. 2019a. SNLOS: Non-line-of-sight Scanning through Temporal Focusing. (2019).
48. Adithya Pediredla, Nathan Matsuda, Oliver Cossairt, and Ashok Veeraraghavan. 2017b. Linear systems approach to identifying performance bounds in indirect imaging. In Acoustics, Speech and Signal Processing (ICASSP), 2017 IEEE International Conference on. IEEE, 6235--6239.
49. Adithya Pediredla, Ashok Veeraraghavan, and Gkioulekas Ioannis. 2019b. Mitsuba Time-of-Flight renderer. https://github.com/cmu-ci-lab/MitsubaToFRenderer.
50. Christoph Peters, Jonathan Klein, Matthias B Hullin, and Reinhard Klein. 2015. Solving trigonometric moment problems for fast transient imaging. ACM Transactions on Graphics (TOG) 34, 6 (2015), 220.
51. Matt Pharr, Wenzel Jakob, and Greg Humphreys. 2016. Physically based rendering: From theory to implementation. Morgan Kaufmann.
52. Guy Satat, Barmak Heshmat, Dan Raviv, and Ramesh Raskar. 2016. All photons imaging through volumetric scattering. Scientific reports 6 (2016), 33946.
53. Guy Satat, Matthew Tancik, and Ramesh Raskar. 2018. Towards photography through realistic fog. In Computational Photography (ICCP), 2018 IEEE International Conference on. IEEE, 1--10.
54. B Schmidt, S Laimgruber, W Zinth, and P Gilch. 2003. A broadband Kerr shutter for femtosecond fluorescence spectroscopy. Applied Physics B 76, 8 (2003), 809--814.
55. Shikhar Shrestha, Felix Heide, Wolfgang Heidrich, and Gordon Wetzstein. 2016. Computational imaging with multi-camera time-of-flight systems. ACM Transactions on Graphics (TOG) 35, 4 (2016), 33.
56. Adam Smith, James Skorupski, and James Davis. 2008. Transient rendering. (2008).
57. Ryuichi Tadano, Adithya Kumar Pediredla, and Ashok Veeraraghavan. 2015. Depth selective camera: A direct, on-chip, programmable technique for depth selectivity in photography. In Proceedings of the IEEE International Conference on Computer Vision. 3595--3603.
58. Akira Takahashi, Mitsunori Nishizawa, Yoshinori Inagaki, Musubu Koishi, and Katsuyuki Kinoshita. 1994. New femtosecond streak camera with temporal resolution of 180 fs. In Generation, amplification, and measurement of ultrashort laser pulses, Vol. 2116. International Society for Optics and Photonics, 275--285.
59. Christos Thrampoulidis, Gal Shulkind, Feihu Xu, William T Freeman, Jeffrey Shapiro, Antonio Torralba, Franco Wong, and Gregory Wornell. 2018. Exploiting occlusion in non-line-of-sight active imaging. IEEE Transactions on Computational Imaging (2018).
60. Chia-Yin Tsai, Kiriakos N Kutulakos, Srinivasa G Narasimhan, and Aswin C Sankaranarayanan. 2017. The geometry of first-returning photons for non-line-of-sight imaging. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 7216--7224.
61. Chia-Yin Tsai, Aswin C. Sankaranarayanan, and Ioannis Gkioulekas. 2019. Beyond Volumetric Albedo-A Surface Optimization Framework for Non-Line-of-Sight Imaging. In CVPR.
62. Eric Veach. 1997. Robust monte carlo methods for light transport simulation. Number 1610. Stanford University PhD thesis.
63. Eric Veach and Leonidas Guibas. 1995a. Bidirectional estimators for light transport. In Photorealistic Rendering Techniques. Springer, 145--167.
64. Eric Veach and Leonidas J Guibas. 1995b. Optimally combining sampling techniques for Monte Carlo rendering. In Proceedings of the 22nd annual conference on Computer graphics and interactive techniques. ACM, 419--428.
65. Eric Veach and Leonidas J Guibas. 1997. Metropolis light transport. In Proceedings of the 24th annual conference on Computer graphics and interactive techniques. ACM Press/Addison-Wesley Publishing Co., 65--76.
66. Andreas Velten, Thomas Willwacher, Otkrist Gupta, Ashok Veeraraghavan, Moungi G Bawendi, and Ramesh Raskar. 2012. Recovering three-dimensional shape around a corner using ultrafast time-of-flight imaging. Nature communications 3 (2012), 745.
67. Andreas Velten, Di Wu, Adrian Jarabo, Belen Masia, Christopher Barsi, Chinmaya Joshi, Everett Lawson, Moungi Bawendi, Diego Gutierrez, and Ramesh Raskar. 2013. Femto-photography: capturing and visualizing the propagation of light. ACM Transactions on Graphics (ToG) 32, 4 (2013), 44.
68. Bruce Walter, Stephen R Marschner, Hongsong Li, and Kenneth E Torrance. 2007. Microfacet models for refraction through rough surfaces. In Proceedings of the 18th Eurographics conference on Rendering Techniques. Eurographics Association, 195--206.
69. Di Wu, Andreas Velten, Matthew O'Toole, Belen Masia, Amit Agrawal, Qionghai Dai, and Ramesh Raskar. 2014a. Decomposing global light transport using time of flight imaging. International journal of computer vision 107, 2 (2014), 123--138.
70. Di Wu, Gordon Wetzstein, Christopher Barsi, Thomas Willwacher, Qionghai Dai, and Ramesh Raskar. 2014b. Ultra-fast lensless computational imaging through 5D frequency analysis of time-resolved light transport. International journal of computer vision 110, 2 (2014), 128--140.
71. Shumian Xin, Sotiris Nousias, Kyriakos N. Kutulakos, Aswin C. Sankaranarayanan, Srinivasa G. Narasimhan, and Ioannis Gkioulekas. 2019. A Theory of Fermat Paths for Non-Line-of-Sight Shape Reconstruction. In CVPR.
72. Pingping Zhan, Junyi Tong, Wenjiang Tan, and Shichao Xu. 2016. Comparison between traditional and heterodyned optical Kerr gated imaging for CS 2 and Te glass Kerr media. Optical Engineering 55, 11 (2016), 110502.
73. Matthias Zwicker, Wojciech Jarosz, Jaakko Lehtinen, Bochang Moon, Ravi Ramamoorthi, Fabrice Rousselle, Pradeep Sen, Cyril Soler, and S-E Yoon. 2015. Recent advances in adaptive sampling and reconstruction for Monte Carlo rendering. In Computer graphics forum, Vol. 34. Wiley Online Library, 667--681.

Index Terms

• Computing methodologies

  • Computer graphics

    • Rendering

      • Ray tracing