Felix Heide
Skip Abstract Section
Skip Supplemental Material SectionAbstract
Imaging objects obscured by occluders is a significant challenge for many applications. A camera that could “see around corners” could help improve navigation and mapping capabilities of autonomous vehicles or make search and rescue missions more effective. Time-resolved single-photon imaging systems have recently been demonstrated to record optical information of a scene that can lead to an estimation of the shape and reflectance of objects hidden from the line of sight of a camera. However, existing non-line-of-sight (NLOS) reconstruction algorithms have been constrained in the types of light transport effects they model for the hidden scene parts. We introduce a factored NLOS light transport representation that accounts for partial occlusions and surface normals. Based on this model, we develop a factorization approach for inverse time-resolved light transport and demonstrate high-fidelity NLOS reconstructions for challenging scenes both in simulation and with an experimental NLOS imaging system.
Supplemental Material
Available for Download
zip
heide.zip (151.9 MB)
Supplemental movie and image files for, Non-line-of-sight Imaging with Partial Occluders and Surface Normals
- Fadel Adib, Chen-Yu Hsu, Hongzi Mao, Dina Katabi, and Frédo Durand. 2015. Capturing the human figure through a wall. ACM Trans. Graph. 34, 6 (2015). Google Scholar
Digital Library
- Yoann Altmann, Stephen McLaughlin, Miles J. Padgett, Vivek K. Goyal, Alfred O. Hero, and Daniele Faccio. 2018. Quantum-inspired computational imaging. Science 361, 6403 (2018).Google Scholar
- Victor Arellano, Diego Gutierrez, and Adrian Jarabo. 2017. Fast back-projection for non-line of sight reconstruction. Opt. Express 25, 10 (2017), 11574--11583.Google ScholarCross Ref
- Katherine Bouman, Vickie Ye, Adam Yedidia, Fredo Durand, Gregory W. Wornell, Antonio Torralba, and William T. Freeman. 2017. Turning corners into cameras: Principles and methods. In Proc. CVPR. 2270--2278.Google Scholar
- Stephen Boyd, Neal Parikh, Eric Chu, Borja Peleato, and Jonathan Eckstein. 2011. Distributed optimization and statistical learning via the alternating direction method of multipliers. Found. Trends Mach. Learn. 3, 1 (2011), 1--122. Google ScholarDigital Library
- Samuel Burri, Harald Homulle, Claudio Bruschini, and Edoardo Charbon. 2016. LinoSPAD: A time-resolved 256x1 CMOS SPAD line sensor system featuring 64 FPGA-based TDC channels running at up to 8.5 giga-events per second. Proc. SPIE 9899, 98990D--10 (2016).Google Scholar
- Mauro Buttafava, Jessica Zeman, Alberto Tosi, Kevin Eliceiri, and Andreas Velten. 2015. Non-line-of-sight imaging using a time-gated single photon avalanche diode. Opt. Express 23, 16 (2015).Google Scholar
- Susan Chan, Ryan E. Warburton, Genevieve Gariepy, Jonathan Leach, and Daniele Faccio. 2017. Non-line-of-sight tracking of people at long range. Opt. Express 25, 9 (2017), 10109--10117.Google ScholarCross Ref
- Quercus Hernandez, Diego Gutierrez, and Adrian Jarabo. 2017. A computational model of a single-photon avalanche diode sensor for transient imagingarXiv preprint arXiv:1703.02635 (2017).Google Scholar
- Adithya Kumar Pediredla, Mauro Buttafava, Alberto Tosi, Oliver Cossairt, and Ashok Veeraraghavan. 2017. Reconstructing rooms using photon echoes: A plane based model and reconstruction algorithm for looking around the corner. In Proc. ICCP. 1--12.Google ScholarCross Ref
- Genevieve Gariepy, Nikola Krstajic, Robert Henderson, Chunyong Li, Robert R. Thomson, Gerald S. Buller, Barmak Heshmat, Ramesh Raskar, Jonathan Leach, and Daniele Faccio. 2015. Single-photon sensitive light-in-fight imaging. Nat. Commun. 6, 6021 (2015).Google Scholar
- Genevieve Gariepy, Francesco Tonolini, Robert Henderson, Jonathan Leach, and Daniele Faccio. 2016. Detection and tracking of moving objects hidden from view. Nat. Photon. 10, 1 (2016).Google Scholar
- Otkrist Gupta, Thomas Willwacher, Andreas Velten, Ashok Veeraraghavan, and Ramesh Raskar. 2012. Reconstruction of hidden 3D shapes using diffuse reflections. OSA Opt. Express 20, 17 (2012).Google Scholar
- Felix Heide, Steven Diamond, David B. Lindell, and Gordon Wetzstein. 2018. Sub-picosecond photon-efficient 3D imaging using single-photon sensors. Sci. Rep. 8, 1 (2018).Google Scholar
- Felix Heide, Lei Xiao, Wolfgang Heidrich, and Matthias B. Hullin. 2014. Diffuse mirrors: 3D reconstruction from diffuse indirect illumination using inexpensive time-of-flight sensors. In Proc. CVPR. Google ScholarDigital Library
- Adrian Jarabo, Julio Marco, Adolfo Muñoz, Raul Buisan, Wojciech Jarosz, and Diego Gutierrez. 2014. A framework for transient rendering. ACM Trans. Graph. 33, 6, Article 177 (Nov. 2014), 10 pages. Google ScholarDigital Library
- Achuta Kadambi, Rafael Whyte, Ayush Bhandari, Lee Streeter, Christopher Barsi, Adrian Dorrington, and Ramesh Raskar. 2013. Coded time of flight cameras: Sparse deconvolution to address multipath interference and recover time profiles. ACM Trans. Graph. (SIGGRAPH Asia) 32, 6 (2013). Google ScholarDigital Library
- Avinash Kak and Malcolm Slaney. 1988. Principles of Computerized Tomographic Imaging. IEEE Press.Google Scholar
- Ori Katz, Pierre Heidmann, Mathias Fink, and Sylvain Gigan. 2014. Non-invasive single-shot imaging through scattering layers and around corners via speckle correlations. Nat. Photon. 8, 10 (2014).Google Scholar
- Ahmed Kirmani, Tyler Hutchison, James Davis, and Ramesh Raskar. 2009. Looking around the corner using transient imaging. Proc. ICCV (2009), 159--166.Google ScholarCross Ref
- Ahmed Kirmani, Dheera Venkatraman, Dongeek Shin, Andrea Colaco, Franco N. C. Wong, Jeffrey H. Shapiro, and Vivek K. Goyal. 2014. First-photon imaging. Science 343, 6166 (2014), 58--61.Google Scholar
- Jonathan Klein, Christoph Peters, Jaime Martin, Martin Laurenzis, and Matthias B. Hullin. 2016. Tracking objects outside the line of sight using 2D intensity images. Sci. Rep. 6 (2016), 177--188.Google Scholar
- David B. Lindell, Matthew O’Toole, and Gordon Wetzstein. 2018a. Single-photon 3D imaging with deep sensor fusion. ACM Trans. Graph. 4, 37 (2018). Google ScholarDigital Library
- David B. Lindell, Matthew O’Toole, and Gordon Wetzstein. 2018b. Towards transient imaging at interactive rates with single-photon detectors. Proc. IEEE ICCP (2018).Google ScholarCross Ref
- Nelson Max. 1995. Optical models for direct volume rendering. IEEE Trans. Vis. Comput. Graph. 1, 2 (1995), 99--108. Google ScholarDigital Library
- Aongus McCarthy, Nils J. Krichel, Nathan R. Gemmell, Ximing Ren, Michael G. Tanner, Sander N. Dorenbos, Val Zwiller, Robert H. Hadfield, and Gerald S. Buller. 2013. Kilometer-range, high resolution depth imaging via 1560 nm wavelength single-photon detection. Opt. Express 21, 7 (2013), 8904--8915.Google ScholarCross Ref
- Desmond O’Connor. 2012. Time-correlated Single Photon Counting. Academic Press.Google Scholar
- Matthew O’Toole, Felix Heide, David Lindell, Kai Zang, Stephen Diamond, and Gordon Wetzstein. 2017. Reconstructing transient images from single-photon sensors. Proc. CVPR (2017).Google ScholarCross Ref
- Matthew O’Toole, David B. Lindell, and Gordon Wetzstein. 2018. Confocal non-line-of-sight imaging based on the light cone transform. Nature 555, 7696 (2018), 338--341.Google Scholar
- Joshua Rapp and Vivek K. Goyal. 2017. A few photons among many: Unmixing signal and noise for photon-efficient active imaging. IEEE Trans. Comput. Imag. 3, 3 (2017), 445--459.Google ScholarCross Ref
- Albert Redo-Sanchez, Barmak Heshmat, Alireza Aghasi, Salman Naqvi, Mingjie Zhang, Justin Romberg, and Ramesh Raskar. 2016. Terahertz time-gated spectral imaging for content extraction through layered structures. Nat. Commun. 7 (2016).Google Scholar
- Brent Schwarz. 2010. LIDAR: Mapping the world in 3D. Nat. Photon. 4, 7 (2010), 1749--4885.Google ScholarCross Ref
- Pradeep Sen, Billy Chen, Gaurav Garg, Stephen R. Marschner, Mark Horowitz, Marc Levoy, and Hendrik P. A. Lensch. 2005. Dual photography. ACM Trans. Graph. 24, 3 (2005), 745--755. Google ScholarDigital Library
- Dongeek Shin, Feihu Xu, Dheera Venkatraman, Rudi Lussana, Federica Villa, Franco Zappa, Vivek K. Goyal, Franco N. C. Wong, and Jeffrey H. Shapiro. 2016. Photon-efficient imaging with a single-photon camera. Nat. Commun. 7, 12046 (2016), 1--7.Google ScholarCross Ref
- Christos Thrampoulidis, Gal Shulkind, Feihu Xu, William T. Freeman, Jeffrey H. Shapiro, Antonio Torralba, Franco N. C. Wong, and Gregory W. Wornell. 2017. Exploiting occlusion in non-line-of-sight active imaging. (2017). arxiv:1711.06297.Google Scholar
- Chia-Yin Tsai, Kiriakos N. Kutulakos, Srinivasa G. Narasimhan, and Aswin C. Sankaranarayanan. 2017. The geometry of first-returning photons for non-line-of-sight imaging. In Proc. CVPR. 1--9.Google Scholar
- Andreas Velten, Thomas Willwacher, Otkrist Gupta, Ashok Veeraraghavan, Moungi G. Bawendi, and Ramesh Raskar. 2012. Recovering three-dimensional shape around a corner using ultrafast time-of-flight imaging. Nat. Commun. 3 (2012).Google Scholar
- Di Wu, Gordon Wetzstein, Christopher Barsi, Thomas Willwacher, Matthew O’Toole, Nikhil Naik, Qionghai Dai, Kyros Kutulakos, and Ramesh Raskar. 2012. Frequency analysis of transient light transport with applications in bare sensor imaging. In Proc. ECCV. Google ScholarDigital Library
- Feihu Xu, Gal Shulkind, Christos Thrampoulidis, Jeffrey H. Shapiro, Antonio Torralba, Franco N. C. Wong, and Gregory W. Wornell. 2018. Revealing hidden scenes by photon-efficient occlusion-based opportunistic active imaging. OSA Opt. Express 26, 8 (2018), 9945--9962.Google ScholarCross Ref
Index Terms
- Non-line-of-sight Imaging with Partial Occluders and Surface Normals
Recommendations
Learned feature embeddings for non-line-of-sight imaging and recognition
Objects obscured by occluders are considered lost in the images acquired by conventional camera systems, prohibiting both visualization and understanding of such hidden objects. Non-line-of-sight methods (NLOS) aim at recovering information about hidden ...
A dataset for benchmarking time-resolved non-line-of-sight imaging
SIGGRAPH '19: ACM SIGGRAPH 2019 PostersTime-resolved imaging has made it possible to look around corners by exploiting information from diffuse light bounces. While there have been successive improvements in the field since its conception, so far it has only been proven to work in very ...
Virtual Mirrors: Non-Line-of-Sight Imaging Beyond the Third Bounce
Non-line-of-sight (NLOS) imaging methods are capable of reconstructing complex scenes that are not visible to an observer using indirect illumination. However, they assume only third-bounce illumination, so they are currently limited to single-corner ...
Comments