Real-world occlusion in optical see-through AR displays

ABSTRACT

In this work we describe a system composed by an optical see-through AR headset---a Microsoft HoloLens---, stereo projectors and shutter glasses. Projectors are used to add to the device the capability of occluding real-world surfaces to make the virtual objects to appear more solid and less transparent.

A framework was developed in order to allow us to evaluate the importance of occlusion capabilities in optical see-through AR headset. We designed and conducted two experiment to test whether making virtual elements solid would improve the performance of certain tasks with an AR system. Results suggest that making virtual objects to appear more solid by projecting an occlusion mask onto the real-world is useful in some situations.

Using an occlusion mask it is also possible to eliminate ambiguities that could arise when enhancing user's perception in some ways that are not possible in real-life, like when a "x-ray vision" is enabled. In this case we wanted to investigate if using an occlusion mask to eliminate perceptual conflicts will hit user's performance in some AR applications.

The framework that allowed us to conduct our experiments is made freely available to anyone interested in conducting future studies.

References

1. Michael Bajura, Henry Fuchs, and Ryutarou Ohbuchi. 1992. Merging virtual objects with the real world: Seeing ultrasound imagery within the patient. In ACM SIGGRAPH Computer Graphics, Vol. 26. ACM, 203--210.
2. Michael Bajura and Ulrich Neumann. 1995. Dynamic registration correction in video-based augmented reality systems. IEEE Computer Graphics and Applications 15, 5 (1995), 52--60.
3. Oliver Bimber and Bernd Fröhlich. 2002. Occlusion shadows: Using projected light to generate realistic occlusion effects for view-dependent optical see-through displays. In Proceedings of the 1st International Symposium on Mixed and Augmented Reality. IEEE Computer Society, 186.
4. Dewen Cheng, Yongtian Wang, Hong Hua, and Jose Sasian. 2011. Design of a wide-angle, lightweight head-mounted display using free-form optics tiling. Optics letters 36, 11 (2011), 2098--2100.
5. David Drascic and Paul Milgram. 1996. Perceptual issues in augmented reality. In Proceedings-Spie The International Society For Optical Engineering. SPIE INTERNATIONAL SOCIETY FOR OPTICAL, 123--134.
6. Stephen R Ellis and Brian M Menges. 1998. Localization of virtual objects in the near visual field. Human Factors: The Journal of the Human Factors and Ergonomics Society 40, 3 (1998), 415--431.
7. Henry Fuchs. 1990. Systems for display of three-dimensional medical image data. In 3D imaging in medicine. Springer, 315--331.
8. Chunyu Gao, Yuxiang Lin, and Hong Hua. 2012. Occlusion capable optical see-through head-mounted display using freeform optics. In Mixed and Augmented Reality (ISMAR), 2012 IEEE International Symposium on. IEEE, 281--282.
9. Xinda Hu and Hong Hua. 2012. Design of an optical see-through multi-focal-plane stereoscopic 3d display using freeform prisms. In Frontiers in Optics. Optical Society of America, FTh1F-2.
10. Makoto Kimura, Masaaki Mochimaru, and Takeo Kanade. 2007. Projector calibration using arbitrary planes and calibrated camera. In 2007 IEEE Conference on Computer Vision and Pattern Recognition. IEEE, 1--2.
11. Kiyoshi Kiyokawa, Yoshinori Kurata, and Hiroyuki Ohno. 2001. An optical see-through display for mutual occlusion with a real-time stereovision system. Computers & Graphics 25, 5 (2001), 765--779.
12. Mark A Livingston, Arindam Dey, Christian Sandor, and Bruce H Thomas. 2013. Pursuit of "X-ray vision" for augmented reality. In Human Factors in Augmented Reality Environments. Springer, 67--107.
13. Mark A Livingston, J Edward Swan II, Joseph L Gabbard, Tobias H Höllerer, Deborah Hix, Simon J Julier, Yohan Baillot, and Dennis Brown. 2003. Resolving multiple occluded layers in augmented reality. In Proceedings of the 2nd IEEE/ACM International Symposium on Mixed and Augmented Reality. IEEE Computer Society, 56.
14. Andrew Maimone and Henry Fuchs. 2013. Computational augmented reality eyeglasses. In Mixed and Augmented Reality (ISMAR), 2013 IEEE International Symposium on. IEEE, 29--38.
15. Andrew Maimone, Xubo Yang, Nate Dierk, Andrei State, Mingsong Dou, and Henry Fuchs. 2013. General-purpose telepresence with head-worn optical see-through displays and projector-based lighting. In 2013 IEEE Virtual Reality (VR). IEEE, 23--26.
16. Ivan Martynov, Joni-Kristian Kamarainen, and Lasse Lensu. 2011. Projector calibration by "inverse camera calibration". In Scandinavian Conference on Image Analysis. Springer, 536--544.
17. Jannick P Rolland. 2000. Wide-angle, off-axis, see-through head-mounted display. Optical engineering 39, 7 (2000), 1760--1767.
18. Jannick P Rolland and Henry Fuchs. 2000. Optical versus video see-through head-mounted displays in medical visualization. Presence: Teleoperators and Virtual Environments 9, 3 (2000), 287--309.
19. Pedro Santos, Thomas Gierlinger, Oliver Machui, and André Stork. 2008. The daylight blocking optical stereo see-through HMD. In Proceedings of the 2008 workshop on Immersive projection technologies/Emerging display technologiges. ACM, 4.
20. Andrei State, Mark A Livingston, William F Garrett, Gentaro Hirota, Mary C Whitton, Etta D Pisano, and Henry Fuchs. 1996. Technologies for augmented reality systems: Realizing ultrasound-guided needle biopsies. In Proceedings of the 23rd annual conference on computer graphics and interactive techniques. ACM, 439--446.
21. Suya You and Ulrich Neumann. 2001. Fusion of vision and gyro tracking for robust augmented reality registration. In Virtual Reality, 2001. Proceedings. IEEE. IEEE, 71--78.
22. Feng Zhou, Henry Been-Lirn Duh, and Mark Billinghurst. 2008. Trends in augmented reality tracking, interaction and display: A review of ten years of ISMAR. In Proceedings of the 7th IEEE/ACM International Symposium on Mixed and Augmented Reality. IEEE Computer Society, 193--202.

Index Terms

• Human-centered computing

  • Human computer interaction (HCI)

    • Interaction paradigms

      • Mixed / augmented reality

  • Visualization

    • Visualization systems and tools

      • Visualization toolkits