RoomAlive: magical experiences enabled by scalable, adaptive projector-camera units

ABSTRACT

RoomAlive is a proof-of-concept prototype that transforms any room into an immersive, augmented entertainment experience. Our system enables new interactive projection mapping experiences that dynamically adapts content to any room. Users can touch, shoot, stomp, dodge and steer projected content that seamlessly co-exists with their existing physical environment. The basic building blocks of RoomAlive are projector-depth camera units, which can be combined through a scalable, distributed framework. The projector-depth camera units are individually auto-calibrating, self-localizing, and create a unified model of the room with no user intervention. We investigate the design space of gaming experiences that are possible with RoomAlive and explore methods for dynamically mapping content based on room layout and user position. Finally we showcase four experience prototypes that demonstrate the novel interactive experiences that are possible with RoomAlive and discuss the design challenges of adapting any game to any room.

References

1. Batlle, J., Mouaddib, E., and Salvi, J. Recent progress in coded structured light as a technique to solve correspondence problem: A survey. Pattern Recognition 31, 7 (1998), 963--982.
2. Behringer, R. Placing Artificial Objects in Real Scenes. Proc. of IWAR, (1998).
3. Benko, H., Jota, R., and Wilson, A. MirageTable : Freehand Interaction on a Projected Augmented Reality Tabletop. ACM CHI, (2012).
4. Bimber, O. and Raskar, R. Spatial augmented reality: Merging real and virtual worlds. AK Peters Ltd, 2005.
5. Bimber, O. PlayReal: Spatial Augmented Reality Games. 2008.
6. Brown, M., Majumder, A., and Yang, R. Camerabased calibration techniques for seamless multiprojector displays. Proc. of Visualization and Computer Graphcis, (2005), 193--206.
7. Brox, T., Papenberg, N., and Weickert, J. High Accuracy Optical Flow Estimation Based on a Theory for Warping. Proc. of ECCV, (2004), 25--36.
8. Butler, A., Izadi, S., Hilliges, O., Molyneaux, D., Hodges, S., and Kim, D. Shake 'n' Sense : Reducing Interference for Overlapping Structured Light Depth Cameras. ACM CHI, (2012), 1933-- 1936.
9. Cruz-Neira, C., Sandin, D., and DeFanti, T. Surround-screen projection-based virtual reality: the design and implementation of the CAVE. ACM SIGGRAPH, (1993).
10. Fuchs, H. and Frahm, J.-M. Scanning and tracking dynamic objects with commodity depth cameras. Proc. of ISMAR, Ieee (2013), 99--106. Figure 15. Calibration errors between units result in ghosting artifacts in projector overlapping regions. (left) The character in a single projector, and (right) in an overlapping region.
11. Hancock, M., Nacenta, M., Gutwin, C., and Carpendale, S. The effects of changing projection geometry on the interpretation of 3D orientation on tabletops. Proc. of ITS, (2009), 175--182.
12. Harrison, C., Benko, H., Wilson, A.D., and Way, O.M. OmniTouch : Wearable multitouch interaction everywhere. Proc. UIST, (2011), 441--450.
13. Hilliges, O., Kim, D., Izadi, S., Weiss, M., and Wilson, A. HoloDesk. Proc. of CHI, ACM Press (2012), 2421--2432.
14. Hou, J., Nam, Y., Peng, W., and Lee, K.M. Effects of screen size, viewing angle, and players' immersion tendencies on game experience. Computers in Human Behavior 28, 2 (2012), 617-- 623.
15. Illingworth, J. and Kittler, J. A survey of the Hough transform. Computer Vision, Graphics, and Image Processing 1, 44 (1988).
16. Jones, B., Benko, H., Ofek, E., and Wilson, A.D. IllumiRoom: Peripheral Projected Illusions for Interactive Experiences. ACM CHI, (2013).
17. Jones, B., Sodhi, R., Campbell, R., Garnett, G., and Bailey, B.P. Build Your World and Play In It: Interacting with Surface Particles on Complex Objects. IEEE ISMAR, (2010).
18. Kainz, B., Hauswiesner, S., Reitmayr, G., et al. OmniKinect : Real-Time Dense Volumetric Data Acquisition and Applications. Proc. of VR, (2012), 25--32.
19. Lee, J.C., Dietz, P.H., Maynes-Aminzade, D., Raskar, R., and Hudson, S.E. Automatic projector calibration with embedded light sensors. UIST, ACM (2004), 123--126.
20. Mine, M.R., van Baar, J., Grundhofer, A., Rose, D., and Yang, B. Projection-Based AR in Disney Theme Parks. Computer 45, 7 (2012), 32--40.
21. Pinhanez, C. The Everywhere Displays Projector: A Device to create ubiquitous graphical interfaces. Proc. UbiComp, (2001), 315--331.
22. ProCams. http://www.procams.org. 2007.
23. Projection Mapping Central. Projection Mapping Central. http://projection-mapping.org, 2014.
24. R. Sekular, R.B. Perception 2nd Edition. (1990).
25. Raskar, R., van Baar, J., Beardsley, P., Willwacher, T., Rao, S., and Forlines, C. iLamps: geometrically aware and self-configuring projectors. ACM TOG 223, (2003), 809--818.
26. Raskar, R. and Beardsley, P. A self-correcting projector. Proc. of CVPR 2001, IEEE Comput. Soc (2001), 504--508.
27. Raskar, R., Brown, M.S., Yang, R., et al. Multiprojector displays using camera-based registration. IEEE Visualization, (1999), 161--168.
28. Raskar, R., Welch, G., Cutts, M., Lake, A., Stesin, L., and Fuchs, H. The office of the future: A unified approach to image-based modeling and spatially immersive displays. Proc. SIGGRAPH, ACM (1998), 179--188.
29. Raskar, R., Welch, G., and Fuchs, H. Spatially Augmented Reality. (1998), 1--7.
30. Raskar, R., Welch, G., Low, K.L., and Bandyopadhyay, D. Shader lamps: Animating real objects with image-based illumination. Proc. Eurographics, Springer Verlag Wien (2001), 89-- 100.
31. Sajadi, B., Member, S., and Majumder, A. AutoCalibration of Multi-Projector CAVE-like Immersive Environments. Trans. on Visualization and Comp. Graphics 17, 1 (2011), 1--13.
32. Schulze, P., Acevedo, D., Mangan, J., et al. Democratizing Rendering for Multiple Viewers in Surround VR Systems. IEEE 3DUI, (2012).
33. Smith, G., Othenin-girard, A., Whitehead, J., and Wardrip-fruin, N. PCG-Based Game Design : Creating Endless Web. In Proc. of Foundations of Digital Games, (2012).
34. Straka, M., Hauswiesner, S., Ruther, M., and Bischof, H. Rapid Skin: Estimating the 3D Human Pose and Shape in Real-Time. Proc of 3DimPVT, Ieee (2012), 41--48.
35. VR, O. http://www.oculusvr.com/. .
36. Willis, K.D.D. SideBySide : Ad-hoc Multi-user Interaction with Handheld Projectors. ACM UIST, (2011), 431--440.
37. Wilson, A., Benko, H., Izadi, S., and Hilliges, O. Steerable augmented reality with the beamatron. Proceedings of the 25th annual ACM symposium on User interface software and technology - UIST '12, ACM Press (2012), 413.
38. Wilson, A. Combining Multiple Depth Cameras and Projectors for Interactions On, Above, and Between Surfaces. ACM CHI, (2010).
39. Wilson, A.D. PlayAnywhere: a compact interactive tabletop projection-vision system. Proc. of UIST, ACM (2005), 92--102.

Index Terms

• Human-centered computing

  • Human computer interaction (HCI)