Trends in augmented reality tracking, interaction and display: A review of ten years of ISMAR

ABSTRACT

Although Augmented Reality technology was first developed over forty years ago, there has been little survey work giving an overview of recent research in the field. This paper reviews the ten-year development of the work presented at the ISMAR conference and its predecessors with a particular focus on tracking, interaction and display research. It provides a roadmap for future augmented reality research which will be of great value to this relatively young field, and also for helping researchers decide which topics should be explored when they are beginning their own studies in the area.

References

1. M. Anabuki and H. Ishii. AR-Jig: A Handheld Tangible User Interface for Modification of 3D Digital Form via 2D Physical Curve. In ISMAR '07, pp. 55-66, 2007.
2. T. Auer and A. Pinz. Building a hybrid tracking system: Integration of optical and magnetic tracking. In IWAR '99, pp.13-22, 1999.
3. R.T. Azuma. A survey of augmented reality. Presence: Teleoperators and Virtual Environments 6:4, 355-385, 1997.
4. R.T. Azuma, Y. Baillot, R. Behringer, S. Feiner, S. Julier, and B. MacIntyre. Recent advances in augmented reality. IEEE Computer Graphics & Applications, 21:6, 34-47, 2001.
5. R.T. Azuma, B.R. Hoff, I. Howard E. Neely, R. Sarfaty, M. J. Daily, G. Bishop, L. Vicci, G. Welch, U. Neumann, S. You, R. Nichols, and J. Cannon. Making augmented reality work outdoors requires hybrid tracking. In IWAR '98, pp. 219-224, 1998.
6. M. Bajura and N. Ulrich. Dynamic registration correction in videobased augmented reality systems. IEEE Computer Graphics and Applications, 15:5 52-60, September 1995.
7. H. Benko, E.W. Ishak and S. Feiner. Collaborative mixed reality visualization of an archaeological excavation. In ISMAR '04, pp. 132-140, 2004.
8. M. Billinghurst and H. Kato. Collaborative augmented reality. Communications of the ACM, 45:7, 64-70, 2002.
9. M. Billinghurst, S. Weghorst and T. Furness III. Shared Space: An augmented reality interface for computer supported collaborative work. In CVE '96, 1996.
10. O. Bimber and B. Fröhlich. Occlusion shadows: using projected light to generate realistic occlusion effects for view-dependent optical see-through displays. In ISMAR '02, pp. 186-319, 2002.
11. O. Bimber, A. Grundhofer, G. Wetzstein and S. Knodel. Consistent illumination within optical see-through augmented environments. In ISMAR '03, pp. 198-207, 2003.
12. O. Bimber and R. Raskar. Spatial augmented reality: Merging real and virtual worlds. A K Peters LTD, 2005.
13. O. Bimber and G. Wetzstein, A. Emmerling, and C. Nitschke. Enabling view-dependent stereoscopic projection in real environments. In ISMAR '05, pp. 14-23, 2005.
14. G. Bleser, H. Wuest and D. Stricker. Online camera pose estimation in partially known and dynamic scenes. In ISMAR '06, pp. 56-65, 2006.
15. A. Butz, T. Hollerer, S. Feiner, B. MacIntyre and C. Beshers. Enveloping users and computers in a collaborative 3D augmented reality. In IWAR '99, pp. 35-44, 1999.
16. O. Cakmakci and J. Rolland. Head-Worn Displays: A Review. IEEE/OSA J. Display Technology 2:3, 199-216, 2006.
17. Y. Cho, J. Lee and U. Neumann. A multi-ring fiducial system and an intensity-invariant detection method for scalable augmented reality. In IWAR '98, pp. 147-156, 1998.
18. A. Comport, E. Marchand and F. Chaumette. A real-time tracker for markerless augmented reality. In ISMAR '03, pp. 36-45, 2003.
19. D. Cotting, M. Naef, M. Gross and H. Fuchs. Embedding imperceptible patterns into projected images for simultaneous acquisition and display. In ISMAR '04, pp. 100-109, 2004.
20. K. Dorfmüller-Ulhaas and D. Schmalstieg. Finger tracking for interaction in augmented environments. In ISAR '01, pp. 55-64, 2001.
21. A. Doucet, N. De Freitas and N. Gordon, editors. Sequential Monte Carlo in Practice. Springer-Verlag, New York, 2001.
22. J. Ehnes, K. Hirota and M. Hirose. Projected augmentation augmented reality using rotatable video projectors. In ISMAR '04, pp. 26-35, 2004.
23. S. Eitoku, T. Tanikawa and M. Hirose. Display composed of water drops for filling space with materialized virtual three-dimensional objects. In IEEE VR '06, pp. 159-166, 2006.
24. S. Feiner, B. MacIntyre, T. Höllerer and T. Webster. A touring machine: prototyping 3D mobile augmented reality systems for exploring the urban environment. In ISWC '97, pp. 74-81, 1997.
25. J. Fergason. Optical system for head mounted display using retroreflector and method of displaying an image. U.S. patent 5,621,572, April 15, 1997.
26. E. Foxlin, Y. Altshuler, L. Naimark, and M. Harrington. Flight Tracker: A novel optical/inertial tracker for cockpit enhanced vision. In ISMAR '04, pp. 212-221, 2004.
27. P. Fua and V. Lepetit. Vision based 3D tracking and pose estimation for mixed reality. In M. Haller, M. Billinghurst, B. H. Thomas Eds, Emerging Technologies of Augmented Reality Interfaces and Design, pp. 43-63, Idea Group, Hershey, 2007.
28. C. Geiger, B. Kleinjohann, C. Reimann and D. Stichling. Mobile AR4ALL. In ISAR '01, pp. 181-182, 2001.
29. Y. Genc, S. Riedel, F. Souvannavong, C. Akinlar and N. Navab. Marker-less tracking for AR: A learning-based approach. In ISMAR '02, pp. 295-304, 2002.
30. J. J. Gibson. The theory of affordances. in R. E. Shaw and J. Bransford, Eds, Perceiving, Acting, and Knowing. Hillsdale, NJ: Lawrence Erlbaum Associates, 1977.
31. M. Gorbet, M. Orth and H. Ishii. Triangles: tangible interface for manipulation and exploration of digital information topography. In CHI '98, pp. 49-56, 1998.
32. S. Gupta and C.O. Jaynes. The universal media book: tracking and augmenting moving surfaces with projected information. In ISMAR '06, page 177-180, 2006.
33. A. Henrysson, M. Billinghurst and M. Ollila. Face to face collaborative AR on mobile phones. In ISMAR '05, pp. 80-89, 2005.
34. K. Higa, T. Nishiura, A. Kimura, F. Shibata and H. Tamura. A twoby-two mixed reality system that merges real and virtual worlds in both audio and visual senses. In ISMAR '07, pp. 203-206. 2007.
35. T. Höllerer, S. Feiner, T. Terauchi, G. Rashid and D. Hallaway. Exploring MARS: developing indoor and outdoor user interfaces to a mobile augmented reality system. Computers and Graphics, 23:6, 779-785, 1999.
36. H. Hua, C Gao and L.D. Brown, J.P. Rolland. Using a head-mounted projective display in interactive augmented environments. In ISAR '01, pp. 217-223, 2001.
37. H. Hua, A. Girardot, C. Gao and J.P. Rolland. Engineering of headmounted projective displays. Applied Optics, 39:22, 3814-3824, 2000.
38. M. Huber, D. Pustka, P. Keitler, F. Echtler and G. Klinker. A system architecture for ubiquitous tracking environments. In ISMAR '07, pp. 211-214, 2007.
39. S. Irawati, S. Green, M. Billinghurst, A. Duenser and K. Heedong. "Move the couch where?" developing an augmented reality multimodal interface. In ISMAR '06, pp. 183-186, 2006.
40. H. Ishii, B. Ullmer. Tangible bits: towards seamless interfaces between people, bits and atoms. In ACM CHI'97, pp. 234-241, 1997.
41. M. Kanbara, H. Fujii, H. Takemura and N. Yokoya. A stereo visionbased augmented reality system with an inertial sensor. In ISAR '00, pp. 97-100, 2000.
42. H. Kato and M. Billinghurst. Marker tracking and HMD calibration for a video-based augmented reality conferencing system. In IWAR '99, pp. 85-94, 1999.
43. H. Kato, M. Billinghurst, I. Poupyrev, K. Imamoto and K. Tachibana. Virtual object manipulation on a table-top AR environment. In ISAR '00, pp. 111-119, 2000.
44. K. Kiyokawa. An introduction to head mounted displays for augmented reality. In M. Haller, M. Billinghurst, B. H. Thomas Eds, Emerging Technologies of Augmented Reality Interfaces and Design, pp. 43-63, Idea Group, Hershey, 2007.
45. K. Kiyokawa, M. Billinghurst, B. Campbell and E. Woods. An occlusion-capable optical see-through head mount display for supporting co-located collaboration. In ISMAR '03, pp. 133-141, 2003.
46. K. Kiyokawa, M. Billinghurst, S. E. Hayes, A. Gupta, Y. Sannohe and H. Kato. Communication behaviors of co-located users in collaborative AR interfaces. In ISMAR '02, pp. 139-148, 2002.
47. K. Kiyokawa, Y. Kurata and H. Ohno, An optical see-through display for mutual occlusion of real and virtual environments. In ISAR '00, pp. 60-67, 2000.
48. G. Klein and T. Drummond. Robust visual tracking for noninstrumented augmented reality. In ISMAR '03, pp. 113-122, 2003.
49. G. Klein and D. Murray. Parallel Tracking and Mapping for Small AR Workspaces. In ISMAR '07, pp. 225-234, 2007.
50. G. Klinker, R. Reicher and B. Brugge. Distributed user tracking concepts for augmented reality applications. ISAR'00, pp. 37-44, 2000.
51. P. Lang, A. Kusej, A. Pinz and G. Brasseur, Inertial tracking for mobile augmented reality. In IEEE IMTC, Vol.2, pp. 1583-1587, 2002.
52. S. Malik, C. McDonald and G. Roth. Tracking for interactive pattern-based augmented reality. In ISMAR '02, pp. 117-126, 2002.
53. M. Möhring, C. Lessig and O. Bimber. Video see-through AR on consumer cell-phones. In ISMAR '04, pp. 252-253, 2004.
54. L. Naimark and E. Foxlin. Circular data matrix fiducial system and robust image processing for a wearable vision-inertial self-tracker. In ISMAR '02, pp. 27-36, 2002.
55. L. Naimark, E. Foxlin. Encoded LED system for optical trackers. In ISMAR '05, pp. 150-153, 2005.
56. J. Newman, M. Wagner, M. Bauer, A. MacWilliams, T. Pintaric, D. Beyer, D. Pustka, F. Strasser, D. Schmalstieg and G. Klinker. Ubiquitous tracking for augmented reality. In ISMAR '04, pp. 192-201, 2004.
57. J. Newman, D. Ingram and A. Hopper. Augmented reality in a wide area sentient environment. In ISMAR '01, pp. 77-86, 2001.
58. J. Neubert, J. Pretlove and T. Drummond. Semi-autonomous generation of appearance-based edge models from image sequences. In ISMAR '07, pp. 79-89, 2007.
59. J. Nielsen. (1994). Usability Engineering (1st ed.). San Francisco: Morgan Kaufmann.
60. D. A. Norman. The Design of Everyday Things. New York: Doubleday Business, 1988.
61. A. Olwal, C. Lindfors, J. Gustafsson, T. Kjellberg and L. Mattsson. ASTOR: An auto-stereoscopic optical see-through augmented reality system. In ISMAR '05, pp. 24-27, 2005.
62. J. Park, B. Jiang and U. Neumann. Vision-based pose computation: robust and accurate augmented reality tracking. In IWAR '99, pp. 3-12, 1999.
63. J. Park, S. You and U. Neumann. Natural feature tracking for extendible robust augmented realities. In IWAR '98, 1998.
64. C. Pinhanez. The Everywhere Displays Projector: A Device to Create Ubiquitous Graphical Interfaces, Proc. Ubiquitous Computing 2001 (Ubicomp '01), Atlanta, Georgia, September 2001, 315-331. Springer Lecture Notes In Computer Science, v. 2201.
65. A. Pinz, M. Brandner, H. Ganster, A. Kusej, P. Lang and M. Ribo. Hybrid tracking for augmented reality. ÖGAI Journal, 21:1 17-24, 2002.
66. M. Pressigout and É. Marchand. Hybrid tracking algorithms for planar and non-planar structures subject to illumination changes. In ISMAR '06, pp. 52-55, 2006.
67. G. Reitmayr E. Eade and T.W. Drummond. Localisation and interaction for augmented maps. In ISMAR '05, pp. 120-129, 2005.
68. G. Reitmayr and T. Drummond. Going out: robust model-based tracking for outdoor augmented reality. In ISMAR '06, pp. 109-118, 2006.
69. G. Reitmayr and T. Drummond. Initialisation for Visual Tracking in Urban Environments, In ISMAR '07, pp. 161-172, 2007.
70. G. Reitmayr and D. Schmalstieg. Mobile collaborative augmented reality. In ISAR '01, pp. 114-123, 2001.
71. J. Rekimoto. Matrix: A realtime object identification and registration method for augmented reality. APCHI '98, pp.63-68, 1998.
72. L.S. Rodrigo Silva. Introduction to augmented reality. http://virtual.lncc.br/~rodrigo/links/AR/node19.html. 2003.
73. J. P. Rolland, L. Davis and Y. Baillot., A survey of tracking technology for virtual environments, In Fundamentals of Wearable Computers and Augmented Reality, 1st ed, W. Barfield and T. Caudell, Eds. Mahwah, NJ: CRC, 2001, pp. 67-112.
74. C. Sandor, A. Olwal, B. Bell and S. Feiner. Immersive mixed-reality configuration of hybrid user interfaces. In ISMAR '05, pp. 110-113, 2005.
75. K. Satoh, S. Uchiyama and H. Yamamoto. A head tracking method using bird's-eye view camera and gyroscope. In ISMAR '04, pp. 202-211, 2004.
76. K. Satoh, S. Uchiyama, H. Yamamoto and H. Tamura. Robust vision-based registration utilizing bird's-eye view with user's view. In ISMAR '03, pp. 46-55, 2003.
77. D. Schmalstieg, A. Fuhrmann and G. Hesina. Bridging multiple user interface dimensions with augmented reality. In ISAR '00, pp. 20-29, 2000.
78. D. Schmalstieg, A. Fuhrmann, Z. Szalavari and M. Gervautz. 1996. Studierstube-An environment for collaboration in augmented reality. In CVE '96, 1996.
79. D. Schmalstieg and D. Wagner. Experiences with Handheld Augmented Reality. In ISMAR '07, pp.3-18, 2007.
80. A. Singer, D. Hindus, L. Stifelman and S. Whitel. Tangible progress: less is more in somewire audio spaces. In CHI, pp. 104-111, 1999.
81. A. Stafford, W. Piekarski and B.H. Thomas. Implementation of god like interaction techniques for supporting collaboration between outdoor AR and indoor tabletop users. In ISMAR '06, pp. 165-172, 2006.
82. A. State, K. Keller and H. Fuchs. Simulation-based design and rapid prototyping of a parallax-free, orthoscopic video see-through headmounted display. In ISMAR '05, pp. 28-31, 2005.
83. I. Skrypnyk and D.G. Lowe. Scene modeling, recognition and tracking with invariant image features. In ISMAR '04, pp. 110-119, 2004.
84. D. Stricker, G. Klinker and D. Reiners, A fast and robust line-based optical tracker for augmented reality applications. In IWAR '98, pp. 31-46, 1998.
85. L. Suchman. Plans and situated actions: The problem of human machine communication, Cambridge: Cambridge University Press, 1987.
86. I. Sutheland, The ultimate display. In IFIP '65, pp. 506-508, 1965.
87. S. Thrun, W. Burgard and D. Fox. Probabilistic Robotics. MIT Press, 2005.
88. L. Vacchetti, V. Lepetit and P. Fua. Combining edge and texture information for real-time accurate 3d camera tracking. In ISMAR '04, pp. 48-57, 2004.
89. S. Vogt, A. Khamene, F. Sauer and H. Niemann. Single camera tracking of marker clusters: Multiparameter cluster optimization and experimental verification. In ISMAR '02, pp. 127-136, 2002.
90. D. Wagner, T. Pintaric, F. Ledermann and D. Schmalstieg. Towards Massively Multi-user Augmented Reality on Handheld Devices. In Pervasive '05, pp.208-219, 2005.
91. D. Wagner and D. Schmalstieg. Artoolkit on the pocketpc platform. Tech. Rep. TR-188-2-2003-23, Technical Univ. of Vienna, 2003.
92. D. Wagner and D. Schmalstieg. Handheld augmented reality displays. In IEEE Virtual Reality Conference, p. 321-321, 2006.
93. D. Wagner and I. Barakonyi. Augmented reality Kanji learning. In ISMAR '03, pp. 335-336, 2003.
94. M. Weiser. The computer for the twenty-first century. Scientific American, 265:3, 94-104, 1991.
95. S. White, L. Lister, and S. Feiner. Visual hints for tangible gestures in augmented reality. In ISMAR '07, pp. 47-50, 2007.
96. H. Wuest, F. Vial and D. Stricker. Adaptive line tracking with multiple hypotheses for augmented reality. In ISMAR '05, pp. 62-69, 2005.
97. X. Zhang, S. Fronz and N. Navab. Visual marker detection and decoding in AR systems: a comparative study. In ISMAR '02, pp. 97- 106, 2002.

Index Terms

• Applied computing

  • Document management and text processing

    • Document capture

      • Graphics recognition and interpretation

• Computing methodologies

  • Artificial intelligence

    • Computer vision

      • Image and video acquisition

        • 3D imaging

  • Computer graphics

    • Animation

    • Graphics systems and interfaces

      • Virtual reality

    • Rendering

• Human-centered computing

  • Human computer interaction (HCI)

    • Interaction devices

      • Graphics input devices

    • Interaction paradigms

      • Mixed / augmented reality

      • Virtual reality