Virtual Replicas for Remote Assistance in Virtual and Augmented Reality

ABSTRACT

In many complex tasks, a remote subject-matter expert may need to assist a local user to guide actions on objects in the local user's environment. However, effective spatial referencing and action demonstration in a remote physical environment can be challenging. We introduce two approaches that use Virtual Reality (VR) or Augmented Reality (AR) for the remote expert, and AR for the local user, each wearing a stereo head-worn display. Both approaches allow the expert to create and manipulate virtual replicas of physical objects in the local environment to refer to parts of those physical objects and to indicate actions on them. This can be especially useful for parts that are occluded or difficult to access. In one approach, the expert points in 3D to portions of virtual replicas to annotate them. In another approach, the expert demonstrates actions in 3D by manipulating virtual replicas, supported by constraints and annotations. We performed a user study of a 6DOF alignment task, a key operation in many physical task domains, comparing both approaches to an approach in which the expert uses a 2D tablet-based drawing system similar to ones developed for prior work on remote assistance. The study showed the 3D demonstration approach to be faster than the others. In addition, the 3D pointing approach was faster than the 2D tablet in the case of a highly trained expert.

References

1. M. Adcock, D. Ranatunga, R. Smith, and B.H. Thomas. 2014. Object-Based Touch Manipulation for Remote Guidance of Physical Tasks. In Proc. ACM SUI. 113--122.
2. M. Bauer, G. Kortuem, and Z. Segall. 1999. "Where Are You Pointing At?" A Study of Remote Collaboration in a Wearable Video conference System. In Proc. IEEE ISWC. 151--158.
3. S. Bottecchia, J. Cieutat, and J. Jessel. 2010. T.A.C: Augmented Reality System for Collaborative Tele-assistance in the Field of Maintenance Through Internet. In Proc. ACM AH. 14:1--14:7.
4. J. Chastine, K. Nagel, Y. Zhu, and M. Hudachek-Buswell. 2008. Studies on the Effectiveness of Virtual Pointers in Collaborative Augmented Reality. In Proc. IEEE 3DUI. 117--124.
5. S. Gauglitz, B. Nuernberger, M. Turk, and T. Hollerer. 2014. World-Stabilized Annotations And Virtual Scene Navigation For Remote Collaboration. In Proc. ACM UIST. 449--459.
6. M. Goto, Y. Uematsu, H. Saito, S. Senda, and A Iketani. 2010. Task Support System by Displaying Instructional Video Onto AR Workspace. In Proc. IEEE ISMAR. 83--90.
7. J. Heiser, B. Tversky, and M. Silverman. 2004. Sketches for and from Collaboration. Visual and spatial reasoning in design III 3 (2004), 69--78.
8. S. Henderson and S. Feiner. 2011. Augmented Reality in the Psychomotor Phase of a Procedural Task. In Proc. IEEE ISMAR. 191--200.
9. S. Kim, G. Lee, N. Sakata, and M. Billinghurst. 2014. Improving Co-Presence with Augmented Visual Communication Cues for Sharing Experience through Video Conference. In Proc. IEEE ISMAR. 83--92.
10. D.S. Kirk and D.S. Fraser. 2005. The Effects of Remote Gesturing on Distance Instruction. In Proc. CSCL. 301--310.
11. T. Kurata, N. Sakata, M. Kourogi, H. Kuzuoka, and M. Billinghurst. 2004. Remote Collaboration Using a Shoulder-Worn Active Camera/laser. In Proc. IEEE ISWC, Vol. 1. 62--69.
12. G. Kurillo, R. Bajcsy, K. Nahrsted, and O. Kreylos. 2008. Immersive 3D Environment for Remote Collaboration and Training of Physical Activities. In Proc. IEEE VR. 269--270.
13. H. Kuzuoka. 1992. Spatial Workspace Collaboration: A SharedView Video Support System for Remote Collaboration Capability. In Proc. ACM CHI. 533--540.
14. J. Lanir, R. Stone, B. Cohen, and P. Gurevich. 2013. Ownership and Control of Point of View in Remote Assistance. In Proc. ACM CHI. 2243--2252.
15. R.A. Newcombe, S. Izadi, O. Hilliges, D. Molyneaux, D. Kim, A.J. Davison, P. Kohli, J. Shotton, S. Hodges, and A. Fitzgibbon. 2011. KinectFusion: Real-time Dense Surface Mapping and Tracking. In Proc. IEEE ISMAR. 127--136.
16. O. Oda and S. Feiner. 2015. Goblin XNA Framework. (2015). http://goblinxna.codeplex.com/
17. J.S. Pierce, B.C. Stearns, and R. Pausch. 1999. Voodoo Dolls: Seamless Interaction at Multiple Scales in Virtual Environments. In Proc. ACM i3D. 141--145.
18. T. Piumsomboon, D. Altimira, H. Kim, A. Clark, G. Lee, and M. Billinghurst. 2014. Grasp-Shell vs Gesture-Speech: A Comparison of Direct and Indirect Natural Interaction Techniques in Augmented Reality. In Proc. IEEE ISMAR. 73--82.
19. R Core Team. 2015. R: A Language and Environment for Statistical Computing. http://www.R-project.org/
20. G. Rizzolatti and L. Craighero. 2004. The Mirror-Neuron System. Annu. Rev. Neurosci. 27 (2004), 169--192.
21. N. Sakata, T. Kurata, and H. Kuzuoka. 2006. Visual Assist with a Laser Pointer and Wearable Display for Remote Collaboration. In Proc CollabTech. 66--71.
22. I. Sipiran and B. Bustos. 2011. Harris 3D: A Robust Extension of the Harris Operator for Interest Point Detection on 3D Meshes. Vis. Comput. 27, 11 (2011), 963--976.
23. R.S. Sodhi, B.R. Jones, D. Forsyth, B.P. Bailey, and G. Maciocci. 2013. BeThere: 3D Mobile Collaboration with Spatial Input. In Proc. ACM CHI. 179--188.
24. A. Stafford and W. Piekarski. 2008. User Evaluation of God-like Interaction Techniques. In Proc. AUIC. 19--27.
25. M. Tait and M. Billinghurst. 2014. View independence in remote collaboration using AR. In Proc. IEEE ISMAR. 309--310.
26. L. Talmy. 2003. Toward a Cognitive Semantics. Vol. 1--2. MIT press.
27. M. Tatzgern, R. Grasset, D. Kalkofen, and D. Schmalstieg. 2014. Transitional Augmented Reality Navigation for Live Captured Scenes. In Proc. IEEE VR. 21--26.
28. F. Tecchia, L. Alem, and W. Huang. 2012. 3D Helping Hands: A Gesture Based MR System for Remote Collaboration. In Proc. ACM VRCAI. 323--328.
29. S.G. Vandenberg and A.R. Kuse. 1978. Mental Rotations, a Group Test of Three-dimensional Spatial Visualization. Perceptual and motor skills 47, 2 (1978), 599--604.
30. R. Wang, S. Paris, and J. Popovi'c. 2011. 6D Hands: Markerless Hand-tracking for Computer Aided Design. In Proc. ACM UIST. 549--558.
31. M. Wexler, S.M. Kosslyn, and A. Berthoz. 1998. Motor Processes in Mental Rotation. Cognition 68, 1 (1998), 77--94.

Index Terms

• Computing methodologies

  • Computer graphics

    • Graphics systems and interfaces

• Human-centered computing

  • Collaborative and social computing

    • Collaborative and social computing theory, concepts and paradigms

      • Computer supported cooperative work

  • Human computer interaction (HCI)

    • Interaction devices

      • Graphics input devices

• Social and professional topics

  • Professional topics

    • Computing and business

      • Computer supported cooperative work