Improving Collocated Robot Teleoperation with Augmented Reality

ABSTRACT

Robot teleoperation can be a challenging task, often requiring a great deal of user training and expertise, especially for platforms with high degrees-of-freedom (e.g., industrial manipulators and aerial robots). Users often struggle to synthesize information robots collect (e.g., a camera stream) with contextual knowledge of how the robot is moving in the environment. We explore how advances in augmented reality (AR) technologies are creating a new design space for mediating robot teleoperation by enabling novel forms of intuitive, visual feedback. We prototype several aerial robot teleoperation interfaces using AR, which we evaluate in a 48-participant user study where participants completed an environmental inspection task. Our new interface designs provided several objective and subjective performance benefits over existing systems, which often force users into an undesirable paradigm that divides user attention between monitoring the robot and monitoring the robot»s camera feed(s).

References

1. Ronald Azuma, Yohan Baillot, Reinhold Behringer, Steven Feiner, Simon Julier, and Blair MacIntyre. 2001. Recent Advances in Augmented Reality. IEEE Computer Graphics and Applications 21, 6 (2001), 34--47.
2. Maria Bualat, Jonathan Barlow, Terrence Fong, Christopher Provencher, Trey Smith, and Allison Zuniga. 2015. Astrobee: Developing a Free-flying Robot forthe International Space Station. In AIAA SPACE 2015 Conference and Exposition. 4643.
3. Jessie YC Chen, Ellen C Haas, and Michael J Barnes. 2007. Human Performance Issues and User Interface Design for Teleoperated Robots. IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews) 37, 6 (2007), 1231--1245.
4. Mark Corcoran. 2014. Drone Journalism Takes Off. http://www.abc.net.au/news/2012-02--21/drone-journalism-takes-off/3840616. (February 2014). Accessed: 2018-01-05.
5. Enrico Costanza, Samuel A Inverso, Elan Pavlov, Rebecca Allen, and Pattie Maes. 2006. Eye-q: Eyeglass Peripheral Display for Subtle Intimate Notifications. In Proceedings of the ACM Conference on Human-Computer Interaction with Mobile Devices and Services. 211--218.
6. Mike Daily, Youngkwan Cho, Kevin Martin, and Dave Payton. 2003. World embedded interfaces for human-robot interaction. In System Sciences (HICSS'03), 2003. Proceedings of the 36th Annual Hawaii International Conference on. IEEE, 6--pp.
7. Anca D Dragan and Siddhartha S Srinivasa. 2013. A Policy-blending Formalism for Shared Control. The International Journal of Robotics Research 32, 7 (2013), 790--805.
8. Jill L Drury, Justin Richer, Nathan Rackliffe, and Michael A Goodrich. 2006. Comparing Situation Awareness for Two Unmanned Aerial Vehicle Human Interface Approaches. Technical Report. DTIC Document.
9. Steven Feiner, Blair MacIntyre, Marcus Haupt, and Eliot Solomon. 1993. Windows on the World: 2D Windows for 3D Augmented Reality. In Proceedings of the 6th annual ACM Symposium on User Interface Software and Technology (UIST'93). 145--155.
10. T. Fong, M. Micire, T. Morse, E. Park, C. Provencher, V. To, DW. Wheeler, D. Mittman, R. J. Torres, and E. Smith. 2013. Smart SPHERES: A Telerobotic Free-Flyer for Intravehicular Activities in Space. In Proceedings of AIAA Space'13.
11. Foyle, David C and Andre, Anthony D and Hooey, Becky L. 2005. Situation Awareness in an Augmented Reality Cockpit: Design, Viewpoints and Cognitive Glue. In Proceedings of the 11th International Conference on Human Computer Interaction. 3--9.
12. Michael A Goodrich, Bryan S Morse, Damon Gerhardt, Joseph L Cooper, Morgan Quigley, Julie A Adams, and Curtis Humphrey. 2008. Supporting Wilderness Search and Rescue using a Camera-equipped Mini UAV. Journal of Field Robotics 25, 1--2 (2008), 89--110.
13. Scott A Green, Mark Billinghurst, XiaoQi Chen, and J Geoffrey Chase. 2008. Human-Robot Collaboration: A Literature Review and Augmented Reality Approach in Design. International Journal of Advanced Robotic Systems 5, 1 (2008), 1.
14. Scott A Green, J Geoffrey Chase, XiaoQi Chen, and Mark Billinghurst. 2009. Evaluating the Augmented Reality Human-Robot Collaboration System. International Journal of Intelligent Systems Technologies and Applications 8, 1--4 (2009), 130--143.
15. Drew Harwell. 2014. This government rule could cripple commercial drone flight. https://www.washingtonpost.com/blogs/the-switch/wp/2014/11/25/this-government-rule-could-cripple-commercial-drone-flight/. (2014).
16. Keita Higuchi, Katsuya Fujii, and Jun Rekimoto. 2013. Flying Head: A Headsynchronization Mechanism for Flying Telepresence. In International Conference on Artificial Reality and Telexistence (ICAT'13). 28--34.
17. Yoshio Ishiguro and Jun Rekimoto. 2011. Peripheral Vision Annotation: Noninterference Information Presentation Method for Mobile Augmented Reality. In Proceedings of the ACM Augmented Human International Conference. 8.
18. J Richard Landis and Gary G Koch. 1977. The Measurement of Observer Agreement for Categorical Data. Biometrics 33 (1977), 159--174. Issue 1.
19. Paula Lavigne. 2014. Eyes in the Sports Sky. ESPN website. (2014). Accessed: 2018-01-05.
20. Li, Hua and Zhang, Xin and Shi, Guangwei and Qu, Hemeng and Wu, Yanxiong and Zhang, Jianping. 2013. Review and Analysis of Avionic Helmet-Mounted Displays. Optical Engineering 52, 11 (2013), 110901--110901.
21. Jeffrey I Lipton, Aidan J Fay, and Daniela Rus. 2018. Baxter's Homunculus: Virtual Reality Spaces for Teleoperation in Manufacturing. IEEE Robotics and Automation Letters 3, 1 (2018), 179--186.
22. M Alejandra Menchaca-Brandan, Andrew M Liu, Charles M Oman, and Alan Natapoff. 2007. Influence of Perspective-taking and Mental Rotation Abilities in Space Teleoperation. In ACM/IEEE International Conference on Human-Robot Interaction (HRI'07). 271--278.
23. Paul Milgram and Fumio Kishino. 1994. A Taxonomy of Mixed Reality Visual Displays. IEICE Transactions on Information and Systems 77, 12 (1994), 1321-- 1329.
24. Paul Milgram, Shumin Zhai, David Drascic, and Julius Grodski. 1993. Applications of Augmented Reality for Human-Robot Communication. In Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS'93), Vol. 3. 1467--1472.
25. Masmoudi Mostefa, L Kaddour El Boudadi, A Loukil, Khelf Mohamed, and Dahane Amine. 2015. Design of Mobile Robot Teleoperation System based on Virtual Reality. In IEEE International Conference on Control, Engineering&Information Technology (CEIT'15). 1--6.
26. Robin R Murphy. 2014. Disaster Robotics. MIT Press.
27. Nicholl, Ryan. 2014. Airline Head-up Display Systems: Human Factors Considerations. Available at SSRN:https:// ssrn.com/ abstract=2384101 (2014).
28. Curtis W Nielsen, Michael A Goodrich, and Robert W Ricks. 2007. Ecological Interfaces for Improving Mobile Robot Teleoperation. IEEE Transactions on Robotics 23, 5 (2007), 927--941.
29. Shahin S Nudehi, Rohan Mukherjee, and Moji Ghodoussi. 2003. A Haptic Interface Design for Minimally Invasive Telesurgical Training and Collaboration in the Presence of Time Delay. In Proceedings. 42nd IEEE Conference on Decision and Control, Vol. 5. 4563--4568.
30. Allison M Okamura. 2004. Methods for Haptic Feedback in Teleoperated Robotassisted Surgery. Industrial Robot: An International Journal 31, 6 (2004), 499--508.
31. Christos Papachristos and Kostas Alexis. 2016. Augmented Reality-Enhanced Structural Inspection Using Aerial Robots. In IEEE International Symposium on Intelligent Control (ISIC'16). IEEE, 1--6.
32. FJ Perez-Grau, R Ragel, F Caballero, A Viguria, and A Ollero. 2017. Semiautonomous Teleoperation of UAVs in Search and Rescue Scenarios. In International Conference on Unmanned Aircraft Systems (ICUAS'17). 1066--1074.
33. David Pitman and Mary L Cummings. 2012. Collaborative exploration with a Micro Aerial Vehicle: A Novel Interaction Method for Controlling a MAV with a Hand-Held Device. Advances in Human-Computer Interaction 2012 (2012), 18.
34. Benjamin Poppinga, Niels Henze, Jutta Fortmann, Wilko Heuten, and Susanne Boll. 2012. AmbiGlasses-Information in the Periphery of the Visual Field. In Mensch&Computer. 153--162.
35. Morgan Quigley, Michael A Goodrich, and Randal W Beard. 2004. Semiautonomous human-UAV interfaces for fixed-wing mini-UAVs. In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS'04), Vol. 3. IEEE, 2457--2462.
36. Emanuele Ruffaldi, Filippo Brizzi, Franco Tecchia, and Sandro Bacinelli. 2016. Third Point of View Augmented Reality for Robot Intentions Visualization. In International Conference on Augmented Reality, Virtual Reality and Computer Graphics. 471--478.
37. Daniel Szafir. 2015. Human Interaction with Assistive Free-Flying Robots. Ph.D. Dissertation. The University of Wisconsin-Madison.
38. Daniel Szafir, Bilge Mutlu, and Terrence Fong. 2017. Designing Planning and Control Interfaces to Support User Collaboration with Flying Robots. The International Journal of Robotics Research 36, 5--7, 514--542.
39. Russell H Taylor, Arianna Menciassi, Gabor Fichtinger, Paolo Fiorini, and Paolo Dario. 2016. Medical Robotics and Computer-Integrated Surgery. In Springer Handbook of Robotics. 1657--1684.
40. John Thomason, Photchara Ratsamee, Kiyoshi Kiyokawa, Pakpoom Kriangkomol, Jason Orlosky, Tomohiro Mashita, Yuki Uranishi, and Haruo Takemura. 2017. Adaptive View Management for Drone Teleoperation in Complex 3D Structures. In Proceedings of the ACM International Conference on Intelligent User Interfaces (IUI'17). ACM, 419--426.

Index Terms

• Computer systems organization

  • Embedded and cyber-physical systems

    • Robotics

      • External interfaces for robotics

• General and reference

  • Cross-computing tools and techniques

    • Design

• Human-centered computing

  • Human computer interaction (HCI)

    • HCI design and evaluation methods

      • Usability testing

      • User studies

    • Interaction paradigms

      • Mixed / augmented reality

  • Interaction design

    • Interaction design process and methods

      • Interface design prototyping