The oz of wizard: simulating the human for interaction research

ABSTRACT

The Wizard of Oz experiment method has a long tradition of acceptance and use within the field of human-robot interaction. The community has traditionally downplayed the importance of interaction evaluations run with the inverse model: the human simulated to evaluate robot behavior, or Oz of Wizard. We argue that such studies play an important role in the field of human-robot interaction. We differentiate between methodologically rigorous human modeling and placeholder simulations using simplified human models. Guidelines are proposed for when Oz of Wizard results should be considered acceptable. This paper also describes a framework for describing the various permutations of Wizard and Oz states.

References

1. Kelley, J. F. 1984. An iterative design methodology for user-friendly natural language office information applications. ACM Trans. Inf. Syst. 2, 1 (Jan. 1984), 26--41. DOI= http://doi.acm.org/10.1145/357417.357420
2. Shiomi, M., Kanda, T., Koizumi, S., Ishiguro, H., and Hagita, N. 2007. Group attention control for communication robots with wizard of OZ approach. In Proceedings of the ACM/IEEE international Conference on Human-Robot interaction (Arlington, Virginia, USA, March 10 - 12, 2007). HRI '07. ACM, New York, NY, 121--128. DOI= http://doi.acm.org/10.1145/1228716.1228733
3. Kahn, P. H., Freier, N. G., Kanda, T., Ishiguro, H., Ruckert, J. H., Severson, R. L., and Kane, S. K. 2008. Design patterns for sociality in human-robot interaction. In Proceedings of the 3rd ACM/IEEE International Conference on Human Robot Interaction (Amsterdam, The Netherlands, March 12 - 15, 2008). HRI '08. ACM, New York, NY, 97--104. DOI= http://doi.acm.org/10.1145/1349822.1349836
4. Shiomi, M., Sakamoto, D., Takayuki, K., Ishi, C. T., Ishiguro, H., and Hagita, N. 2008. A semi-autonomous communication robot: a field trial at a train station. In Proceedings of the 3rd ACM/IEEE International Conference on Human Robot Interaction (Amsterdam, The Netherlands, March 12 - 15, 2008). HRI '08. ACM, New York, NY, 303--310. DOI= http://doi.acm.org/10.1145/1349822.1349862
5. Trafton, J. G., Schultz, A. C., Perznowski, D., Bugajska, M. D., Adams, W., Cassimatis, N. L., and Brock, D. P. 2006. Children and robots learning to play hide and seek. In Proceedings of the 1st ACM SIGCHI/SIGART Conference on Human-Robot Interaction (Salt Lake City, Utah, USA, March 02 - 03, 2006). HRI '06. ACM, New York, NY, 242--249. DOI= http://doi.acm.org/10.1145/1121241.1121283
6. Granda, T., Kirkpatrick, M., Julien, T., and Peterson, L. 1990. The evolutionary role of humans in the human-robot system. In Proc. Human Factors Society 34th Annual Meeting. 664--668.
7. Steinfeld, A., Fong, T., Kaber, D., Lewis, M., Scholtz, J., Schultz, A., and Goodrich, M. 2006. Common metrics for human-robot interaction. In Proceedings of the 1st ACM SIGCHI/SIGART Conference on Human-Robot Interaction (Salt Lake City, Utah, USA, March 02 - 03, 2006). HRI '06. ACM, New York, NY, 33--40. DOI= http://doi.acm.org/10.1145/1121241.1121249
8. Forsyth, D. and Ponce, J. 2003. Computer Vision A Modern Approach, Prentice Hall.
9. Jenkins, O., González, G., and Loper, M. 2007. Interactive human pose and action recognition using dynamical motion primitives. International Journal of Humanoid Robotics, 4(2):365--385.
10. Grollman, D. H., and Jenkins, O. C. 2008. Sparse incremental learning for interactive robot control policy estimation. In International Conference on Robotics and Automation (ICRA, Pasadena, CA, USA, May 2008).
11. Chernova, S. and Veloso, M. 2008. Multi-thresholded approach to demonstration selection for interactive robot learning. In Proceedings of the 3rd ACM/IEEE International Conference on Human Robot Interaction (Amsterdam, The Netherlands, March 12 - 15, 2008). HRI '08. ACM, New York, NY, 225--232. DOI= http://doi.acm.org/10.1145/1349822.1349852
12. Doshi, F. and Roy, N. 2007. Efficient model learning for dialog management. In Proceedings of the ACM/IEEE International Conference on Human-Robot Interaction (Arlington, Virginia, USA, March 10 - 12, 2007). HRI '07. ACM, New York, NY, 65--72. DOI= http://doi.acm.org/10.1145/1228716.1228726
13. Schmidt-Rohr, S. R., Knoop, S., Lösch, M., and Dillmann, R. 2008. Reasoning for a multi-modal service robot considering uncertainty in human-robot interaction. In Proceedings of the 3rd ACM/IEEE International Conference on Human Robot Interaction (Amsterdam, The Netherlands, March 12 - 15, 2008). HRI '08. ACM, New York, NY, 249--254. DOI= http://doi.acm.org/10.1145/1349822.1349855
14. Vondrak, M., Sigal, L., and Jenkins, O. 2008. Physical simulation for probabilistic motion tracking. In Computer Vision and Pattern Recognition (CVPR, Anchorage, AK, USA, Jun 2008).
15. Srinivasa, S., Ferguson, D., Vande Weghe, M., Diankov, R., Berenson, D., Helfrich, C. and Strasdat, H. 2008. The Robotic Busboy: steps towards developing a mobile robotic home assistant. In International Conference on Intelligent Autonomous Systems.
16. Humphrey, C. M. and Adams, J. A. 2008. Compass visualizations for human-robotic interaction. In Proceedings of the 3rd ACM/IEEE International Conference on Human Robot Interaction (Amsterdam, The Netherlands, March 12 - 15, 2008). HRI '08. ACM, New York, NY, 49--56. DOI= http://doi.acm.org/10.1145/1349822.1349830
17. Yamaoka, F., Kanda, T., Ishiguro, H., and Hagita, N. 2008. How close?: model of proximity control for information-presenting robots. In Proceedings of the 3rd ACM/IEEE International Conference on Human Robot Interaction (Amsterdam, The Netherlands, March 12 - 15, 2008). HRI '08. ACM, New York, NY, 137--144. DOI= http://doi.acm.org/10.1145/1349822.1349841
18. Hoffman, G. and Breazeal, C. 2007. Effects of anticipatory action on human-robot teamwork efficiency, fluency, and perception of team. In Proc. of the ACM/IEEE International Conference on Human-Robot Interaction (Arlington, Virginia, USA, March 10-12, 2007). ACM, New York, NY, 1-8. DOI= http://doi.acm.org/10.1145/1228716.1228718
19. Gemperle, F., DiSalvo, C., Forlizzi, J., and Yonkers, W. 2003. The Hug: a new form for communication. In Proceedings of the 2003 Conference on Designing For User Experiences (San Francisco, California, June 06 - 07, 2003). DUX '03. ACM, New York, NY, 1--4. DOI= http://doi.acm.org/10.1145/997078.997103
20. Takayama, L., Ju, W., and Nass, C. 2008. Beyond dirty, dangerous and dull: what everyday people think robots should do. In Proceedings of the 3rd ACM/IEEE International Conference on Human Robot Interaction (Amsterdam, The Netherlands, March 12-15, 2008). HRI '08. ACM, New York, NY, 25--32. DOI= http://doi.acm.org/10.1145/1349822.1349827
21. Scholtz, J., Young, J., Drury, J., and Yanco, H. 2004. Evaluation of human-robot interaction awareness in search and rescue. In Proc. IEEE International Conference on Robotics and Automation.
22. Casper, J., and Murphy, R. 2003. Human-robot interactions during the robot-assisted urban search and rescue response at the World Trade Center. IEEE Transactions on Systems, Man and Cybernetics B, 33.
23. Mutlu, B. and Forlizzi, J. 2008. Robots in organizations: the role of workflow, social, and environmental factors in human-robot interaction. In Proceedings of the 3rd ACM/IEEE International Conference on Human Robot Interaction (Amsterdam, The Netherlands, March 12 - 15, 2008). HRI '08. ACM, New York, NY, 287--294. DOI= http://doi.acm.org/10.1145/1349822.1349860
24. Krishnan, H., Gibb, S., Steinfeld, A., and Shladover, S. E. 2001. Rear-end collision-warning system: Design and evaluation via simulation, Transportation Research Record - Journal of the Transportation Research Board 1759, 52--60.
25. Bainbridge, W., Hart, J., Kim, E. and Scassellati, B. 2008. The effect of presence on human-robot interaction. In IEEE International Symposium on Robot and Human Interactive Communication (Munich, Germany).
26. Tapus, A., Mataric, M. and Scassellati, B. 2007. The grand challenges in socially assistive robotics. IEEE Robotics and Automation Magazine. Vol. 4, No. 1. p. 35--427.
27. Jenkins, O. C., González, G., and Loper, M. M. 2007. Tracking human motion and actions for interactive robots. In Proceedings of the ACM/IEEE International Conference on Human-Robot Interaction (Arlington, Virginia, USA, March 10 - 12, 2007). HRI '07. ACM, New York, NY, 365--372. DOI= http://doi.acm.org/10.1145/1228716.1228765
28. Koenig, N., Chernova, S., Jones, C., Loper, M., and Jenkins, O. 2008. Hands-free interaction for human-robot teams. In ICRA 2008 Workshop on Social Interaction with Intelligent Indoor Robots (Pasadena, CA, USA, May 2008).
29. Koenig, N., Chernova, S., Jones, C., Loper, M., and Jenkins, O. 2008. Hands-free human-robot interaction. In HRI Caught on Film 2, Proc. of the 3rd ACM/IEEE International Conference on Human Robot Interaction. ACM, New York, NY, 383--388.
30. Gold, K. and Scassellati, B. 2006. Using context and sensory data to learn first and second person pronouns. In Proceedings of the 1st ACM SIGCHI/SIGART Conference on Human-Robot Interaction (Salt Lake City, Utah, USA, March 02 - 03, 2006). HRI '06. ACM, New York, NY, 110--117. DOI= http://doi.acm.org/10.1145/1121241.1121262
31. Gold K. and Scassellati, B. 2006. Grounded pronoun learning and pronoun reversal. In 5th International Conference on Development and Learning (ICDL-06, Bloomington, IN).
32. Gold, K., Doniec, M., and Scassellati, B. 2007. Learning grounded semantics with word trees: prepositions and pronouns. In Proceedings of the 6th IEEE International Conference on Development and Learning (ICDL, London, England, July 2007).
33. Gold, K. and Scassellati, B. 2007. A robot that uses existing vocabulary to infer non-visual word meanings from observation. In Proceedings of the Twenty-Second Annual Meeting of the Association for the Advancement of Artificial Intelligence (AAAI. Vancouver, BC, Canada, August, 2007).
34. MacWhinney, B. (2000). The CHILDES Project: Tools for Analyzing Talk, 3rd ed. Mahwah, NJ: Lawrence Erlbaum Associates.
35. Gold, K. (2008). Using Sentence Context and Implicit Contrast to Learn Sensor-Grounded Meanings for Relational and Deictic Words: The TWIG System. Yale University, May 2008.

Index Terms

• Human-centered computing

  • Human computer interaction (HCI)

    • HCI design and evaluation methods

    • HCI theory, concepts and models

  • Interaction design

    • Interaction design theory, concepts and paradigms