What happened to remote usability testing?: an empirical study of three methods

ABSTRACT

The idea of conducting usability tests remotely emerged ten years ago. Since then, it has been studied empirically, and some software organizations employ remote methods. Yet there are still few comparisons involving more than one remote method. This paper presents results from a systematic empirical comparison of three methods for remote usability testing and a conventional laboratory-based think-aloud method. The three remote methods are a remote synchronous condition, where testing is conducted in real time but the test monitor is separated spatially from the test subjects, and two remote asynchronous conditions, where the test monitor and the test subjects are separated both spatially and temporally. The results show that the remote synchronous method is virtually equivalent to the conventional method. Thereby, it has the potential to conveniently involve broader user groups in usability testing and support new development approaches. The asynchronous methods are considerably more time-consuming for the test subjects and identify fewer usability problems, yet they may still be worthwhile.

References

1. Ames, M. Final Report on Remote Usability Studies.
2. Andreasen, M. S., Nielsen, H. V., Schræder, S. O. and Stage, J. Usability in open source software development: Opinions and practice. Information Technology and Control 35A, 3 (2006), 303--312.
3. Bartek, V. and Cheatham, D. Experience Remote Usability Testing, Part 1.
4. Bartek, V. and Cheatham, D. Experience Remote Usability Testing, Part 2.
5. Bartek, V. and Cheatham, D. Experiences in Remote Rsability Evaluations.
6. Benson, C., Muller-Prove, M. and Mzourek, J. Professional usability in open source projects: Gnome, openoffice.org, netbeans. Proceedings of CHI 2004, ACM Press (2004), 1083--1084.
7. Brush, A. B., Ames, M. and Davis, J. A comparison of synchronous remote and local usability studies for an expert interface. Proceedings of CHI 2004, ACM Press (2004), 1179--1182.
8. Castillo, J. C., Hartson, H. R. andHix, D. Remote usability evaluation: Can users report their own critical incidents? Proceedings of CHI 1998, ACM Press (1998), 253--254.
9. de Vreede, G.-J., Fruhling, A. and Chakrapani, A. A repeatable collaboration process for usability testing. Proceedings of HICSS 2005, IEEE Computer Society (2005), Track 1, p. 46.
10. Dempsey, B. J., Weiss, D., Jones, P. and Greenberg, J. Who is an open source software developer? Communications of the ACM 45, 2 (2002), 67--72.
11. Dray, S. and Siegel, D. Remote possibilities?: International usability testing at a distance. interactions 11, 2 (2004), 10--17.
12. Eklund, S., Feldman, M., Trombley, M. and Sinha, R. Improving the Usability of Open Source Software: Usability Testing of staroffice calc. http://www.sims.berkeley.edu/~sinha/opensource.html.
13. Frishberg, N., Dirks, A. M., Benson, C., Nickell, S. and Smith, S. Getting to know you: Open source development meets usability. Proceedings of CHI 2002, ACM Press (2002), 932--933.
14. Gough, D. and Phillips, H. Remote Online Usability Testing: Why, How, and When to Use it.
15. Hammontree, M., Weiler, P. and Nayak, N. Remote usability testing. Interactions 1, 3 (1994), 21--25.
16. Hartson, H. R. and Castillo, J. C. Remote evaluation for post-deployment usability improvement. Proceedings of AVI 1998, ACM Press (1998), 22--29.
17. Hartson, H. R., Castillo, J. C., Kelso, J. and Neale, W. C. Remote evaluation: The network as an extension of the usability laboratory. Proceedings of CHI 1996, ACM Press (1996), 228--235.
18. Hertzum, M. and Jacobsen, N. E. The evaluator effect: A chilling fact about usability evaluation methods. International Journal of Human-Computer Interaction 15, 1 (2003), 183--204.
19. Houck-Whitaker, J. Remote Testing versus Lab Testing. http://boltpeters.com/articles/versus.html.
20. Karat, C.-M., Campbell, R. and Fiegel, T. Comparison of empirical testing and walkthrough methods in user interface evaluation. Proceedings of CHI 1992, ACM Press (1992), 397--404.
21. Kjeldskov, J., Skov, M. B. and Stage, J. Does time heal: A longitudinal study of usability. Proceedings of OZCHI 2005, ACM Press (2005), 1--10.
22. Krauss, F. S. H. Methodology for remote usability activities: A case study. IBM Systems Journal 42, 4 (2003), 582--593.
23. McFadden, E., Hager, D. R., Elie, C. J. and Blackwell, J. M. Remote usability evaluation: Overview and case studies. International Journal of Human-Computer Interaction 14, 3&4 (2002), 489--502.
24. Moon, J. Y. and Sproull, L. Essence of Distributed Work: The Case of the linux Kernel. http://www.firstmonday.org/issues/issue511/moon/index.html.
25. Murphy J., Howard S., Kjeldskov K. and Goschnick, S. Location, location, location: Challenges of outsourced usability evaluation. Proceedings of the Workshop on Improving the Interplay between Usability Evaluation and User Interface Design, NordiCHI 2004, Aalborg University, Department of Computer Science, HCI-Lab Report no. 2004/2 (2004), 12--15.
26. Nichols, D. M. and Twidale, M. B. Usability and open source software. Technical Report 10/02, Department of Computer Science, University of Waikato, Working Paper Series ISSN 1170--487X, 2002.
27. Olmsted, E. and Gill, M. In-person usability study compared with self-administered web (remote-different time-place) study: Does mode of study produce similar results? Proceedings of UPA 2005, UPA (2005).
28. Petrie, H., Hamilton, F., King, N. and Pavan, P. Remote usability evaluation with disabled people. Proceedings of CHI 2006, ACM Press (2006), 1133--1141.
29. Raymond. E. The Revenge of the Hackers. O'Reilly and Associates, 1999.
30. Rubin, J. Handbook of Usability Testing. Wiley, 1994.
31. Safire, M. Remote moderated usability. http://www.upassoc.org/usability resources/conference/ 2004/im safire.html, 2004.
32. Scholtz, J. Adaption of traditional usability testing methods for remote testing. Proceedings of HICCS '01, IEEE (2001).
33. Skov, M. B. and Stage, J. Supporting problem identification in usability evaluations. Proceedings of OzCHI 2005, ACM Press (2005), 1--9.
34. Thompson, K. E., Rozanski, E. P. and Haake, A. R. Here, there, anywhere: Remote usability testing that works. Proceedings of CITC5 2004, ACM Press (2004), 132--137.
35. Winckler, M. A. A., Freitas, C. M. D. S. and de Lima, J. V. Usability remote evaluation for www. Proceedings of CHI 2000, ACM Press (2000), 131--132.

Index Terms

• Human-centered computing

  • Human computer interaction (HCI)

    • HCI theory, concepts and models

    • Interaction paradigms

      • Graphical user interfaces

  • Interaction design

    • Interaction design theory, concepts and paradigms

• Software and its engineering

  • Software creation and management

    • Software development techniques

      • Object oriented development

  • Software notations and tools

    • Development frameworks and environments

      • Integrated and visual development environments

      • Object oriented frameworks