Asbjørn Følstad
ABSTRACT
In usability testing, usability problems are often found for only one test participant. The literature does not help in deciding whether such single-user problems should be accepted or rejected as usability problems. To help us understand how such decisions are made in practical usability testing, 89 practitioners described how they dealt with single-user problems in their latest usability test. Single-user problems was accepted, rejected, or reported as outliers. This decision depended on problem severity, participant profile, sample size, and judgments on whether the problem is an artifact of the test situation.
- Dumas, J. S., Fox, J. E. Usability testing: Current practices and future directions. In A. Sears, J. Jacko (eds.) The Human-Computer Interaction Handbook, 2nd edition, Lawrence Erlbaum Associates (2008), 1129--1150.Google Scholar
- Dumas, J. S., Redish, J. A practical guide to usability testing. Ablex Publishing Corporation, 1993. Google ScholarDigital Library
- Ezzy, D. Qualitative Analysis: Practice and Innovation. Routledge, 2002.Google Scholar
- Følstad, A., Law, E.L-C., Hornbæk, K. Analysis in Practical Usability Evaluation: A Survey Study. In Proc. CHI '12, ACM Press (2012), 2127--2136. Google ScholarDigital Library
- Hartson, H. R., Andre, T. S., Williges, R. C. Criteria for Evaluating Usability Evaluation Methods. International Journal of Human-Computer Interaction 13, 4 (2001), 373--410.Google ScholarCross Ref
- Hornbæk, K. Usability Evaluation as Idea Generation. In E.L-C. Law, E. T. Hvannberg, G. Cockton (Eds.) Maturing Usability: Quality in Software, Interaction and Value, Springer (2008), 267--286.Google Scholar
- Hornbæk, K. Dogmas in the assessment of usability evaluation methods. Behaviour & Information Technology 29, 1 (2010), 97--111. Google ScholarDigital Library
- Kjeldskov J., Skov M. B., Stage J. Instant Data Analysis: Evaluating Usability in a Day. In Proc. NordiCHI '04, ACM Press (2004), 233--240. Google ScholarDigital Library
- Law, E.L.-C. Hvannberg, E. T. Analysis of Combinatorial User Effect in International Usability Tests. In Proc. CHI '04, ACM Press (2004), 9--16. Google ScholarDigital Library
- Nielsen, J. Why You Only Need to Test with 5 Users, Jakob Nielsen's Alertbox, March 19, 2000, http://www.useit.com/alertbox/20000319.htmlGoogle Scholar
- Nielsen, J., Landauer, T. K. A mathematical model of the finding of usability problems. In Proc. CHI '93, ACM Press (1993), 206--213. Google ScholarDigital Library
- Randolph, J. J. Online Kappa Calculator. http://justus.randolph.name/kappaGoogle Scholar
- Rubin, J. and Chisnell, D. Handbook of usability testing (2nd.edition). Wiley Publishing, 2008.Google Scholar
- Sauro, J, Lewis, J. R. Estimating completion rates from small samples using binomial confidence intervals: Comparisons and recommendations. In Proc. HFES, (2005), 2100--2104.Google ScholarCross Ref
- Sauro, J. Confidence Interval Calculator for a Completion Rate. http://www.measuringusability.com/waldGoogle Scholar
- Turner, C. W., Lewis, J. R., Nielsen, J. Determining usability test sample size. In W. Karwowski (ed.) International Encyclopedia of Ergonomics and Human Factors, 2nd Edition, CRC Press (2006), 3084--3088.Google Scholar
- Woolrych, A., Cockton, G. Why and when five test users aren't enough. In Proc. IHM-HCI 2001, Cépadèus Éditions (2001), 105--108.Google Scholar
Index Terms
- Outliers in usability testing: how to treat usability problems found for only one test participant?
Recommendations
Usability testing: what have we overlooked?
CHI '07: Proceedings of the SIGCHI Conference on Human Factors in Computing SystemsFor more than a decade, the number of usability test participants has been a major theme of debate among usability practitioners and researchers keen to improve usability test performance. This paper provides evidence suggesting that the focus be ...
Group usability testing: evolution in usability techniques
Usability testing has a long history. In its early form, it was conducted with many individual participants much like traditional research experiments. With the advent of discount usability engineering techniques, fewer participants were required (5-7 ...
Low-cost usability testing
Software quality and software testing in internet timesUsability is an important aspect of software products. However, in practice not much attention is given to this issue during testing. Testers often do not have the knowledge, instruments and/or time available to test for usability. This article ...
Comments