Short-term methodology for long-term usability

ABSTRACT

Approaches to understanding usability of computer interfaces over the long term typically rely on longitudinal studies, which are limited in scope to the period of the experiment. In this study, we explore whether a non-longitudinal, cross-sectional approach can reliably detect useful differences in usability between novices and experts. Our approach takes a "snapshot" of usability problems and behaviors across a heterogeneous sample of users, ranging from novice to expert. Our analysis suggests that a cross-sectional methodology can distinguish between less experienced and more experienced users with respect to the kinds of applications that cause frustration, frequency of use of help, and whether the problem was solved. Our analysis also suggests that the method is poor at distinguishing causes of frustration and the overall distribution of types of solutions tried. The data also suggest that three months of use of an application is the most useful point at which to distinguish less-experienced from more-experienced users.

References

1. Jacobsson, M. and Nylander, S. (2012) Always-On + Adoption -- a method for longitudinal studies. CHI 2012 Workshop on Theories, Methods and Case Studies of Longitudinal HCI Research, May 5, 2012, Austin, TX.
2. Jain, J., Rosenbaum, S., and Courage, C. (2010). Best practices in longitudinal research, Proceedings of the 28th International Conference Extended Abstracts on Human Factors in Computing Systems (CHI '10), April 10--15, 2010, Atlanta, GA, 4791--4794.
3. Karapanos, E., Hassenzahl, M., and Martens, J.-B. (2008). User experience over time: An initial framework. Proceedings of the 26th International Conference on Human Factors in Computing Systems (CHI '08), Florence, Italy, 3561--3566.
4. Karapanos, E., Martens, J.-B., and Hassenzahl, M. (2012). CHI 2012 Workshop on Theories, Methods and Case Studies of Longitudinal HCI Research, May 5, 2012, Austin, TX, http://longitudinalusability.wikispaces.com/CHI2012Workshop.
5. Karapanos, E., Martens, J.-B., and Hassenzahl, M. (2012). On the retrospective assessment of users' experiences over time: Memory or actuality? In: CHI 2012 Workshop on Theories, Methods and Case Studies of Longitudinal HCI Research, May 5, 2012, Austin, TX.
6. Khanlarian, C., 2010, A Longitudinal Study of Web-Based Homework, Ph.D. Dissertation, University of North Carolina at Greensboro.
7. Mendoza, V., and Novick, D. (2005). Usability over time, Proceedings of SIGDOC 2005, Coventry, UK, September 21--23, 2005, 151--158.
8. Novick, D., Andrade, O., Bean, N., and Elizalde, E. (2008). Help-based tutorials, Proceedings of SIGDOC 2008, Lisbon, Portugal, September 22--25, 2008, 1--8.
9. Novick, D., Elizalde, E., and Bean, N. (2007). Toward a more accurate view of when and how people seek help with computer applications, Proceedings of SIGDOC 2007, El Paso, TX, October 22--24, 2007, 95--102.
10. Prümper, J., Zapf, D., Brodbeck, F.C., and Frese, M. (1992). Some surprising differences between novice and expert errors in computerized office work, Behaviour & Information Technology 11(6), 319--328.
11. Roto, V., and Kujala, S. (2012). Studying six months in two hours. In: CHI 2012 Workshop on Theories, Methods and Case Studies of Longitudinal HCI Research, May 5, 2012, Austin, TX.
12. Szostek, A., and Walo, K. (2012). Prospective Day Reconstruction method: a way to validate service design concepts. In: CHI 2012 Workshop on Theories, Methods and Case Studies of Longitudinal HCI Research, May 5, 2012, Austin, TX.
13. van Lumig, C.J.L. (2009). The influence of user expertise on the usability experience: Interfaces for different users at Vodafone call centers. Master's thesis, Eindhoven University of Technology.
14. Vaughan, M., and Courage, C. (2007). SIG: Capturing longitudinal usability: what really affects user performance over time?, CHI '07 extended abstracts on Human factors in computing systems, April 28-May 03, 2007, San Jose, CA, 2149--2152.
15. Vaughan, M., Courage, C., Rosenbaum, S., Jain, J., Hammontree, M., Beale R., and Welsh, D. (2008). Longitudinal usability data collection: art versus science?, CHI '08 extended abstracts on Human factors in computing systems, April 05--10, 2008, Florence, Italy, 2261--2264.
16. von Wilamowitz Moellendorff, M., Hassenzahl, M., and Platz, A. (2006). Dynamics of user experience: How the perceived quality of mobile phones changes over time. User Experience - Towards a unified view, Workshop, 4th Nordic Conference on Human-Computer Interaction, October 14 - 18, 2006, Oslo, Norway, 74--78.

Index Terms

• Human-centered computing

  • Human computer interaction (HCI)

    • HCI design and evaluation methods