skip to main content
Skip header Section
Quantifying the User Experience: Practical Statistics for User ResearchMarch 2012
Publisher:
  • Morgan Kaufmann Publishers Inc.
  • 340 Pine Street, Sixth Floor
  • San Francisco
  • CA
  • United States
ISBN:978-0-12-384968-7
Published:16 March 2012
Pages:
312
Skip Bibliometrics Section
Bibliometrics
Skip Abstract Section
Abstract

Youre being asked to quantify usability improvements with statistics. But even with a background in statistics, you are hesitant to statistically analyze the data, as you may be unsure about which statistical tests to use and have trouble defending the use of the small test sample sizes associated with usability studies. The book is about providing a practical guide on how to use statistics to solve common quantitative problems arising in user research. It addresses common questions you face every day such as: Is the current product more usable than our competition Can we be sure at least 70% of users can complete the task on the 1st attempt How long will it take users to purchase products on the website This book shows you which test to use, and how provide a foundation for both the statistical theory and best practices in applying them. The authors draw on decades of statistical literature from Human Factors, Industrial Engineering and Psychology, as well as their own published research to provide the best solutions. They provide both concrete solutions (excel formula, links to their own web-calculators) along with an engaging discussion about the statistical reasons for why the tests work, and how to effectively communicate the results. Provides practical guidance on solving usability testing problems with statistics for any project, including those using Six Sigma practices Show practitioners which test to use, why they work, best practices in application, along with easy-to-use excel formulas and web-calculators for analyzing data Recommends ways for practitioners to communicate results to stakeholders in plain English Table of Contents Dedication Acknowledgements About the Authors Chapter 1: Introduction and How to Use This Book Chapter 2: Quantifying User Research Chapter 3: How Precise Are Our Estimates Confidence Intervals Chapter 4: Did We Meet or Exceed Our Goal Chapter 5: Is There a Statistical Difference between Designs Chapter 6: What Sample Sizes Do We Need Part 1: Summative Studies Chapter 7: What Sample Sizes Do We Need Part 2: Formative Studies Chapter 8: Standard Usability Questionnaires Chapter 9: Six Enduring Controversies in Measurement and Statistics Chapter 10: Wrapping Up Appendix Index

Cited By

  1. ACM
    Zheng Z, Yan B and Shi J User Experience Analysis of Library Intelligent Book-Lending Cabinets Service Based on MOA Model༚Case Study of China Proceedings of the Eleventh International Symposium of Chinese CHI, (290-301)
  2. Federici S, Bifolchi G, Bracalenti M, Ansani A, Napoletti A, Lanzillotti R, Desolda G, Mele M, Borsci S, de Filippis M, Gaudino G, Amendola M, Cocco A, Doria A and Simonetti E eGLU-Box Mobile: A Smartphone App for Usability Testing by Italian Public Administration Webmasters Human-Computer Interaction, (64-79)
  3. ACM
    Evin I, Hämäläinen P and Guckelsberger C (2022). Cine-AI: Generating Video Game Cutscenes in the Style of Human Directors, Proceedings of the ACM on Human-Computer Interaction, 6:CHI PLAY, (1-23), Online publication date: 25-Oct-2022.
  4. ACM
    Blattgerste J, Behrends J and Pfeiffer T A Web-Based Analysis Toolkit for the System Usability Scale Proceedings of the 15th International Conference on PErvasive Technologies Related to Assistive Environments, (237-246)
  5. Barneche-Naya V and Hernández-Ibáñez L Assessement of Assisted Navigation in NUI Virtual Architectural Environments Learning and Collaboration Technologies. Novel Technological Environments, (14-23)
  6. ACM
    Schaffer O Development and Preliminary Validation of the Enjoyment Questionnaire and the Sources of Enjoyment Questionnaire CHI Conference on Human Factors in Computing Systems Extended Abstracts, (1-7)
  7. ACM
    Bourgos N, Chroni A, Kounalaki M, Papadopoulou E, Skarmitzos A, Triantafilli A, Zagota I, Sintoris C and Avouris N On evaluating online teaching and learning experience: A usability evaluation study of synchronous teaching platforms CHI Greece 2021: 1st International Conference of the ACM Greek SIGCHI Chapter, (1-7)
  8. ACM
    Assmann M, Zanatta A and De Marchi A Long-term user experience in software crowdsourcing platform Proceedings of the XX Brazilian Symposium on Human Factors in Computing Systems, (1-7)
  9. Norem S, Rice A, Erwin S, Bridges R, Oesch S and Weber B A Mathematical Framework for Evaluation of SOAR Tools with Limited Survey Data Computer Security. ESORICS 2021 International Workshops, (557-575)
  10. ACM
    Wang T, Noaki Y and Kuzuoka H Exploring How to Display Referential Action to Support Remote Group Discussion Proceedings of the Asian CHI Symposium 2021, (89-96)
  11. Gray T and De Vries M Empirical Evaluation of a New DEMO Modelling Tool that Facilitates Model Transformations Advances in Conceptual Modeling, (189-199)
  12. ACM
    Lusk C and Jones M Cake Cam Proceedings of the 21st International Conference on Human-Computer Interaction with Mobile Devices and Services, (1-9)
  13. Barneche-Naya V and Hernández-Ibáñez L UX Aspects of Kinect-Based Movement Schemes Inside Virtual Environments for Museum Installations Learning and Collaboration Technologies. Ubiquitous and Virtual Environments for Learning and Collaboration, (133-150)
  14. Maldar M Engaging end-users as UX designers Proceedings of the 14th International Conference on Global Software Engineering, (62-65)
  15. ACM
    Sciascio C, Strohmaier D, Errecalde M and Veas E (2019). Interactive Quality Analytics of User-generated Content, ACM Transactions on Interactive Intelligent Systems, 9:2-3, (1-42), Online publication date: 25-Apr-2019.
  16. Galea C and Porter C Accessible choral ensembles for visually impaired singers Proceedings of the 32nd International BCS Human Computer Interaction Conference, (1-10)
  17. ACM
    Brühlmann F, Vollenwyder B, Opwis K and Mekler E Measuring the “Why” of Interaction Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, (1-13)
  18. Cristea D, Pistol I and Gîfu D Connecting the Content of Books to the Web and the Real World Computational Linguistics and Intelligent Text Processing, (50-61)
  19. Tesoriero R, Fardoun H, Awada H and Raisinghani M (2018). Accessing Map Information Using NFC-Based User Interfaces for In-Situ Learning Environments, International Journal of Online Pedagogy and Course Design, 8:1, (13-28), Online publication date: 1-Jan-2018.
  20. ACM
    Sciascio C, Sabol V and Veas E (2017). Supporting Exploratory Search with a Visual User-Driven Approach, ACM Transactions on Interactive Intelligent Systems, 7:4, (1-35), Online publication date: 31-Dec-2018.
  21. Gomes D, dos Reis P, de Paiva A, Silva A, Braz G, de Arajo A and Gattass M (2017). Semi-automatic methodology for augmented panorama development in industrial outdoor environments, Advances in Engineering Software, 114:C, (282-294), Online publication date: 1-Dec-2017.
  22. Rummel B (2017). Predicting post-task user satisfaction with weibull analysis of task completion times, Journal of Usability Studies, 13:1, (5-16), Online publication date: 1-Nov-2017.
  23. ACM
    Ferracani A, Faustino M, Giannini G, Landucci L and Del Bimbo A Natural Experiences in Museums through Virtual Reality and Voice Commands Proceedings of the 25th ACM international conference on Multimedia, (1233-1234)
  24. ACM
    Khaet F and Alfimtsev A The extended model of goals, operators, methods and selection rules (GOMS) for gesture interfaces Proceedings of the 13th Central & Eastern European Software Engineering Conference in Russia, (1-9)
  25. ACM
    Huenerfauth M, Patel K and Berke L Design and Psychometric Evaluation of an American Sign Language Translation of the System Usability Scale Proceedings of the 19th International ACM SIGACCESS Conference on Computers and Accessibility, (175-184)
  26. ACM
    Schiavo G, Leonardi C, Pasolli M, Sarti S and Zancanaro M Weigh it and share it! Proceedings of the 19th International Conference on Human-Computer Interaction with Mobile Devices and Services, (1-12)
  27. ACM
    Seidenari L, Baecchi C, Uricchio T, Ferracani A, Bertini M and Bimbo A (2017). Deep Artwork Detection and Retrieval for Automatic Context-Aware Audio Guides, ACM Transactions on Multimedia Computing, Communications, and Applications, 13:3s, (1-21), Online publication date: 31-Aug-2017.
  28. Bond R, Mulvenna M, Wallace J, Finlay D, Boyd K and Patterson M UX-handle Proceedings of the 31st British Computer Society Human Computer Interaction Conference, (1-2)
  29. Frank J and Kapila V (2017). Mixed-reality learning environments, Computers & Education, 110:C, (88-104), Online publication date: 1-Jul-2017.
  30. ACM
    Wang C, Geelgoed E and Cesar P eTheatre Proceedings of the Fifth International Symposium of Chinese CHI, (1-10)
  31. Borrego G, Morán A and Palacio R Preliminary evaluation of a tag-based knowledge condensation tool in agile and distributed teams Proceedings of the 12th International Conference on Global Software Engineering, (51-55)
  32. ACM
    Gai W, Yang C, Bian Y, Shen C, Meng X, Wang L, Liu J, Dong M, Niu C and Lin C Supporting Easy Physical-to-Virtual Creation of Mobile VR Maze Games Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems, (5016-5028)
  33. Gennari R, Melonio A, Raccanello D, Brondino M, Dodero G, Pasini M and Torello S (2017). Children's emotions and quality of products in participatory game design, International Journal of Human-Computer Studies, 101:C, (45-61), Online publication date: 1-May-2017.
  34. ACM
    Nguyen T and Ricci F Dynamic elicitation of user preferences in a chat-based group recommender system Proceedings of the Symposium on Applied Computing, (1685-1692)
  35. ACM
    di Sciascio C, Strohmaier D, Errecalde M and Veas E WikiLyzer Proceedings of the 22nd International Conference on Intelligent User Interfaces, (377-388)
  36. Koutsabasis P (2017). Empirical Evaluations of Interactive Systems in Cultural Heritage, International Journal of Computational Methods in Heritage Science, 1:1, (100-122), Online publication date: 1-Jan-2017.
  37. Moretti G, Sprugnoli R, Menini S and Tonelli S (2016). ALCIDE, Knowledge-Based Systems, 111:C, (100-112), Online publication date: 1-Nov-2016.
  38. ACM
    Ruoti S, Andersen J, Hendershot T, Zappala D and Seamons K Private Webmail 2.0 Proceedings of the 29th Annual Symposium on User Interface Software and Technology, (461-472)
  39. Cutting E, Banchero M, Beitelshees A, Cimino J, Fiol G, Gurses A, Hoffman M, Jeng L, Kawamoto K, Kelemen M, Pincus H, Shuldiner A, Williams M, Pollin T and Overby C (2016). User-centered design of multi-gene sequencing panel reports for clinicians, Journal of Biomedical Informatics, 63:C, (1-10), Online publication date: 1-Oct-2016.
  40. ACM
    di Sciascio C, Sabol V and Veas E Rank As You Go Proceedings of the 21st International Conference on Intelligent User Interfaces, (118-129)
  41. ACM
    Brown M, Chinthammit W and Nixon P An Implementation of Tangible Interactive Mapping to Improve Adult Learning for Preparing for Bushfire Proceedings of the Annual Meeting of the Australian Special Interest Group for Computer Human Interaction, (545-548)
  42. ACM
    Miller C, Settle A and Lalor J Learning Object-Oriented Programming in Python Proceedings of the 16th Annual Conference on Information Technology Education, (59-64)
  43. ACM
    Elhart I, Scacchi F, Niforatos E and Langheinrich M ShadowTouch Proceedings of the 4th International Symposium on Pervasive Displays, (209-216)
  44. Pang C, Neustaedter C, Procyk J and Riecke B A comparison of visual and textual city portal designs on desktop and mobile interfaces Proceedings of the 41st Graphics Interface Conference, (211-218)
  45. ACM
    Doumanis I and Smith S Evaluating the Impact of Embodied Conversational Agents (ECAs) Attentional Behaviors on User Retention of Cultural Content in a Simulated Mobile Environment Proceedings of the 7th Workshop on Eye Gaze in Intelligent Human Machine Interaction: Eye-Gaze & Multimodality, (27-32)
  46. ACM
    Staheli D, Yu T, Crouser R, Damodaran S, Nam K, O'Gwynn D, McKenna S and Harrison L Visualization evaluation for cyber security Proceedings of the Eleventh Workshop on Visualization for Cyber Security, (49-56)
  47. ACM
    Alves R, Valente P and Nunes N The state of user experience evaluation practice Proceedings of the 8th Nordic Conference on Human-Computer Interaction: Fun, Fast, Foundational, (93-102)
  48. ACM
    Dodero G, Gennari R, Melonio A and Torello S Towards tangible gamified co-design at school Proceedings of the first ACM SIGCHI annual symposium on Computer-human interaction in play, (77-86)
  49. ACM
    Oh C, Park J and Suh B Gravity Proceedings of the 16th international conference on Human-computer interaction with mobile devices & services, (519-524)
  50. ACM
    Schild J, LaViola J and Masuch M Altering gameplay behavior using stereoscopic 3D vision-based video game design Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, (207-216)
  51. ACM
    Pilloni P, Mulas F, Piredda L and Carta S Assessing the User Experience Design as a Persuasive Methodology in a Real World Sport Application Proceedings of International Conference on Advances in Mobile Computing & Multimedia, (387-394)
  52. ACM
    Bias R, Kortum P, Sauro J and Gillan D (2013). Clothing the naked emperor, Interactions, 20:6, (72-77), Online publication date: 1-Nov-2013.
  53. ACM
    Borsci S, Macredie R, Barnett J, Martin J, Kuljis J and Young T (2013). Reviewing and Extending the Five-User Assumption, ACM Transactions on Computer-Human Interaction, 20:5, (1-23), Online publication date: 1-Nov-2013.
  54. Bartuskova A and Krejcar O Evaluation Framework for User Preference Research Implemented as Web Application Proceedings of the 5th International Conference on Computational Collective Intelligence. Technologies and Applications - Volume 8083, (537-548)
  55. ACM
    Lewis J, Utesch B and Maher D UMUX-LITE Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, (2099-2102)
  56. ACM
    Bertolus C, Feuillet T and Bailleul D Analyse de l'usage fonctionnel des services audiovisuels des foyers disposant de l'IPTV Proceedings of the 2012 Conference on Ergonomie et Interaction homme-machine, (161-168)
  57. ACM
    Courtemanche F, Dufresne A and Aïmeur E Un Outil d'Évaluation de l'Interaction intégrant l'Oculométrie et les Signaux Physiologiques Proceedings of the 2012 Conference on Ergonomie et Interaction homme-machine, (33-40)
Contributors
  • Oracle Corporation
  • International Business Machines

Index Terms

  1. Quantifying the User Experience: Practical Statistics for User Research

    Recommendations

    Reviews

    Egon L. van den Broek

    Before I started to read this book, I searched for the authors' definition of user experience (UX). To my surprise, I did not find one. Of course, I may have missed it; however, if it is really missing, then this is definitely a weak aspect of the book. The reason I started with this search is because many people have issues with UX, usability, and related concepts. These terms are often used without scoping or defining them, which diminishes the value of work conducted in this field. For the purposes of this review, I will adopt the International Organization for Standardization (ISO) 9241-210 definition of UX: "A person's perceptions and responses that result from the use or anticipated use of a product, system, or service" (http://www.iso.org/iso/catalogue_detail.htm__?__csnumber=52075). This comes quite close to the book's definition of user research: "the systematic study of the goals, needs, and capabilities of users so as to specify design, construction, or improvement of tools to benefit how users work and live" (p. 9). However, although related, these definitions do differ. The definition of user research includes the definition of UX. This makes for an odd start to the book. Fortunately, some resources available online provide more information on the ins and outs of UX [1,2,3]. These sources can help readers get a solid footing for reading this book. Many books have already been published on UX, some even by the same publisher [4,5,6]. Where Buxton's book [4] emphasizes the qualitative aspects of UX, Tullis and Albert [5] and Goodman et al. [6] focus on quantitative aspects of UX, as this book does. However, they focus on data acquisition and not on data analysis, which is the prime focus here. This is what makes this book valuable for both students and practitioners. Both these categories of readers can also benefit from a complementary book by the same authors [7]. This book provides more hands-on computing practice, examples, and exercises to give readers a jump-start on quantifying UX. The book takes a pragmatic and realistic-hence, the best possible-approach. In chapter 1, the authors provide four decision trees that help the practitioner or student choose the right chapter to start with. It even provides pointers up to section level for the methods denoted in the decision trees, so there are no more excuses for making a mistake when choosing your quantitative analysis design. Having said this, it should be noted that the book provides an overview of rather basic statistical analysis and does not touch advanced statistical analysis (for example, mixed models and structural equation models). As such, the book provides room for follow-up books to continue where it leaves off. Without going into too much depth, I can say that the authors provide the basic mathematics underlying their analysis. For those that lack the fundamentals of statistics, the authors provide an appendix with a nutshell overview. Reading this book requires high-school-level mathematics, nothing more, which makes it accessible to a very wide audience. On the other hand, those already in the field can use it as a refresher. Moreover, each chapter takes a historical stance, providing sufficient explanations and a decent list of references. I will admit that I had a false start with this book. I noticed its weak spot-perhaps its only one-from the beginning. However, taken as a whole, it provides a pragmatic approach to quantifying UX, without oversimplifying or claiming too much. It delivers what it promises. This book is valuable for both practitioners and students, in virtually any discipline. It can help psychologists transfer their statistical knowledge to UX practice, practitioners quickly assess their envisioned design and analysis, engineers demystify UX, and students appreciate UX's merits. Online Computing Reviews Service

    Access critical reviews of Computing literature here

    Become a reviewer for Computing Reviews.