Was it worth the hassle?: ten years of mobile HCI research discussions on lab and field evaluations

ABSTRACT

Evaluation is considered one of the major cornerstones of human-computer interaction (HCI). During the last decade, several studies have discussed pros and cons of lab and field evaluations. Based on these discussions, we conduct a review to explore the past decade of mobile HCI research on field and lab evaluation, investigating responses in the literature to the "is it worth the hassle?" paper from 2004. We find that while our knowledge and experience with both lab and field studies have grown considerably, there is still no definite answer to the lab versus field question. In response we suggest that the real question is not if -- but when and how -- to go into the field. In response we suggest moving beyond usability evaluations, and to engage with field studies that are truly in-the-wild, and longitudinal.

References

1. Abdulrazak, B. and Malik, Y. Review of Challenges, Requirements, and Approaches of Pervasive Computing System Evaluation. IETE Technical Review 29, 6 (2012), 506--522.
2. Alsos, O. A. and Dabelow, B. A comparative evaluation study of basic interaction techniques for PDAs in pointof-care situations. Proc. P-Health'10, IEEE (2010), 1--8.
3. Axup, J. Building a Path For Future Communities. In Handbook of Research on Socio-Technical Design, (2008), 3--20.
4. Baillie, L. and Schatz, R. Exploring Multimodality in the Laboratory and the Field. Proc. CMI'05, ACM (2005), 100--107.
5. Barnard, L., Y, J. S., Jacko, J. A. and Sears, A. An empirical comparison of use-in-motion evaluation scenarios for mobile computing devices. IJHCS 62 (2005), 487--520.
6. Barnard, L., Yi, J.S., Jacko, J. and Sears, A. Capturing the effect of context on human performance in mobile computing. Pers Ubiquit Comput 11 (2007), 81--96.
7. Billi, M., Burzagli, L., Catarci, T., Santucci, G., Bertini, E., Gabbanini, F. and Palchetti, E. Unified methodology for evaluation of accessibility and usability of mobile applications. Univ. Access Inf. Soc., 9 (2010), 337--356.
8. Brown, B., Reeves, S., and Sherwood, S. Into the Wild: Challenges and Opportunities for Field Trial Methods. Proc. CHI'11, ACM (2011), 1657--1666.
9. Burghardt, D. and Wirth, K. Comparison of Evaluation Methods for Field-Based Usability Studies of Mobile Map Applications. Proc. International Cartographic Conference (2011).
10. Carter, S. Techniques and tools for field-based early-stage study and iteration of ubicomp applications: A dissertation proposal. University of California, 2005.
11. Carter, S., Mankoff, J., Klemmer, S. R. and Matthews, T. Exiting the Cleanroom: On Ecological Validity and Ubiquitous Computing. Human-Computer Interaction 23, 1, (2008), 47--99.
12. Crabtree, A., Chamberlain, A., Grinter, R. E., Jones, M., Rodden, T. and Rogers, Y. Introduction to the Special Issue of "The Turn to The Wild". TOCHI 20, 3 (2013).
13. Dahl, Y., Alsos, O. A. and Svanæs, D. Evaluating Mobile Usability: The Role of Fidelity in Full-Scale Laboratory Simulations with Mobile ICT for Hospitals, Proc. HCII'09, Springer (2009), 232--241.
14. Dahl, Y. Seeking a Theoretical Foundation for Design of In Sitro Usability Assessments. Proc. NordiCHI'10, ACM (2010), 623--626.
15. Davies, N., Cheverst, K., Dix, A. and Hesse, A. Understanding the Role of Image Recognition in Mobile Tour Guides. Proc. Mobile HCI'05, ACM (2005), 191--198.
16. Dearman, D., Hawkey, K. and Inkpen, K.M. Rendezvousing with location-aware devices. IwC 17 (2005), 524--566.
17. Duh, H. B., Tan, G. and Chen, V. H. Usability Evaluation for Mobile Devices: A Comparison of Laboratory and Field Tests. Proc. Mobile HCI'06, ACM (2006), 181--186.
18. Fiotakis, G., Raptis, D. and Avouris, N. Considering Cost in Usability Evaluation of Mobile Applications: Who, Where and When. Proc. Interact'09, Springer (2009), 231--234.
19. Gelderblom, H., Bruin, J. and Singh, A. Three Methods for Evaluating Mobile Phone Applications Aimed Users in a Developing Environment: AComparative Case Study. Proc. M4D'12 (2012), 321--334.
20. Hagen, P., Robertson, T., Kan, M. and Sadler, K. Emerging research methods for understanding mobile technology use. Proc. OzCHI'05, CHISIG (2005), 1--10.
21. Holone, H., Mislund, G., Tolsby, H. and Kristoffersen, S. Aspects of personal navigation with collaborative feedback. Proc. NordiCHI'08, ACM (2008), 182--191.
22. Howell, M., Love, S. and Turner, M. The impact of interface metaphor and context of use on the usability of a speech-based mobile city guide service. Behaviour & Information Technology 24, 1 (2005): 67--78.
23. Holzinger, A., Schlögl, M., Peischl, B. and Debevc, M. Optimization of a handwriting recognition algorithm for a mobile enterprise health information system on the basis of real-life usability research. Proc. ICETE'10, Springer (2010), 97--111.
24. Høegh, R. T., Kjeldskov, J., Skov, M. B. and Stage J. Setting Up A Field Laboratory for Evaluating In Situ. In Handbook of Research on User Interface Design and Evaluation for Mobile Technology, ISR, 2008.
25. Iachello, G. and Terrenghi, L. Mobile HCI 2004: Experience and Reflection. Pervasive Computing, JanMar (2005), 88--91.
26. Jambon, F., Golanski, C. and Pommier, P. J. Meta-evaluation of a context-aware mobile device usability. Proc. UBICOMM, IEEE (2007), pp. 21--26.
27. Jambon, F. and Meillon, B. User Experience in the Wild. Proc. CHI'09 EA, ACM (2009), 4069--4074.
28. Johnson, P. Usability and Mobility; Interactions on the move. Proc. Mobile HCI'98, GIST Technical Report G98-1 (1998)
29. Jumisko-Pyykkö, S. and Utriainen, T. (2011) A Hybrid Method for Quality Evaluation in the Context of Use for Mobile (3D) Television. Multimedia Tools and Applications, 55(2): 185--225.
30. Kaikkonen, A., Kekäläinen, A., Cankar, M., Kallio, T. and Kankainen, A. Usability Testing of Mobile Applications: A Comparison between Laboratory and Field Testing. Journal of Usability Studies 1, 1 (2005), 4--16.
31. Kaikkonen, A., Kekäläinen, A., Cankar, M., Kallio, T., and Kankainen, A. Will laboratory test results be valid in mobile contexts? In Handbook of Research on User Interface Design and Evaluation for Mobile Technology, ISR, 2008.
32. Kalnikaite, V., Bird, J. and Rogers, Y. Decision-making in the aisles: informing, overwhelming or nudging supermarket shoppers' Pers Ubiquit Comput 17 (2013), 1247--1259.
33. Kellar, M., Inkpen, K., Dearman, D., et al. Evaluation of Mobile Collaboration: Learning from our Mistakes. Technical Report 2004-13, Dalhousie University, 2004.
34. Khanum, M. A. and Trivedi, M. C. Comparison of Testing Environments with Children for Usability Problem Identification. International Journal of Engineering and Technology 5, 3 (2013), 2048--2053.
35. Kjeldskov J. and Graham C. A Review of Mobile HCI Research Methods. Proc. Mobile HCI'03, Springer (2003), 317--335.
36. Kjeldskov, J., Skov, M. B., Als, B. S. and Høegh, R. T. Is it Worth the Hassle? Exploring the Added Value of Evaluating the Usability of Context-Aware Mobile Systems in the Field. Proc. Mobile HCI'04, Springer (2004), 61--73.
37. Kjeldskov, J., Graham, C., Pedell, S., Vetere, F., Howard, S., Balbo, S. and Davies, J. Evaluating the usability of a mobile guide: The influence of location, participants and resources. Behaviour & Information Technology 24, 1 (2005), 51--65.
38. Kjeldskov, J. and Stage, J. Exploring 'Canned Communication' for coordinating distributed mobile work activities. IwC 18 (2006) 1310--1335.
39. Kjeldskov, J. and Paay, J. A Longitudinal Review of Mobile HCI Research Methods. Proc. Mobile HCI'12, ACM (2012), 69--78.
40. Kondratova, I., Lumsden, J. and Langton, N. Multimodal Field Data Entry: Performance and Usability Issues. Proc. International Conference on Computing and Decision Making NRC-CNRC (2006).
41. Korn, M. and Bødker, S. Looking ahead: how field trials can work in iterative and exploratory design of ubicomp systems. Proc. UbiComp'12, ACM (2012), 21--30.
42. Kray, C., Olivier, P., Guo, A. W., Singh, P., Ha, H. N. and Blythe, P. Taming Context: A Key Challenge in Evaluating the Usability of Ubiquitous Systems. Proc. USE'07 Workshop at Ubicomp'07 (2007).
43. Larsen, J. E., Petersen, M. K., Handler, R. and Zandi, N. Observing the Context of Use of a Media Player on Mobile Phones using Embedded and Virtual Sensors. Proc. NordiCHI'10, ACM (2010), 33--36.
44. Leitner, G., Ahlström, D. and Hitz, M. Usability of Mobile Computing in Emergency Response Systems - Lessons Learned and Future Directions. Proc. USAB'07. Springer (2007), 241--254.
45. Lumsden, J., Kondratova, I. and Durling, S. Investigating microphone efficacy for mobile speech-based data entry. Proc. HCI'07, Springer (2007), 89--97.
46. Lumsden, J. and MacLean, R. A Comparison of PseudoPaper and Paper Prototyping Methods for Mobile Evaluations. Proc. MONET'08 (2008), 538--457.
47. Lumsden, J., Langton, N., and Kondratova, I. Bringing the High Seas into the Lab to Evaluate Speech Input Feasibility: A Case Study. Proc. SiMPE Workshop at Mobile HCI'10 (2010).
48. Maly, I., Mikovec, Z., Vystrcil, J., Franc, J. and Slavik, P. An evaluation tool for research of user behavior in a realistic mobile environment. Pers Ubiquit Comput 17 (2013), 3--14.
49. Morrison, A., McMillan, D., Reeves, S., Sherwood, S., and Chalmers, M. A Hybrid Mass Participation Approach to Mobile Software Trials. Proc. CHI'12, ACM (2012), 1311--1320.
50. Nielsen, C. M., Overgaard, M., Pedersen, M. B., Stage, J. and Stenild, S. It's Worth the Hassle! The Added Value of Evaluating the Usability of Mobile Systems in the Field. Proc. NordiCHI'06, ACM (2006), 272--280.
51. Oulasvirta, A., Tamminen, S., Roto, V. and Kuorelahti. Interaction in 4-second Bursts: The Fragmented Nature of Attentional Resources in Mobile HCI. Proc. CHI'05, ACM (2005), 919--928.
52. Oulasvirta, A. and Nyyssönen, T. Flexible Hardware Configurations for Studying Mobile Usability. Journal of Usability Studies 4, 2 (2009), 93--105.
53. Oulasvirta, A. Rethinking Experimental Designs for Field Evaluations. Pervasive Computing, Oct-Dec (2012), 60--67.
54. Rogers, Y., Connelly, K., Tedesco, L., Hazlewood, W., Kurtz, A., Hall, R. E., Hursey, J., and Toscos, T. Why It's Worth the Hassle: The Value of In-Situ Studies When Designing Ubicomp. Proc. UbiComp'07, Springer (2007), 336--353.
55. Roto, V., Väätäjä, H., Jumisko-Pyykkö, S., and Väänänen-Vainio-Mattila, K. Best Practices for Capturing Context in User Experience Studies in the Wild. Proc. MindTrek'11 (2011), 91--98.
56. Skattør, B. Training and Deployment as a basis for Usability Engineering of Mobile Systems. Proc. ACHI, IEEE (2008), 277--284.
57. Streefkerk, J. W., van Esch-Bussemakers, M. P. and Neerincx, M. A. Field Evaluation of a Mobile Location-Based Notification System for Police Officers. Proc. Mobile HCI'08, ACM (2008), 101--108.
58. Sun, X. and May, A. A Comparison of Field-Based and Lab-Based Experiments to Evaluate User Experience of Personalised Mobile Devices. Adv. in Hum.-Comp. Int., Hindawi (2013), Article 2.
59. Tolmie, P., Crabtree, A., Egglestone, S., Humble, J., Greenhalgh, C. and Rodden, T. Digital Plumbing: the mundane work of deploying UbiComp in the home. Pers Ubiquit Comput 14 (2010), 181--196.
60. Vastenburg, M. H., Keyson, D. V. and de Ridder, H. Measuring User Experiences of Prototypical Autonomous Products in a Simulated Home Environment. Proc. HCII'07, Springer (2007), 998--1007.
61. Wilfinger, D., Pirker, M., Bernhaupt, R. and Tscheligi, M. Evaluating and Investigating an iTV Concept in the Field. Proc. EuroITV'09, ACM (2009), 175--178.
62. Wilson, M. L, Russel, A., Smith, D. A. and schraefel, m. c. mSpace Mobile: Exploring Support for Mobile Tasks. Proc. HCI'06, Springer (2006), 193--202.
63. Wilson, S., Galliers, J. and Fone, J. (2007) Cognitive Artifacts in Support of Medical Shift Handover: An in Use, in Situ Evaluation. IJHCS 22, 1&2 (2007), 59--80
64. Wilson, M. L., Mackay, W., Chi, E., Berstein, M., Russell, D. and Thimbleby, H. RepliCHI - CHI should be replicating and validating results more: discuss. Proc. CHI'11 EA, ACM (2011), 463--466.

Index Terms

• Human-centered computing