UX_Mate: from facial expressions to UX evaluation

ABSTRACT

In this paper we propose and evaluate UX_Mate, a non-invasive system for the automatic assessment of User eXperience (UX). In addition, we contribute a novel database of annotated and synchronized videos of interactive behavior and facial expressions. UX_Mate is a modular system which tracks facial expressions of users, interprets them based on pre-set rules, and generates predictions about the occurrence of a target emotional state, which can be linked to interaction events. The system simplifies UX evaluation providing an indication of event occurrence. UX_Mate has several advantages compared to other state of the art systems: easy deployment in the user's natural environment, avoidance of invasive devices, and extreme cost reduction. The paper reports a pilot and a validation study on a total of 46 users, where UX_Mate was used for identifying interaction difficulties. The studies show encouraging results that open possibilities for automatic real-time UX evaluation in ecological environments.

References

1. Ariely, D. and Cannon, Z. Gestalt characteristics of experiences: the defining feature of summarized events. Journal of Behavioural Decision Making (2000), 13 (2), 191--201.
2. Benedek, J., and Miner, T. Measuring desirability: New methods for evaluating desirability in a usability lab setting. UPA (2002).
3. Bradley, M., and Lang, P. J. Measuring emotion: The self-assessment semantic differential manikin and the semantic differential. Science 25, I (1994), 49--59.
4. Branco, P., Firth, P., Encarnação, L. M., and Bonato, P. Faces of emotion in human-computer interaction. In CHI EA (2005), 1236--1239.
5. Burr, B. Vaca: a tool for qualitative video analysis. In CHI EA '06 (2006), 622--627.
6. Chorianopoulos, K., and Spinellis, D. User interface evaluation of interactive tv: a media studies perspective. Univers. Access Inf. Soc. 5 (2006), 209--218.
7. Cohen, I., Sebe, N., Chen, L., Garg, A., and Huang, T. S. Facial expression recognition from video sequences: Temporal and static modelling. In Comput. Vis. Imag. Und. (2003), 160--187.
8. Dacher, K. Signs of appeasement: Evidence for the distinct displays of embarrassment, amusement, and shame. Pers. Soc. Psychol. 68, 3 (1995), 441--454.
9. Davidson, R. J., Ekman, P., Saron, C. D., Senulis, J. A., and Friesen, W. V. Approach-withdrawal and cerebral asymmetry: Emotional expression and brain physiology. Pers. Soc. Psychol. 58, 2 (1990), 330--341.
10. Den Uyl, M., Van Kuilenburg, H., and Lebert, E. Facereader: an online facial expression recognition system. In Measuring Behavior (2005).
11. Derbaix, C. M. The impact of affective reactions on attitudes toward the advertisement and the brand: A step toward ecological validity. J. Marketing Res. 32, 4 (1995), 470--479.
12. Desmet, P. Measuring emotion: development and application of an instrument to measure emotional responses to products. Kluwer Academic Publishers (2004), 111--123.
13. Ekman, P. Telling Lies: Clues to Deceit in the Marketplace, Politics, and Marriage. W. W. Norton & Company, (1992).
14. Ekman, P., and Friesen, W. Facial Action Coding System: A Technique for the Measurement of Facial Movement. Consulting Psychologists Press, (1978).
15. Grootjen, M., Neerincx, M. A., van Weert, J. C. M., and Truong, K. P. Measuring cognitive task load on a naval ship: implications of a real world environment. In Proc. FAC (2007), 147--156.
16. Hall, M., Frank, E., Holmes, G., Pfahringer, B., Reutemann, P., and Witten, I. H. The weka data mining software: an update. SIGKDD Explor. Newsl. 11, (2009), 10--18.
17. Hassenzahl, M., and Tractinsky, N. User experience - a research agenda. Behav. Inf. Technol. 25, 2 (2006), 91--97.
18. Hazlett, R. L., and Benedek, J. Measuring emotional valence to understand the user's experience of software. Int. J. Hum.-Comput. Stud. 65 (2007), 306--314.
19. Isbister, K., Hk, K., Laaksolahti, J., and Sharp, M. The sensual evaluation instrument: Developing a trans-cultural self-report measure of affect. Int. J. Hum.-Comput. Stud. 65, 4 (2007), 315--328.
20. Kahneman, D., Fredrickson, B. L., Schreiber, C. A. and Redelmeier, D. A. When more pain is preferred to less: Adding a better end. Psychological Science, 4 (1993), 401--405.
21. Kuilenburg, H. V., Wiering, M., and Uyl, M. D. A model based method for automatic facial expression recognition. In Proc. ECML (2005), 194--205.
22. Lavie, T. and Tractinsky, N.: Assessing Dimensions of Perceived Visual Aesthetics of Web Sites. Int. J. Hum.-Comput. Stud. 60, 3 (2004), 269--298.
23. Lopatovska, I., and Arapakis, I. Theories, methods and current research on emotions in library and information science, information retrieval and human-computer interaction. Inf. Process. Manage. 47 (2011), 575--592.
24. Mahlke, S., Minge, M., and Thüring, M. Measuring multiple components of emotions in interactive contexts. In CHI EA '06 (2006), 1061--1066.
25. Mcdaniel, B., D'Mello, S., King, B., Chipman, P., Tapp, K., and Graesser, A. Facial Features for Affective State Detection in Learning Environments. In Proc. 29th Annual Meeting of the Cognitive Science Society (2007).
26. Mehrabian, A. Pleasure-arousal-dominance: A general framework for describing and measuring individual differences in temperament. Curr. Psychol. 14, 4 (1996), 261--292.
27. Pantic, M., and Rothkrantz, L. Automatic Analysis of Facial Expressions: The State of the Art. IEEE T. Pattern Anal. 22, 12 (2000), 1424--1445.
28. Picard, R. W. Affective Computing for HCI. In Proc. HCI 1999, 1, 1 (1999), 829--833.
29. Ravaja, N., Turpeinen, M., Saari, T., Puttonen, S., and Keltikangas-Järvinen, L. The psychophysiology of James Bond: Phasic emotional responses to violent video game events. Emotion 8, 1 (2008), 114--120.
30. Ryan, A., Cohn, J., Lucey, S., Saragih, J., Lucey, P., De la Torre, F., and Rossi, A. Automated facial expression recognition system. In Proc. IEEE ICCST (2009), 172--177.
31. Scheirer, J., Fernandez, R., Klein, J., and Picard, R. W. Frustrating the user on purpose: A step toward building an affective computer. Interact. Comput. (2001) 93--118.
32. Sebe, N., Lew, M. S., Cohen, I., Sun, Y., Gevers, T., and Huang, T. S. Authentic facial expression analysis. In Proc. IEEE AFGR (2004), 517--522.
33. Swindells, C., Maclean, K. E., Booth, K. S., and Meitner, M. A case-study of affect measurement tools for physical user interface design. In Proc. GI (2006), 243--250.
34. Tractinsky, N. and Zmiri, D.: Exploring attributes of skins as potential antecedents of emotion in HCI. Aesthetic computing (2006), 405--422
35. Tullis, T. S. Screen design. Handbook of Human-Computer Interaction (1997).
36. Westerman, S. J., Sutherland, E. J., Robinson, L., Powell, H., and Tuck, G. A multi-method approach to the assessment of web page designs. In Proc. ACII (2007), 302--313.
37. Zaman, B., and Shrimpton-Smith, T. The facereader: measuring instant fun of use. In Proc., NordiCHI '06 (2006), 457--460.
38. Joho, H., Staiano, J., Sebe, N., and Jose, J. Looking at the Viewer: Exploiting Facial Activities to Detect Personal Highlights of Multimedia Contents. Int. J. Multimed. Tools App. 51, 2 (2011), 505--523.

Index Terms

• Human-centered computing

  • Human computer interaction (HCI)

    • HCI design and evaluation methods