Egon L. Broek
Skip Abstract Section
Abstract
Emotions are a crucial element for personal and ubiquitous computing. What to sense and how to sense it, however, remain a challenge. This study explores the rare combination of speech, electrocardiogram, and a revised Self-Assessment Mannequin to assess people's emotions. 40 people watched 30 International Affective Picture System pictures in either an office or a living-room environment. Additionally, their personality traits neuroticism and extroversion and demographic information (i.e., gender, nationality, and level of education) were recorded. The resulting data were analyzed using both basic emotion categories and the valence--arousal model, which enabled a comparison between both representations. The combination of heart rate variability and three speech measures (i.e., variability of the fundamental frequency of pitch (F0), intensity, and energy) explained 90% (p < .001) of the participants' experienced valence--arousal, with 88% for valence and 99% for arousal (ps < .001). The six basic emotions could also be discriminated (p < .001), although the explained variance was much lower: 18---20%. Environment (or context), the personality trait neuroticism, and gender proved to be useful when a nuanced assessment of people's emotions was needed. Taken together, this study provides a significant leap toward robust, generic, and ubiquitous emotion-aware computing.
- Alemdar H, Ersoy C (2010) Wireless sensor networks for healthcare: a survey. Comput Netw 54(15):2688-2710. Google Scholar
- Amft O, Lukowicz P (2009) From backpacks to smartphones: past, present, and future of wearable computers. IEEE Pervasive Comput 8(3):8-13. Google Scholar
- Appelhans BM, Luecken LJ (2006) Heart rate variability as an index of regulated emotional responding. Rev Gen Psychol 10(3):229-240.Google Scholar
- Arnrich B, Setz C, Marca RL, Troster G, Ehlert U (2010) What does your chair know about your stress level? IEEE Trans Inf Technol Biomed 14(2):207-214. Google Scholar
- Bailenson JN, Pontikakis ED, Mauss IB, Gross JJ, Jabon ME, Hutcherson CA, Nass C, John O (2008) Real-time classification of evoked emotions using facial feature tracking and physiological responses. Int J Hum Comput Stud 66(5):303-317. Google Scholar
- Barakova EI, Lourens T (2010) Expressing and interpreting emotional movements in social games with robots. Pers Ubiquitous Comput 14(5):457-467. Google Scholar
- Barrett LF, Kensinger EA (2010) Context is routinely encoded during emotion perception. Psychol Sci 21(4):595-599.Google Scholar
- Boersma P (1993) Accurate short-term analysis of the fundamental frequency and the harmonics-to- noise ratio of a sampled sound. In: Proceedings of the Institute of Phonetic Sciences, University of Amsterdam, vol 17, pp 97-110.Google Scholar
- Boersma PPG, Weenink DJM (2006) Praat 4.0.4. URL: http://www.praat.org {Last accessed on June 22, 2011}.Google Scholar
- Borders A, Liang CTH (2011) Rumination partially mediates the associations between perceived ethnic discrimination, emotional distress, and aggression. Cult Divers Ethn Minor Psychol 17(2): 125-133.Google Scholar
- Bose R (2009) Sensor networks motes, smart spaces, and beyond. IEEE Pervasive Comput 8(3):84-90. Google Scholar
- Bradley MM, Lang PJ (1994) Measuring emotion: the self-assessment manikin and the semantic differential. J Behav Ther Exp Psychol 25(1):49-59.Google Scholar
- Cacioppo JT, Berntson GG (1994) Relationship between attitudes and evaluative space: a critical review, with emphasis on the separability of positive and negative substrates. Psychol Bull 115(3):401-423.Google Scholar
- Cacioppo JT, Tassinary LG, Berntson GG (2007) Handbook of psychophysiology. 3rd edn. Cambridge University Press, NewYork.Google Scholar
- Carrera P, Oceja L (2007) Drawing mixed emotions: sequential or simultaneous experiences?. Cogn Emot 21(2):422-441.Google Scholar
- Casson AJ, Abd SJM, Smith DCY, Rodriguez-Villegas JSDE (2010) Wearable electroencephalography. IEEE Eng Med Biol Mag 29(3):44-56.Google Scholar
- Christie IC, Friedman BH (2004) Autonomic specificity of discrete emotion and dimensions of affective space: a multivariate approach. Int J Psychophysiol 51(2):143-153.Google Scholar
- Collet C, Vernet-Maury E, Delhomme G, Dittmar A (1997) Autonomic nervous system response patterns specificity to basic emotions. J Auton Nervous Syst 62(1-2):45-57.Google Scholar
- Cook DJ, Augusto JC, Jakkula VR (2009) Ambient intelligence: technologies, applications, and opportunities. Pervasive Mob Comput 5(4):277-298. Google Scholar
- Costa PT Jr, McCrae RR (1980) Influence of extraversion and neuroticism on subjective well-being: happy and unhappy people. J Personal Soc Psychol 38(4):668-678.Google Scholar
- Cowie R, Douglas-Cowie E, Tsapatsoulis N, Votsis G, Kollias S, Fellenz W, Taylor JG (2001) Emotion recognition in human-computer interaction. IEEE Signal Process Mag 18(1):32-80.Google Scholar
- Cutmore TRH, James DA (2007) Sensors and sensor systems for psychophysiological monitoring: a review of current trends. J Psychophysiol 21(1):51-71.Google Scholar
- Dalgleish T, Dunn BD, Mobbs D (2009) Affective neuroscience: past, present, and future. Emot Rev 1(4):355-368.Google Scholar
- Davidson RJ, Scherer KR, Hill Goldsmith H (2003) Handbook of affective sciences. Oxford University Press, New York.Google Scholar
- de Berg M, van Kreveld M, Overmars M (2008) Computational geometry: algorithms and applications. 3rd edn. Springer, Berlin. Google Scholar
- Decety J, Ickes W (2009) The social neuroscience of empathy. Social neuroscience series. A Bradford Book, The MIT Press, Cambridge.Google Scholar
- Eerola T, Vuoskoski JK (2011) A comparison of the discrete and dimensional models of emotion in music. Psychol Music 39(1):18-49.Google Scholar
- Ekman P, Levenson RW, Friesen WV (1983) Autonomic nervous system activity distinguishes between emotions. Science 221(4616):1208-1210.Google Scholar
- El Ayadi M, Kamel MS, Karray F (2011) Survey on speech emotion recognition: features, classification schemes, and databases. Patt Recognit 44(3):572-587. Google Scholar
- Eysenck HJ, Eysenck SBG (1991) Manual of the Eysenck personality scales (EPS adult). Hodder and Stoughton, London.Google Scholar
- Fagerberg P, Ståhl A, Höök K (2004) eMoto: emotionally engaging interaction. Pers Ubiquitous Comput 8(1):377-381. Google Scholar
- Fairclough SH (2009) Fundamentals of physiological computing. Interact Comput 21(1-2):133-145. Google Scholar
- Fontaine JRJ, Scherer KR, Roesch EB, Ellsworth P (2007) The world of emotion is not two-dimensional. Psychol Sci 18(12):1050-1057.Google Scholar
- Gershenfeld N, Krikorian R, Cohen D (2004) The Internet of things. Sci Am 291(4):76-81.Google Scholar
- Grossmann I, Ellsworth PC, Hong Y (2011) Culture, attention, and emotion. J Exp Psy: General. doi:10.1037/a0023817.Google Scholar
- Gunes H, Piccardi M (2009) Automatic temporal segment detection and affect recognition from face and body display. IEEE Trans Syst Man Cybern B Cybern 39(1):64-84. Google Scholar
- Hao Y, Foster R (2008) Wireless body sensor networks for health-monitoring applications. Physiol Meas 29(11):R27-R56.Google Scholar
- Izard CE et al (2010) Special section: on defining emotion. Emot Rev 2(4):363-385.Google Scholar
- Kamarck TW, Lovallo WR (2003) Cardiovascular reactivity to psychological challenge: conceptual and measurement considerations. Psychosom Med 65(1):9-21.Google Scholar
- Keltikangas JK, Järvinen L (2001) Intraindividual analysis of instantaneous heart rate variability. Psychophysiology 38(4): 659-668.Google Scholar
- Kim J (2007) Bimodal emotion recognition using speech and physiological changes, I-Tech Education and Publishing, Vienna, chap 15, pp 265-280.Google Scholar
- Kim J, André E (2006) Emotion recognition using physiological and speech signal in short-term observation. Lect Notes Comput Sci (Percept Interact Technol) 4021:53-64. Google Scholar
- Kim J, André E (2008) Emotion recognition based on physiological changes in music listening. IEEE Trans Pattern Anal Mach Intell 30(12):2067-2083. Google Scholar
- Kim J, André E, Rehm M, Vogt T, Wagner J (2005) Integrating information from speech and physiological signals to achieve emotional sensitivity. In: Proceedings of the 9th European conference on speech communication and technology, L2F-- Spoken Language Systems Laboratory, Lisboa, pp 809-812.Google Scholar
- Kim J, André E, Vogt T (2009) Towards user-independent classification of multimodal signals. In: Proceedings of the IEEE 3rd international conference on affective computing and intelligent interaction, ACII, IEEE Press, Amsterdam, vol 1, pp 722-728.Google Scholar
- Kleiger RE, Bigger JT, Bosner MS, Chung MK, Cook JR, Rolnitzky LM, Steinman R, Fleiss JL (1991) Stability over time of variables measuring heart rate variability in normal subjects. Am J Cardiol 68(6):626-630.Google Scholar
- Kleinsmith A, Bianchi-Berthouze N, Steed A (2011) Automatic recognition of non-acted affective postures. IEEE Trans Syst Man Cybern B Cybern 41(4):1027-1038. Google Scholar
- Kopetz H (2011) Real-time systems: design principles for distributed and embedded applications, 2nd edn. Real-Time Systems Series, Springer Science+Business Media, LCC, New York. Google Scholar
- Kring AM, Gordon AH (1998) Sex differences in emotion: expression, experience, and physiology. J Pers Soc Psychol 74(3):686-703.Google Scholar
- Krumm J (2010) Ubiquitous computing fundamentals. Chapman & Hall, CRC Press, Boca Raton. Google Scholar
- Landis C, Dewick HN (1929) The electrical phenomena of the skin (psychogalvanic reflex). Psychol Bull 26(2):64-119.Google Scholar
- Lang PJ, Bradley MM, Cuthbert BN (2008) International affective picture system (IAPS): affective ratings of pictures and instruction manual. Tech. Rep. A-8, University of Florida, Gainesville.Google Scholar
- Lemov RM (2005) World as laboratory: experiments with mice, mazes, and men. Hill and Wang, New York.Google Scholar
- Levenson RW, Ruef AM (1992) Empathy: a physiological substrate. J Pers Soc Psychol 63(2):234-246.Google Scholar
- Levenson RW, Carstensen LL, Friesen WV, Ekman P (1991) Emotion, physiology, and expression in old age. Psychol Aging 6(1):28-35.Google Scholar
- Levenson RW, Ekman P, Heider K, Friesen WV (1992) Emotion and the autonomous nervous system activity in the Minangkabau of West Sumatra. J Pers Soc Psychol 62(6):972-988.Google Scholar
- Lewis M, Haviland-Jones JM, Barrett LF (2008) Handbook of emotions. 3rd edn. The Guilford Press, New York.Google Scholar
- Liu X, Zheng Y, Phyu M, Zhao B, Je M, Yuan X (2011) Multiple functional ECG signal is processing for wearable applications of long-term cardiac monitoring. IEEE Trans Biomed Eng 58(2):380-389.Google Scholar
- Llabre MM, Spitzer SB, Saab PG, Ironson GH, Schneiderman N (1991) The reliability and specificity of delta versus residualized change as a measure of cardiovascular reactivity to behavioral challenges. Psychophysiology 28(6):701-711.Google Scholar
- Logothetis NK (2008) What we can do and what we cannot do with fMRI. Nature 453(7197):869-878.Google Scholar
- Lourens T, van Berkel R, Barakova E (2010) Communicating emotions and mental states to robots in a real time parallel framework using Laban movement analysis. Robot Auton Syst 58(12):1256-1265. Google Scholar
- Lucas RE, Baird BM (2004) Extraversion and emotional reactivity. J Pers Soc Psychol 86(3):473-485.Google Scholar
- Malik M, Camm AJ (1995) Heart rate variability. Futura Publishing Company, Inc., Armonk, NY, USA.Google Scholar
- Martínez JF, Familiar MS, Corredor I, García AB, Bravo S, López L (2011) Composition and deployment of e-Health services over Wireless Sensor Networks. Math Comput Model 53(3-4):485-503. Google Scholar
- Matzler K, Faullant R, Renzl B, Leiter V (2005) The relationship between personality traits (extraversion and neuroticism), emotions and customer self-satisfaction. Innov Mark 1(2):32-39.Google Scholar
- Mikels JA, Fredrickson BL, Larkin GR, Lindberg CM, Maglio SJ, Reuter-Lorenz PA (2005) Emotional category data on images from the international affective picture system. Behav Res Methods 37(4):626-630.Google Scholar
- Millon T, Lerner MJ (2003) Personality and social psychology, handbook of psychology. vol 5, Wiley, Hoboken.Google Scholar
- Miyamoto Y, Uchida Y, Ellsworth PC (2001) Culture and mixed emotions: co-occurrence of positive and negative emotions in Japan and the United States. Emotions 10(3):404-415.Google Scholar
- Na J, Grossmann I, Varnum MEW, Gonzalez R, Kitayama S, Nisbett RE (2010) When cultural differences are not reducible to individual differences. Proc Nat Acad Sci USA 107(14): 6192-6197.Google Scholar
- Nelson BJ, Kaliakatsos IK, Abbott JJ (2010) Microrobots for minimally invasive medicine. Ann Rev Biomed Eng 12:55-85.Google Scholar
- Neumann SA, Waldsein SR (2001) Similar patterns of cardiovascular response during emotional activation as a function of affective valence and arousal and gender. J Psychosom Res 50(5):245-253.Google Scholar
- Nevin JA, Reynolds GS (1973) The study of behavior: learning, motivation, emotion, and instinct. Scott, Foresman and Company, Glenview.Google Scholar
- Niedenthal PM, Augustinova M, Rychlowska M (2010) Body and mind: Zajonc's (re)introduction of the motor system to emotion and cognition. Emot Rev 2(4):340-347.Google Scholar
- Olejnik S, Li J, Supattathum S, Huberty CJ (1997) Multiple testing and statistical power with modified Bonferroni procedures. J Educ Behav Stat 22(4):389-406.Google Scholar
- Pantelopoulos A, Bourbakis NG (2010) A survey on wearable sensor-based systems for health monitoring and prognosis. IEEE Trans Syst Man Cybern C Appl Rev 40(1):1-12. Google Scholar
- Pantic M, Rothkrantz LJM (2003) Toward an affect-sensitive multimodal human-computer interaction. Proc IEEE 91(9): 1370-1390.Google Scholar
- Pawar T, Anantakrishnan NS, Chaudhuri S, Pawar TD, Duttagupta S (2008) Impact of ambulation in wearable-ECG. Ann Biomed Eng 36(9):1547-1557.Google Scholar
- Petridis S, Pantic M (2011) Audiovisual discrimination between speech and laughter: why and when visual information might help. IEEE Trans Multimed 13(2):216-234. Google Scholar
- Picard RW (1997) Affective computing. MIT Press, Boston. Google Scholar
- Picard RW, Vyzas E, Healey J (2001) Toward machine emotional intelligence: analysis of affective physiological state. IEEE Trans Pattern Anal Mach Intell 23(10):1175-1191. Google Scholar
- Rainville P, Bechara A, Naqvi N, Damasio AR (2006) Basic emotions are associated with distinct patterns of cardiorespiratory activity. Int J Psychophysiol 61(1):5-18.Google Scholar
- Regier T, Kay P (2009) Language, thought, and color: whorf was half right. Trends Cogn Sci 13(10):411-454.Google Scholar
- Richter D, Dietzel C, Kunzmann U (2010) Age differences in emotion recognition: the task matters. J Gerontol Psy Sci 66B(1):48-55.Google Scholar
- Russell JA (2003) Core affect and the psychological construction of emotion. Psychol Rev 110(1):145-172.Google Scholar
- Rutkowski TM, Tanaka T, Cichocki A, Erickson D, Cao J, Mandic DP (2011) Interactive component extraction from fEEG, fNIRS and peripheral biosignals for affective brain-machine interfacing paradigms. Comput Hum Behav 27(5):1512-1518. Google Scholar
- Schuller B, Batliner A, Steidl S, Seppi D (2011) Recognising realistic emotions and affect in speech: state of the art and lessons learnt from the first challenge. Speech Commun 27(5):1512-1518. Google Scholar
- Shen BJ, Stroud LR, Niaura R (2004) Ethnic differences in cardiovascular responses to laboratory stress: a comparison between Asian and white Americans. Int J Behav Med 11(3): 181-186.Google Scholar
- Skinner BF (1971) Beyond freedom and dignity. Alfred A. Knopf Inc., New York.Google Scholar
- Stead L, Goulev P, Evans C, Mamdani E (2004) The emotional wardrobe. Pers Ubiquitous Comput 8(3-4):282-290. Google Scholar
- Stemmler G, Wacker J (2010) Personality, emotion, and individual differences in physiological responses. Biol Psychol 83(3):541-551.Google Scholar
- Tan DS, Nijholt A (2010) Brain-Computer Interfaces: Applying our minds to human-computer interaction. Human-computer interaction series. Springer, London. Google Scholar
- Tawari A, Trivedi MM (2010) Speech emotion analysis: exploring the role of context. IEEE Trans MultiMed 12(6):502-509. Google Scholar
- Uchino BN, Uno D, Holt-Lunstad J, Flinders JB (1999) Agerelated differences in cardiovascular reactivity during acute psychological stress in men and women. J Gerontol Psychol Sci 54B(6):P339-P346.Google Scholar
- van den Broek EL et al (2009/2010/2011) Prerequisites for Affective signal processing (ASP)--Parts I-V. In: Fred A, Filipe J, Gamboa H (eds) BioSTEC 2009/2010/2011: Proceedings of the international joint conference on biomedical engineering systems and technologies, INSTICC Press, Porto, Portugal, Valencia, Spain, Rome, Italy.Google Scholar
- van den Broek EL (2010) Robot nannies: future or fiction?. Interact Stud 11(2):274-282.Google Scholar
- van den Broek EL, Westerink JHDM (2009) Considerations for emotion-aware consumer products. Appl Ergon 40(6):1055- 1064.Google Scholar
- van den Broek EL, Schut MH, Westerink JHDM, Tuinenbreijer K (2009) Unobtrusive sensing of emotions (USE). J Ambient Intell Smart Env 1(3):287-299. Google Scholar
- van den Broek EL, Lisy V, Janssen JH, Westerink JHDM, Schut MH, Tuinenbreijer K (2010) Affective man-machine interface: unveiling human emotions through biosignals, communications in computer and information science, vol 52, Springer, Berlin, pp 21-47.Google Scholar
- van den Broek EL, Nijholt A, Westerink JHDM (2010) Unveiling affective signals. In: Barakova E, de Ruyter B, Spink A (eds) ACM proceedings of measuring behavior 2010: selected papers from the 7th international conference on methods and techniques in behavioral research, ACM, New York, Eindhoven, The Netherlands, Article no. a6. Google Scholar
- van Drunen A, van den Broek EL, Spink AJ, Heffelaar T (2009) Exploring workload and attention measurements with uLog mouse data. Behav Res Methods 41(3):868-875.Google Scholar
- Wang L, Gu T, Tao X, Chen H, Lu J (2011) Recognizing multiuser activities using wearable sensors in a smart home. Pervasive Mob Comput 7(3):287-298. Google Scholar
- Weigert AJ (1991) Mixed emotions: certain steps toward understanding ambivalence. SUNY series in the sociology of emotions. State University of New York Press, Albany.Google Scholar
- Whitehill J, Littlewort G, Fasel I, Bartlett M, Movellan J (2009) Towards practical smile detection. IEEE Trans Pattern Anal Mach Intell 31(11):2106-2111. Google Scholar
- Whittle S, Yücel M, Yap MBH, Allen NB (2011) Sex differences in the neural correlates of emotion: evidence from neuroimaging. Biol Psychol 87(3):319-333.Google Scholar
- Wu S, Falk TH, Chan WY (2011) Automatic speech emotion recognition using modulation spectral features. Speech Commun 53(5):768-785. Google Scholar
- Xiao S, Dhamdhere A, Sivaraman V, Burdett A (2009) Transmission power control in body area sensor networks for healthcare monitoring. IEEE J Sel Areas Commun 27(1):37-48. Google Scholar
- Xiao R, Zhao Q, Zhang D, Shi P (2011) Facial expression recognition on multiple manifolds. Pattern Recognit 44(1):107- 116. Google Scholar
- Young MA (1993) Supplementing tests of statistical significance: variation accounted for. J Speech Hear Res 36(4):644- 656.Google Scholar
- Zeng Z, Pantic M, Roisman GI, Huang TS (2009) A survey of affect recognition methods: Audio, visual, and spontaneous expressions. IEEE Trans on Pattern Anal and Mach Intell 31(1):39-58. Google Scholar
- Zheng N, Wu Z, Lin M, Yang LT (2010) Enhancing battery efficiency for pervasive health-monitoring systems based on electronic textiles. IEEE Trans Inf Technol Biomed 14(2):350- 359. Google Scholar
Index Terms
- Ubiquitous emotion-aware computing
Recommendations
Unobtrusive Sensing of Emotions (USE)
Emotions are acknowledged as a crucial element for artificial intelligence; this is, as is illustrated, no different for Ambient Intelligence (AmI). Unobtrusive Sensing of Emotions (USE) is introduced to enrich AmI with empathic abilities. USE coins the ...
Unobtrusive Sensing of Emotions (USE)
Emotions are acknowledged as a crucial element for artificial intelligence; this is, as is illustrated, no different for Ambient Intelligence (AmI). Unobtrusive Sensing of Emotions (USE) is introduced to enrich AmI with empathic abilities. USE coins the ...
Are emoji valid indicators of in-the-moment mood?
AbstractDespite widespread assumptions that emoji represent emotions, research findings show that we do not process emoji in a way which we would expect for emotional stimuli. As such, we might be better placed to consider them more in line with mood ...
Highlights- The Emoji PANAS scale can reduce linguistic confounds when reporting current affective state
- Emoji are unlikely to be a valid means for reporting mood for those high in emotionality
- Emoji may be better consider mood indicators ...
Comments