Emotion recognition using wireless signals

Abstract

This paper demonstrates a new technology that can infer a person's emotions from RF signals reflected off his body. EQ-Radio transmits an RF signal and analyzes its reflections off a person's body to recognize his emotional state (happy, sad, etc.). The key enabler underlying EQ-Radio is a new algorithm for extracting the individual heartbeats from the wireless signal at an accuracy comparable to on-body ECG monitors. The resulting beats are then used to compute emotion-dependent features which feed a machine-learning emotion classifier. We describe the design and implementation of EQ-Radio, and demonstrate through a user study that its emotion recognition accuracy is on par with state-of-the-art emotion recognition systems that require a person to be hooked to an ECG monitor.

References

1. Microsoft project oxford emotion api. https://www.projectoxford.ai/emotion.
2. Noise robust differentiators for second derivative estimation. http://www.holoborodko.com/pavel/downloads/NoiseRobustSecondDerivative.
3. Acharya, U.R., Joseph, K.P., Kannathal, N., Lim, C.M., Suri, J.S. Heart rate variability: a review. Medical and Biological Engineering and Computing 44, 12 (2006), 1031--1051.
4. Adib, F., Katabi, D. See through walls with wifi!. Proceedings of the ACMSIGCOMM 2013 (New York, NY, USA, 2013), ACM, 75--86.
5. Adib, F., Kabelac, Z., Katabi, D., Miller, R.C. 3d tracking via body radio reflections. 11th USENIX Symposium on Networked Systems Design and Implementation (2014).
6. Adib, F., Mao, H., Kabelac, Z., Katabi, D., Miller, R.C. Smart homes that monitor breathing and heart rate. Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (2015), ACM, 837--846.
7. Agrafioti, F., Hatzinakos, D., Anderson, A.K. Ecg pattern analysis for emotion detection. Affective Computing, IEEE Transactions on 3, 1 (2012),102--115.
8. Ali, K., Liu, A.X., Wang, W., Shahzad, M. Keystroke recognition using wifi signals. Proceedings of the 21st Annual International Conference on Mobile Computing and Networking (2015), ACM, 90--102.
9. Anitori, L., de Jong, A., Nennie, F. Fmcw radar for life-sign detection. Radar Conference, 2009 IEEE (2009), IEEE, 1--6.
10. Appelhans, B.M., Luecken, L.J. Heart rate variability as an index of regulated emotional responding. Review of General Psychology 10, 3 (2006), 229.
11. Boric-Lubecke, O., Massagram, W., Lubecke, V.M., Host-Madsen, A., Jokanovic, B. Heart rate variability assessment using doppler radar with linear demodulation. Microwave Conference, 2008. EuMC 2008. 38th European (2008), IEEE, 420--423.
12. Calvo, R.A., D'Mello, S. Affect detection: An interdisciplinary review of models, methods, and their applications. Affective Computing, IEEE Transactions on 1, 1 (2010), 18--37.
13. Cowie, R., Douglas-Cowie, E., Tsapatsoulis, N., Votsis, G., Kollias, S., Fellenz, W., Taylor, J.G. Emotion recognition in human-computer interaction. Signal Processing Magazine, IEEE 18, 1 (2001), 32--80.
14. Devijver, P.A., Kittler, J. Pattern Recognition: A Statistical Approach, volume 761. Prentice-Hall, London, 1982.
15. Droitcour, A.D., Boric-Lubecke, O., Kovacs, G.T. Signal-to-noise ratio in doppler radar system for heart and respiratory rate measurements. Microwave Theory and Techniques, IEEE Transactions on 57, 10 (2009), 2498--2507.
16. Ekman, P., Friesen, W.V., Ellsworth, P. Emotion in the Human Face: Guidelines for Research and an Integration of Findings. Elsevier, 2013.
17. El Ayadi, M., Kamel, M.S., Karray, F. Survey on speech emotion recognition: Features, classification schemes, and databases. Pattern Recognition 44, 3 (2011), 572--587.
18. Elfenbein, H.A., Ambady, N. 0n the universality and cultural specificity of emotion recognition: a meta-analysis. Psychological Bulletin 128, 2 (2002), 203.
19. Fletcher, R., Han, J. Low-cost differential front-end for doppler radar vital sign monitoring. Microwave Symposium Digest, 2009. MTT'09. IEEE MTT-S International (2009), IEEE, 1325--1328.
20. Hu, W., Zhao, Z., Wang, Y., Zhang, H., Lin, F. Noncontact accurate measurement of cardiopulmonary activity using a compact quadrature doppler radar sensor. Biomedical Engineering, IEEE Transactions on 61, 3 (2014), 725--735.
21. Jerritta, S., Murugappan, M., Nagarajan, R., Wan, K. Physiological signals based human emotion recognition: a review. Signal Processing and its Applications (CSPA), 2011 IEEE 7th International Colloquium on (2011), IEEE, 410--415.
22. Kahou, S.E., Bouthillier, X., Lamblin, P., Gulcehre, C., Michalski, V., Konda, K., Jean, S., Froumenty, P., Dauphin, Y., Boulanger-Lewandowski, N., et al. Emonets: Multimodal deep learning approaches for emotion recognition in video. Journal on Multimodal User Interfaces (2015), 1--13.
23. Kaltiokallio, O., Yigitler, H., Jantti, R., Patwari, N. Non-invasive respiration rate monitoring using a single cots tx-rx pair. Information Processing in Sensor Networks, IPSN-14 Proceedings of the 13th International Symposium on (2014), IEEE, 59--69.
24. Kamen, P.W., Krum, H., Tonkin, A.M. Poincare plot of heart rate variability allows quantitative display of parasympathetic nervous activity in humans. Clinical Science 91, 2 (1996), 201--208.
25. Kashdan, T.B., Mishra, A., Breen, W.E., Froh, J.J. Gender differences in gratitude: Examining appraisals, narratives, the willingness to express emotions, and changes in psychological needs. J. Pers. 77, 3 (2009), 691--730.
26. Kim, J., André, E. Emotion recognition based on physiological changes in music listening. Pattern Analysis and Machine Intelligence, IEEE Transactions on 30, 12 (2008), 2067--2083.
27. Kreibig, S.D. Autonomic nervous system activity in emotion: A review. Biological Psychology 84, 3 (2010), 394--421.
28. Lake, D.E., Richman, J.S., Griffin, M.P., Moorman, J.R. Sample entropy analysis of neonatal heart rate variability. American Journal of Physiology-Regulatory, Integrative and Comparative Physiology 283, 3 (2002), R789--R797.
29. Lane, R.D., McRae, K., Reiman, E.M., Chen, K., Ahern, G.L., Thayer, J.F. Neural correlates of heart rate variability during emotion. Neuroimage 44, 1 (2009), 213--222.
30. Lang, P.J. The emotion probe: studies of motivation and attention. American Psychologist 50, 5 (1995), 372.
31. Massagram, W., Lubecke, V.M., Host-Madsen, A., Boric-Lubecke, 0. Assessment of heart rate variability and respiratory sinus arrhythmia via doppler radar. Microwave Theory and Techniques, IEEE Transactions on 57, 10 (2009), 2542--2549.
32. McKinley, S., Levine, M. Cubic spline interpolation. College of the Redwoods 45, 1 (1998), 1049--1060.
33. Patwari, N., Brewer, L., Tate, Q., Kaltiokallio, O., Bocca, M.. Breathfinding: A wireless network that monitors and locates breathing in a home. Selected Topics in Signal Processing, IEEE Journal of 8, 1 (2014), 30--42.
34. Penzel, T., Kantelhardt, J.W., Grote, L., Peter, J.-H., Bunde, A. Comparison of detrended fluctuation analysis and spectral analysis for heart rate variability in sleep and sleep apnea. Biomedical Engineering, IEEE Transactions on 50, 10 (2003), 1143--1151.
35. Picard, R.W. Affective computing: from laughter to ieee. Affective Computing, IEEE Transactions on 1, 1 (2010), 11--17.
36. Picard, R.W., Vyzas, E., Healey, J. Toward machine emotional intelligence: Analysis of affective physiological state. Pattern Analysis and Machine Intelligence, IEEE Transactions on 23, 10 (2001). 1175--1191.
37. Postolache, O., Girão, P.S., Postolache, G., Gabriel, J. Cardiorespiratory and daily activity monitor based on fmcw doppler radar embedded in a wheelchair. Engineering in Medicine and Biology Society, EMBC, 2011 Annual International Conference of the IEEE (2011), IEEE, 1917--1920.
38. Pu, Q., Gupta, S., Gollakota, S., Patel, S. Whole-home gesture recognition using wireless signals. Proceedings of the 19th Annual International Conference on Mobile Computing & Networking (2013), ACM, 27--38.
39. Quintana, D.S., Guastella, A.J., Outhred, T., Hickie, I.B., Kemp, A.H. Heart rate variability is associated with emotion recognition: direct evidence for a relationship between the autonomic nervous system and social cognition. International Journal of Psychophysiology 86, 2 (2012), 168--172.
40. Saha, D.P., Martin, T.L., Knapp, R.B. Towards incorporating affective feedback into context-aware intelligent environments. Affective Computing and Intelligent Interaction (ACII), 2015 International Conference on (2015), IEEE, 49--55.
41. Sakamoto, T., Imasaka, R., Taki, H., Sato, T., Yoshioka, M., Inoue, K., Fukuda, T., Sakai, H. Feature-based correlation and topological similarity for interbeat interval estimation using ultra-wideband radar. Biomedical Engineering, IEEE Transactions on (2015).
42. Scherer, K.R. Vocal communication of emotion: A review of research paradigms. Speech Communication 40, 1 (2003), 227--256.
43. Sun, L., Sen, S., Koutsonikolas, D., Kim, K.-H. Widraw: Enabling hands-free drawing in the air on commodity wifi devices. Proceedings of the 21st Annual International Conference on Mobile Computing and Networking (2015), ACM, 77--89.
44. Sztajzel, J., et al. Heart rate variability: a noninvasive electrocardiographic method to measure the autonomic nervous system. Swiss Medical Weekly 134 (2004), 514--522.
45. Tse, D., Viswanath, P. Fundamentals of Wireless Communication. Cambridge university press, 2005.
46. Wang, W., Liu, A.X., Shahzad, M., Ling, K., Lu, S. Understanding and modeling of wifi signal based human activity recognition. Proceedings of the 21st Annual International Conference on Mobile Computing and Networking (2015), ACM, 65--76.
47. Wiens, A.D., Etemadi, M., Roy, S., Klein, L., Inan, O.T. Toward continuous, noninvasive assessment of ventricular function and hemodynamics: Wearable ballistocardiography. Biomedical and Health Informatics, IEEE Journal of 19, 4 (2015), 1435--1442.
48. Zeng, Z., Pantic, M., Roisman, G.I., Huang, T.S. A survey of affect recognition methods: Audio, visual, and spontaneous expressions. Pattern Analysis and Machine Intelligence, IEEE Transactions on 31, 1 (2009), 39--58.
49. Zhao, M., Adib, F., Katabi, D. Emotion recognition using wireless signals. Proceedings of the 22nd Annual International Conference on Mobile Computing and Networking (2016), ACM, 95--108.
50. Zhu, J., Rosset, S., Hastie, T., Tibshirani, R. 1-norm support vector machines. Advances in Neural Information Processing Systems 16, 1 (2004), 49--56.

Index Terms

• Human-centered computing

  • Human computer interaction (HCI)

    • Interaction techniques

  • Ubiquitous and mobile computing

    • Ubiquitous and mobile computing theory, concepts and paradigms

• Networks

  • Network components

    • Wireless access points, base stations and infrastructure

  • Network types

    • Cyber-physical networks