How Can Affect Be Detected and Represented in Technological Support for Physical Rehabilitation?

Abstract

Although clinical best practice suggests that affect awareness could enable more effective technological support for physical rehabilitation through personalisation to psychological needs, designers need to consider what affective states matter, and how they should be tracked and addressed. In this article, we set the standard by analysing how the major affective factors in chronic pain (pain, fear/anxiety, and low/depressed mood) interfere with everyday physical functioning. Further, based on discussion of the modality that should be used to track these states to enable technology to address them, we investigated the possibility of using movement behaviour to automatically detect the states. Using two body movement datasets on people with chronic pain, we show that movement behaviour enables very good discrimination between two emotional distress levels (F1=0.86), and three pain levels (F1=0.9). Performance remained high (F1=0.78 for two pain levels) with a reduced set of movement sensors. Finally, in an overall discussion, we suggest how technology-provided encouragement and awareness can be personalised given the capability to automatically monitor the relevant states, towards addressing the barriers that they pose. In addition, we highlight movement behaviour features to be tracked to provide technology with information necessary for such personalisation.

References

1. Jyoti Joshi, Abhinav Dhall, Roland Goecke, and Jeffrey F. Cohn. 2013. Relative body parts movement for automatic depression analysis. In Proceedings of ACII. 492--497.
2. Jyoti Joshi, Roland Göecke, Sharifa Alghowinem, Abhinav Dhall, Michael Wagner, Julien Epps, Gordon Parker, and Michael Breakspear. 2013. Multimodal assistive technologies for depression diagnosis and monitoring. J. Multimodal User Interfaces 7, 3 (2013), 217--228.
3. Jyoti Joshi, Roland Göecke, Gordon Parker, and Michael Breakspear. 2013. Can body expressions contribute to automatic depression analysis? In Proceedings of FG. 1--7.
4. David DeVault, Ron Artstein, Grace Benn, Teresa Dey, Ed Fast, Alesia Gainer, Kallirroi Georgila, Jon Gratch, Arno Hartholt, Margaux Lhommet, Gale Lucas, Stacy Marsella, Fabrizio Morbini, Angela Nazarian, Stefan Scherer, Giota Stratou, Apar Suri, David Traum, Rachel Wood, Yuyu Xu, Albert Rizzo, and Louis-Philippe Morency. 2014. SimSensei kiosk: A virtual human interviewer for healthcare decision support. In Proceedings of AAMAS. 1061--1068.
5. Neeru Rathee and Dinesh Ganotra. 2016. Multiview distance metric learning on facial feature descriptors for automatic pain intensity detection. Comput. Vis. Image Underst. 147 (2016), 77--86.
6. George Michael, Kyriakos Tsaparellas, Gabriel Panis, Christos P. Loizou, and Andreas Lanitis. 2016. Towards non-invasive patient monitoring through iris tracking and pain detection. In Proceedings of Mediterranean Conference on Medical and Biological Engineering and Computing. 361--366.
7. Philipp Werner, Ayoub Al-Hamadi, Kerstin Limbrecht-Ecklundt, Steffen Walter, Sascha Gruss, and Harald Traue. 2016. Automatic pain assessment with facial activity descriptors. IEEE Trans. Affect. Comput. 8, 3 (2017), 286--299
8. Sourav Dey Roy, Mrinal Kanti Bhowmik, Priya Saha, and Anjan Kumar Ghosh. 2016. An approach for automatic pain detection through facial expression. Procedia Comput. Sci. 84 (2016), 99--106.
9. Jing Zhou, Xiaopeng Hong, Fei Su, and Guoying Zhao. 2016. Recurrent convolutional neural network regression for continuous pain intensity estimation in video. In Proceedings of ICVR. 84--92.
10. Christian Schönauer, T. Pintaric, H. Kaufmann, S. Jansen-Kosterink, and M. Vollenbroek-Hutten. 2011. Chronic pain rehabilitation with a serious game using multimodal input. In Proceedings of ICVR. 1--8.
11. Aneesha Singh, Annina Klapper, Jinni Jia, Antonio Fidalgo, Ana Tajadura-Jiménez, Natalie Kanakam, Nadia Bianchi-Berthouze, and Amanda Williams. 2014. Motivating people with chronic pain to do physical activity: Opportunities for technology design. In Proceedings of CHI. 2803--2812.
12. Johan W. S. Vlaeyen, Stephen Morley, and Geert Crombez. 2016. The experimental analysis of the interruptive, interfering, and identity-distorting effects of chronic pain. Behav. Res. Ther. 86 (2016), 23--34.
13. Ali Asghari and Michael K. Nicholas. 2001. Pain self-efficacy beliefs and pain behaviour. A prospective study. Pain 94, 1 (2001), 85--100.
14. Dennis Turk. 2015. Generalization and maintenance of performance. In Fordyce's Behavioural Methods for Chronic Pain and Illness. C. Main, F. Keefe, M. Jensen, J. Vlaeyan, and K. Vowles (Eds.), IASP Press, Washington, DC, 415--427.
15. Michael J. L. Sullivan, Pascal Thibault, André Savard, Richard Catchlove, John Kozey, and William D. Stanish. 2006. The influence of communication goals and physical demands on different dimensions of pain behavior. Pain 125, 3 (2006), 270--277.
16. Temitayo Olugbade, M. Aung, N. Bianchi-Berthouze, N. Marquardt, and A. Williams. 2014. Bi-modal detection of painful reaching for chronic pain rehabilitation systems. In Proceedings of ICMI. 455--458.
17. Temitayo A. Olugbade, Nadia Bianchi-Berthouze, Nicolai Marquardt, and Amanda C. Williams. 2015. Pain level recognition using kinematics and muscle activity for physical rehabilitation in chronic pain. In Proceedings of ACII. 243--249.
18. Naveen L. Bagalkot, Tomas Sokoler, and Suraj Baadkar. 2016. ReRide: Performing lower back rehabilitation while riding your motorbike in traffic. In Proceedings of Pervasive Health. 77--80.
19. Aneesha Singh, Nadia L. Berthouze-Bianchi, and A. Williams. 2017. Supporting everyday function in chronic pain using a wearable device. In Proceedings of CHI.
20. Harald Breivik, Beverly Collett, Vittorio Ventafridda, Rob Cohen, and Derek Gallacher. 2006. Survey of chronic pain in Europe: Prevalence, impact on daily life, and treatment. Eur. J. Pain 10, 4 (2006), 287--333.
21. Catherine Johannes, T. Kim Le, X. Zhou, J. Johnston, and R. Dworkin. 2010. The prevalence of chronic pain in United States adults: results of an Internet-based survey. J. Pain 11, 11 (2010), 1230--1239.
22. Irene Tracey and M. Catherine Bushnell. 2009. How neuroimaging studies have challenged us to rethink: is chronic pain a disease? J. Pain 10, 11 (2009), 1113--1120.
23. The Pain Consortium. 2016. UK Pain Messages. Pain News.
24. Eva Denison, P. Åsenlöf, and P. Lindberg. 2004. Self-efficacy, fear avoidance, and pain intensity as predictors of disability in subacute and chronic musculoskeletal pain patients in primary health care. Pain 111, 3 (2004), 245--252.
25. Johan W. S. Vlaeyen and Steven J. Linton. 2000. Fear-avoidance and its consequences in chronic musculoskeletal pain: A state of the art. Pain 85, 3 (2000), 317--332.
26. Sara M. Banks and Robert D. Kerns. 1996. Explaining high rates of depression in chronic pain: A diathesis-stress framework. Psychological Bulletin 119, 1 (1996), 95--110.
27. Adina Rusu, Tamar Pincus, and Stephen Morley. 2012. Depressed pain patients differ from other depressed groups: Examination of cognitive content in a sentence completion task. Pain 153, 9 (2012), 1898--1904.
28. Virginia Braun and V. Clarke. 2006. Using thematic analysis in psychology. Qual. Res. Psychol. 3, 2 (2006), 77--101.
29. Vicki Harding and Paul J. Watson. 2000. Increasing activity and improving function in chronic pain management. Physiotherapy 86, 12 (2000), 619--630.
30. Chris Eccleston. 2001. Role of psychology in pain management. Br. J. Anaesth. 87, 1 (2001), 144--152.
31. Zina Trost, Karoline Vangronsveld, Steven J. Linton, Phillip J. Quartana, and Michael J. L. Sullivan. 2012. Cognitive dimensions of anger in chronic pain. Pain 153, 3 (2012), 515--517.
32. Tamar Pincus, Rob Smeets, Maureen Simmonds, and Michael Sullivan. 2010. The fear avoidance model disentangled: improving the clinical utility of the fear avoidance model. Clin. J. Pain 26, 9 (2010), 739--746.
33. Angela S. Lee, Jacek Cholewicki, N. Peter Reeves, Bohdanna T. Zazulak, and Lawrence W. Mysliwiec. 2010. Comparison of trunk proprioception between patients with low back pain and healthy controls. Arch. Phys. Med. Rehabil. 91, 9 (2010), 1327--1331.
34. Andrea Kleinsmith and Nadia Bianchi-Berthouze. 2013. Affective body expression perception and recognition: A survey. IEEE Trans. Affect. Comput. 4, 1 (2013), 15--33.
35. Hatice Gunes, Caifeng Shan, Shizhi Chen, and YingLi Tian. 2015. Bodily expression for automatic affect recognition. In Emotion Recognition: A Pattern Analysis Approach (1st ed.). Amit Konar and Aruna Chakraborty (Eds.), Wiley and Sons, 343--377.
36. Min S. H. Aung, Sebastian Kaltwang, Bernardino Romera-Paredes, Brais Martinez, Aneesha Singh, Matteo Cella, Michel Valstar, Hongying Meng, Andrew Kemp, Moshen Shafizadeh, Aaron C. Elkins, Natalie Kanakam, Amschel de Rothschild, Nick Tyler, Paul J. Watson, Amanda C. de C. Williams, Maja Pantic, and Nadia Bianchi-Berthouze. 2016. The automatic detection of chronic pain-related expression: requirements, challenges and the multimodal EmoPain dataset. IEEE Trans. Affect. Comput. 7, 4 (2016), 435--451.
37. Rafael A. Calvo and Sidney D'Mello. 2010. Affect detection: An interdisciplinary review of models, methods, and their applications. IEEE Trans. Affect. Comput. 1, 1 (2010), 18--37.
38. Paul J. Watson, C. Kerry Booker, and Chris J. Main. 1997. Evidence for the role of psychological factors in abnormal paraspinal activity in patients with chronic low back pain. J. Musculoskelet. Pain 5, 4 (1997), 41--56.
39. Paul J. Watson, C. K. Booker, Chris J. Main, and A. C. N. Chen. 1997. Surface electromyography in the identification of chronic low back pain patients: the development of the flexion relaxation ratio. Clin. Biomech. 12, 3 (1997), 165--171.
40. Steffen Walter, Sascha Gruss, Kerstin Limbrecht-Ecklundt, Harald C. Traue, Philipp Werner, Ayoub Al-Hamadi, Nicolai Diniz, Gustavo Moreira da Silva, and Adriano O. Andrade. 2014. Automatic pain quantification using autonomic parameters. Psychol. Neurosci. 7, 3 (2014), 363.
41. Markus Kächele, Mohammadreza Amirian, Patrick Thiam, Philipp Werner, Steffen Walter, Günther Palm, and Friedhelm Schwenker. 2016. Adaptive confidence learning for the personalization of pain intensity estimation systems. Evol. Syst. 8, 1 (2017), 71--83.
42. Valéry Legrain, Stefaan Van Damme, Christopher Eccleston, Karen D. Davis, David A. Seminowicz, and Geert Crombez. 2009. A neurocognitive model of attention to pain: behavioral and neuroimaging evidence. Pain 144, 3 (2009), 230--232.
43. S. Pearce, S. Isherwood, D. Hrouda, P. Richardson, A. Erskine, and J. Skinner. 1990. Memory and pain: tests of mood congruity and state dependent learning in experimentally induced and clinical pain. Pain 43, 2 (1990), 187--193.
44. David K. Ahern, Michael J. Follick, James R. Council, Nancy Laser-Wolston, and Henry Litchman. 1988. Comparison of lumbar paravertebral EMG patterns in chronic low back pain patients and non-patient controls. Pain 34, 2 (1998), 153--160.
45. Helena Grip, Fredrik Ohberg, Urban Wiklund, Ylva Sterner, J. Stefan Karlsson, and Björn Gerdle. 2003. Classification of neck movement patterns related to whiplash-associated disorders using neural networks. IEEE Trans. Inf. Technol. Biomed. 7, 4 (2003), 412--418.
46. Daniel T. H. Lai, Pazit Levinger, Rezaul K. Begg, Wendy Lynne Gilleard, and Marimuthu Palaniswami. 2009. Automatic recognition of gait patterns exhibiting patellofemoral pain syndrome using a support vector machine approach. IEEE Trans. Inf. Technol. Biomed. 13, 5 (2009), 810--817.
47. James P. Dickey, Michael R. Pierrynowski, Drew A. Bednar, and Simon X. Yang. 2002. Relationship between pain and vertebral motion in chronic low-back pain subjects. Clin. Biomech. 17, 5 (2002), 345--352.
48. Jeffrey F. Cohn, Tomas Simon Kruez, Iain Matthews, Ying Yang, Minh Hoai Nguyen, Margara Tejera Padilla, Feng Zhou, and Fernando De la Torre. 2009. Detecting depression from facial actions and vocal prosody. In Proceedings of ACII. 1--7.
49. Sharifa Alghowinem, Roland Goecke, Michael Wagner, Gordon Parkerx, and Michael Breakspear. 2013. Head pose and movement analysis as an indicator of depression. In Proceedings of ACII. 283--288.
50. Sharifa Alghowinem, Roland Goecke, Julien Epps, Michael Wagner, and Jeffrey Cohn. 2016. Cross-cultural depression recognition from vocal biomarkers. In Proceedings of Interspeech. 1943--1947.
51. Anastasia Pampouchidou, Anastasia Pampouchidou, Olympia Simantiraki, Amir Fazlollahi, Matthew Pediaditis, Dimitrios Manousos, Alexandros Roniotis, Georgios Giannakakis, Fabrice Meriaudeau, Panagiotis Simos, Kostas Marias, Fan Yang, and Manolis Tsiknakis. 2016. Depression assessment by fusing high and low level features from audio, video, and text. In Proceedings of AVEC. 27--34.
52. Michel Valstar, Björn Schuller, Kirsty Smith, Florian Eyben, Bihan Jiang, Sanjay Bilakhia, Sebastian Schnieder, Roddy Cowie, and Maja Pantic. 2013. AVEC 2013: The continuous audio/visual emotion and depression recognition challenge. In Proceedings of AVEC. 3--10.
53. Md. Nasir, Arindam Jati, Prashanth Gurunath Shivakumar, Sandeep Nallan Chakravarthula, and Panayiotis Georgiou. 2016. Multimodal and multiresolution depression detection from speech and facial landmark features. In Proceedings of AVEC. 43--50.
54. Le Yang, Dongmei Jiang, Lang He, Ercheng Pei, Meshia Cedric Oveneke, and Hichem Sahli. 2016. Decision tree based depression classification from audio video and language information. In Proceedings of AVEC. 89--96.
55. Tamar Pincus and Stephen Morley. 2001. Cognitive-processing bias in chronic pain: a review and integration. Psych. Bull. 127, 5 (2001), 599.
56. Hongying Meng, Andrea Kleinsmith, and Nadia Bianchi-Berthouze. 2011. Multi-score learning for affect recognition: The case of body postures. In Proceedings of ACII. 225--234.
57. Shizhi Chen, Y. Tian, Q. Liu, and D. Metaxas. 2013. Recognizing expressions from face and body gesture by temporal normalized motion and appearance features. Image Vis. Comput. 31, 2 (2013), 175--185.
58. Stefano Piana, Alessandra Stagliano, Francesca Odone, Alessandro Verri, and Antonio Camurri. 2014. Real-time automatic emotion recognition from body gestures. arXiv:1402.5047.
59. Ali-Akbar Samadani, Ali Ghodsi, and Dana Kulić. 2013. Discriminative functional analysis of human movements. Pattern Recognit. Lett. 34, 15 (2013), 1829--1839.
60. Pramila Rani, Nilanjan Sarkar, and Changchun Liu. 2005. Maintaining optimal challenge in computer games through real-time physiological feedback. In Proceedings of CHI.
61. Francis J. Keefe and Andrew R. Block. 1982. Development of an observation method for assessing pain behavior in chronic low back pain patients. Behav. Ther. 13, 4 (1982), 363--375.
62. Aneesha, Singh, Stefano Piana, Davide Pollarolo, Gualtiero Volpe, Giovanna Varni, Ana Tajadura-Jiménez, Amanda C. de C. Williams, Antonio Camurri, and Nadia Bianchi-Berthouze. 2016. Go-with-the-flow: Tracking, analysis and sonification of movement and breathing to build confidence in activity despite chronic pain. Hum.-Comput. Interact. 31, 3-4 (2016), 335--383.
63. Temitayo A. Olugbade, Nadia Bianchi-Berthouze, Nicolai Marquardt, and Amanda C. Williams. 2018. Human observer and automatic assessment of movement related self-efficacy in chronic pain: From exercise to functional activity. IEEE Trans. Affect. Comput. (2018).
64. Marc Thioux, Valeria Gazzola, and Christian Keysers. 2008. Action understanding: How, what and why. Curr. Biol. 18, 10 (2008), R431--34.
65. Wim G. M. Janssen, Hans BJ Bussmann, and Henk J. Stam. 2002. Determinants of the sit-to-stand movement: A review. Phys. Ther. 82, 9 (2002), 866.
66. Anthony S. Zigmond and R. Philip Snaith. 1983. The hospital anxiety and depression scale. Acta Psychiatr. Scand. 67, 6 (1983), 361--370.
67. Theodore D. Cosco, Frank Doyle, Mark Ward, and Hannah McGee. 2012. Latent structure of the hospital anxiety and depression scale: A 10-year systematic review. J. Psychosom. Res. 72, 3 (2012), 180--184.
68. Adina C. Rusu, Rita Santos, and Tamar Pincus. 2016. Pain-related distress and clinical depression in chronic pain: A comparison between two measures. Scand. J. Pain 12 (2016), 62--67.
69. Tamar Pincus, Amanda C. de C. Williams, Steven Vogel, and Andy Field. 2004. The development and testing of the depression, anxiety, and positive outlook scale (DAPOS). Pain 109, 1 (2004), 181--188.
70. Julie F. Pallant and Catherine M. Bailey. 2005. Assessment of the structure of the hospital anxiety and depression scale in musculoskeletal patients. Health Qual. Life Outcomes 3, 1 (2005), 82.
71. Mark Jensen and P. Karoly. 1992. Self-report scales and procedures for assessing pain in adults. In Handbook of Pain Assessment (2nd ed.). D. Turk and R. Melzack (Eds.), Guilford Press, New York, 135--151.
72. Peter Waxer. 1974. Nonverbal cues for depression. J. Abnorm. Psychol. 83, 3 (1974), 319--322.
73. Stefan Scherer, Giota Stratou, Marwa Mahmoud, Jill Boberg, Jonathan Gratch, Albert Rizzo, and Louis-Philippe Morency. 2013. Automatic behavior descriptors for psychological disorder analysis. In Proceedings of FG. 1--8.
74. Nesrine Fourati and Catherine Pelachaud. 2015. Relevant body cues for the classification of emotional body expression in daily actions. In Proceedings of ACII. 267--273.
75. José Luís Pimentel do Rosário, Maria Suely Bezerra Diógenes, Rita Mattei, and José Roberto Leite. 2013. Can sadness alter posture? J. Bodyw. Mov. Ther. 17, 3 (2013), 328--331.
76. Johannes Michalak, Nikolaus F. Troje, Julia Fischer, Patrick Vollmar, Thomas Heidenreich, and Dietmar Schulte. 2009. Embodiment of sadness and depression—Gait patterns associated with dysphoric mood. Psychosom. Med. 71, 5 (2009), 580--587.
77. M. Wada, N. Sunaga, and M. Nagai. 2001. Anxiety affects the postural sway of the antero-posterior axis in college students. Neurosci. Lett. 302, 2 (2001), 157--159.
78. Matthias R. Lemke, Thomas Wendorff, Brigitt Mieth, Katharina Buhl, and Martin Linnemann. 2000. Spatiotemporal gait patterns during over ground locomotion in major depression compared with healthy controls. J. Psychiatr. Res. 34, 4 (2000), 277--283.
79. Gary Shum, Jack Crosbie, and Raymond Lee. 2005. Effect of low back pain on the kinematics and joint coordination of the lumbar spine and hip during sit-to-stand and stand-to-sit. Spine 30, 17 (2005), 1998--2004.
80. G. Gioftsos and D. W. Grieve. 1996. The use of artificial neural networks to identify patients with chronic low-back pain conditions from patterns of sit-to-stand manoeuvres. Clin. Biomech. 11, 5 (1996), 275--280.
81. D. E. Marple-Horvat and S. L. Gilbey. 1992. A method for automatic identification of periods of muscular activity from EMG recordings. J. Neurosci. Methods 42, 3 (1992), 163--167.
82. Corinna Cortes and Vladimir Vapnik. 1995. Support-vector networks. Mach. Learn. 20, 3 (1995), 273--297.
83. Leo Breiman. 2001. Random forests. Mach. Learn. 45, 1 (2001), 5--32.
84. Leo Breiman, Jerome H. Friedman, Richard A. Olshen, and Charles J. Stone. 1984. Classification and Regression Trees. Wadsworth 8 Brooks. Monterey.
85. Andy Field. 2013. Discovering Statistics Using IBM SPSS Statistics (4th ed.). SAGE Publications, London.
86. Patrenahalli M. Narendra and Keinosuke Fukunaga. 1977. A branch and bound algorithm for feature subset selection. IEEE Trans. Comput. 26, 9 (1977), 917--922.
87. Sunny Consolvo, David W. McDonald, and James A. Landay. 2009. Theory-driven design strategies for technologies that support behavior change in everyday life. In Proceedings of CHI. 405--414.
88. Glenn Affleck, Howard Tennen, Alex Zautra, Susan Urrows, Micha Abeles, and Paul Karoly. 2001. Women's pursuit of personal goals in daily life with fibromyalgia: A value-expectancy analysis. J. Consult. Clin. Psychol. 69, 4 (2001), 587.
89. Kristina Höök, Martin Jonsson, Anna Ståhl, and Johanna Mercurio. 2016. Somaesthetic appreciation design. In Proceedings of CHI. 3131--3142.
90. Filippo Casamassima, Alberto Ferrari, Bojan Milosevic, Pieter Ginis, Elisabetta Farella, and Laura Rocchi. 2014. A wearable system for gait training in subjects with Parkinson's disease. Sensors 14, 4 (2014), 6229--6246.
91. Stephanie M. Jansen-Kosterink, Rianne M. H. A. Huis In't Veld, Christian Schönauer, Hannes Kaufmann, Hermie J. Hermens, and Miriam M. R. Vollenbroek-Hutten. 2013. A serious exergame for patients suffering from chronic musculoskeletal back and neck pain: a pilot study. Games Health J. 2, 5 (2013), 299--307.
92. Sergio Felipe, Aneesha Singh, Caroline Bradley, Amanda C. de C. Williams, and Nadia Bianchi-Berthouze. 2015. Roles for personal informatics in chronic pain. In Proceedings of PervasiveHealth. 161--168.
93. Dennis C. Turk and Akiko Okifuji. 2002. Psychological factors in chronic pain: Evolution and revolution. J. Consult. Clin. Psychol. 70, 3 (2002), 678--690.
94. Geoffrey B. Duggan, Edmund Keogh, Gail A. Mountain, Paul McCullagh, Jason Leake, and Christopher Eccleston. 2015. Qualitative evaluation of the SMART2 self-management system for people in chronic pain. Disabil. Rehabil. Assist. Technol. 10, 1 (2015), 53--60.
95. Enrica Papi, Athina Belsi, and Alison H. McGregor. 2015. A knee monitoring device and the preferences of patients living with osteoarthritis: a qualitative study. BMJ Open 5, 9 (2015), e007980.
96. Aisling Ann O'Kane, Yvonne Rogers, and Ann E. Blandford. 2015. Concealing or revealing mobile medical devices?: Designing for onstage and offstage presentation. In Proceedings of CHI. 1689--1698.
97. Jhim Kiel M. Verame, Enrico Costanza and Sarvapali D. Ramchurn. 2016. The effect of displaying system confidence information on the usage of autonomous systems for non-specialist applications: A lab study. In Proceedings of CHI. 4908--4920.

Index Terms

• Applied computing

  • Life and medical sciences

    • Health informatics

• Human-centered computing

  • Human computer interaction (HCI)

    • HCI design and evaluation methods

  • Ubiquitous and mobile computing

    • Empirical studies in ubiquitous and mobile computing