Timothy Greer
ABSTRACT
Music has a powerful influence on human experience. In this paper, we investigate how music affects brain activity, physiological response, and human-reported behavior. Using auditory features related to dynamics, timbre, harmony, rhythm, and register, we predicted brain activity in the form of phase synchronizations in bilateral Heschl's gyri and superior temporal gyri; physiological response in the form of galvanic skin response and heart activity; and emotional experience in the form of continuous, subjective descriptions reported by music listeners. We found that using multivariate time series models with attention mechanisms are effective in predicting emotional ratings, while vector-autoregressive models are effective in predicting involuntary human responses. Musical features related to dynamics, register, rhythm, and harmony were found to be particularly helpful in predicting these human reactions. This work adds to our understanding of how music affects multimodal human experience and has applications in affective computing, music emotion recognition, neuroscience, and music information retrieval.
- Vinoo Alluri, Petri Toiviainen, Iiro P Jaaskelainen, Enrico Glerean, Mikko Sams, and Elvira Brattico. 2012. Large-scale brain networks emerge from dynamic processing of musical timbre, key and rhythm. Neuroimage, Vol. 59, 4 (2012), 3677--3689.Google Scholar
Cross Ref
- David Radford Bakker and Frances Heritage Martin. 2015. Musical chords and emotion: Major and minor triads are processed for emotion. Cognitive, Affective, & Behavioral Neuroscience, Vol. 15, 1 (2015), 15--31.Google ScholarCross Ref
- K Bischoff, C S Firan, R Paiu, W Nejdl, C Laurier, and M Sordo. 2009. Music Mood and Theme Classification-a Hybrid Approach.. In ISMIR. 657--662.Google Scholar
- Abdelhamid Bouchachia and Saliha Bouchachia. 2008. Ensemble learning for time series prediction .na.Google Scholar
- Elvira Brattico, Vinoo Alluri, Brigitte Bogert, Thomas Jacobsen, Nuutti Vartiainen, Sirke Katriina Nieminen, and Mari Tervaniemi. 2011. A functional MRI study of happy and sad emotions in music with and without lyrics. Frontiers in psychology, Vol. 2 (2011), 308.Google Scholar
- Walter Bradford Cannon. 1916. Bodily changes in pain, hunger, fear, and rage: An account of recent researches into the function of emotional excitement. D. Appleton.Google Scholar
- Theodora Chaspari, Andreas Tsiartas, Leah I Stein, Sharon A Cermak, and Shrikanth S Narayanan. 2015. Sparse representation of electrodermal activity with knowledge-driven dictionaries. IEEE Transactions on Biomedical Engineering, Vol. 62, 3 (2015), 960--971.Google ScholarCross Ref
- Sheng Chen, XX Wang, and Chris J Harris. 2008. NARX-based nonlinear system identification using orthogonal least squares basis hunting. IEEE Transactions on Control Systems Technology, Vol. 16, 1 (2008), 78--84.Google ScholarCross Ref
- Ching-Hua Chuan and Dorien Herremans. 2018. Modeling temporal tonal relations in polyphonic music through deep networks with a novel image-based representation. In Thirty-second AAAI conference on artificial intelligence.Google Scholar
- TM Cover and JA Thomas. 2006. Elements of Information Theory 2nd edn (Hoboken, NJ, John Wiley & Sons). (2006).Google ScholarDigital Library
- Franz A de Leon and Kirk Martinez. 2014. Music genre classification using polyphonic timbre models. In 2014 19th International Conference on Digital Signal Processing. IEEE, 415--420.Google ScholarCross Ref
- FFmpeg Developers. 2019. ffmpeg Tool. http://ffmpeg.org/.Google Scholar
- Francesca R Dillman Carpentier and Robert F Potter. 2007. Effects of music on physiological arousal: Explorations into tempo and genre. Media Psychology, Vol. 10, 3 (2007), 339--363.Google ScholarCross Ref
- Douglas S Ellis and Gilbert Brighouse. 1952. Effects of music on respiration-and heart-rate. The American journal of psychology, Vol. 65, 1 (1952), 39--47.Google Scholar
- J Fan, K Tatar, M Thorogood, and P Pasquier. 2017. RANKING-BASED EMOTION RECOGNITION FOR EXPERIMENTAL MUSIC. In International Symposium on Music Information Retrieval .Google Scholar
- Jianyu Fan, Miles Thorogood, and Philippe Pasquier. 2016. Automatic soundscape affect recognition using a dimensional approach. Journal of the Audio Engineering Society, Vol. 64, 9 (2016), 646--653.Google ScholarCross Ref
- Roger Frigola, Yutian Chen, and Carl Edward Rasmussen. 2014. Variational Gaussian process state-space models. In Advances in neural information processing systems. 3680--3688.Google Scholar
- Enrico Glerean, Juha Salmi, Juha M Lahnakoski, Iiro P Jaaskelainen, and Mikko Sams. 2012. Functional magnetic resonance imaging phase synchronization as a measure of dynamic functional connectivity. Brain connectivity, Vol. 2, 2 (2012), 91--101.Google Scholar
- Timothy Greer, Karan Singla, Benjamin Ma, and Shrikanth Narayanan. 2019. Learning Shared Vector Representations of Lyrics and Chords in Music. In ICASSP 2019--2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 3951--3955.Google Scholar
- James D Hamilton. 1995. Time series analysis. Economic Theory. II, Princeton University Press, USA (1995), 625--630.Google Scholar
- B Han, S Rho, S Jun, and E Hwang. 2010. Music emotion classification and context-based music recommendation. Multimedia Tools and Applications, Vol. 47, 3 (01 May 2010), 433--460.Google Scholar
- Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation, Vol. 9, 8 (1997), 1735--1780.Google Scholar
- X Hu, J S Downie, and A F Ehmann. 2009. Lyric text mining in music mood classification. American music, Vol. 183, 5,049 (2009), 2--209.Google Scholar
- David Brian Huron. 2006. Sweet anticipation: Music and the psychology of expectation .MIT press.Google Scholar
- Stéphanie Khalfa, Peretz Isabelle, Blondin Jean-Pierre, and Robert Manon. 2002. Event-related skin conductance responses to musical emotions in humans. Neuroscience letters, Vol. 328, 2 (2002), 145--149.Google Scholar
- Y E Kim, E M Schmidt, R Migneco, B G Morton, P Richardson, J Scott, J A Speck, and D Turnbull. 2010. Music emotion recognition: A state of the art review. In Proc. ISMIR. Citeseer, 255--266.Google Scholar
- S Koelsch. 2005. Investigating emotion with music: neuroscientific approaches. Annals of the New York Academy of Sciences (2005), 412--418.Google Scholar
- Stefan Koelsch, Thomas Fritz, Katrin Schulze, David Alsop, and Gottfried Schlaug. 2005. Adults and children processing music: an fMRI study. Neuroimage, Vol. 25, 4 (2005), 1068--1076.Google ScholarCross Ref
- Naveen Kumar, Rahul Gupta, Tanaya Guha, Colin Vaz, Maarten Van Segbroeck, Jangwon Kim, and Shrikanth S Narayanan. 2014. Affective Feature Design and Predicting Continuous Affective Dimensions from Music.. In MediaEval. Citeseer.Google Scholar
- Guokun Lai, Wei-Cheng Chang, Yiming Yang, and Hanxiao Liu. 2018. Modeling long-and short-term temporal patterns with deep neural networks. In The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval. ACM, 95--104.Google ScholarDigital Library
- Olivier Lartillot, Tuomas Eerola, Petri Toiviainen, and Jose Fornari. 2008. Multi-Feature Modeling of Pulse Clarity: Design, Validation and Optimization.. In ISMIR. Citeseer, 521--526.Google Scholar
- Olivier Lartillot, Petri Toiviainen, and Tuomas Eerola. 2008. A matlab toolbox for music information retrieval. In Data analysis, machine learning and applications. Springer, 261--268.Google Scholar
- Lie Lu, Dan Liu, and Hong-Jiang Zhang. 2006. Automatic mood detection and tracking of music audio signals. IEEE Transactions on audio, speech, and language processing, Vol. 14, 1 (2006), 5--18.Google ScholarDigital Library
- Vinod Menon and Daniel J Levitin. 2005. The rewards of music listening: response and physiological connectivity of the mesolimbic system. Neuroimage, Vol. 28, 1 (2005), 175--184.Google ScholarCross Ref
- Panagiotis C Petrantonakis and Leontios J Hadjileontiadis. 2009. Emotion recognition from EEG using higher order crossings. IEEE Transactions on Information Technology in Biomedicine, Vol. 14, 2 (2009), 186--197.Google ScholarDigital Library
- Panagiotis C Petrantonakis and Leontios J Hadjileontiadis. 2010. Emotion recognition from brain signals using hybrid adaptive filtering and higher order crossings analysis. IEEE Transactions on affective computing, Vol. 1, 2 (2010), 81--97.Google ScholarDigital Library
- F Riganello, A Candelieri, M Quintieri, and G Dolce. 2010. Heart rate variability, emotions, and music. Journal of Psychophysiology (2010).Google Scholar
- F Riganello, M Quintieri, A Candelieri, D Conforti, and G Dolce. 2008. Heart rate response to music: an artificial intelligence study on healthy and traumatic brain-injured subjects. Journal of Psychophysiology, Vol. 22, 4 (2008), 166--174.Google ScholarCross Ref
- Srikanth Ryali, Kaustubh Supekar, Daniel A Abrams, and Vinod Menon. 2010. Sparse logistic regression for whole-brain classification of fMRI data. NeuroImage, Vol. 51, 2 (2010), 752--764.Google ScholarCross Ref
- Fabienne Samson, Thomas A Zeffiro, Alain Toussaint, and Pascal Belin. 2011. Stimulus complexity and categorical effects in human auditory cortex: an activation likelihood estimation meta-analysis. Frontiers in Psychology, Vol. 1 (2011), 241.Google ScholarCross Ref
- Kristina Schaaff and Marc TP Adam. 2013. Measuring emotional arousal for online applications: Evaluation of ultra-short term heart rate variability measures. In 2013 Humaine Association Conference on Affective Computing and Intelligent Interaction. IEEE, 362--368.Google ScholarDigital Library
- B Schuller, J Dorfner, and G Rigoll. 2010. Determination of nonprototypical valence and arousal in popular music: features and performances. EURASIP Journal on Audio, Speech, and Music Processing, Vol. 2010, 1 (2010), 735854.Google ScholarCross Ref
- David W Scott. 2015. Multivariate density estimation: theory, practice, and visualization .John Wiley & Sons.Google Scholar
- Fred Shaffer and JP Ginsberg. 2017. An overview of heart rate variability metrics and norms. Frontiers in public health, Vol. 5 (2017), 258.Google Scholar
- Shun-Yao Shih, Fan-Keng Sun, and Hung-yi Lee. 2018. Temporal Pattern Attention for Multivariate Time Series Forecasting. arXiv preprint arXiv:1809.04206 (2018).Google Scholar
- Kai Siedenburg, Charalampos Saitis, and Stephen McAdams. 2019. The Present, Past, and Future of Timbre Research. In Timbre: Acoustics, Perception, and Cognition. Springer, 1--19.Google Scholar
- Neomi Singer, Nori Jacoby, Tamar Lin, Gal Raz, Lavi Shpigelman, Gadi Gilam, Roni Y Granot, and Talma Hendler. 2016. Common modulation of limbic network activation underlies musical emotions as they unfold. Neuroimage, Vol. 141 (2016), 517--529.Google ScholarCross Ref
- Yading Song, Simon Dixon, and Marcus Pearce. 2012. A survey of music recommendation systems and future perspectives. In 9th International Symposium on Computer Music Modeling and Retrieval, Vol. 4.Google Scholar
- Andrea Stocco. 2014. Coordinate-Based Meta-Analysis of fMRI Studies with R. The R Journal, Vol. 2 (12 2014), 5--15. https://doi.org/10.32614/RJ-2014-020Google ScholarCross Ref
- Petri Toiviainen, Vinoo Alluri, Elvira Brattico, Mikkel Wallentin, and Peter Vuust. 2014. Capturing the musical brain with Lasso: Dynamic decoding of musical features from fMRI data. Neuroimage, Vol. 88 (2014), 170--180.Google ScholarCross Ref
- Konstantinos Trohidis, Grigorios Tsoumakas, George Kalliris, and Ioannis P Vlahavas. 2008a. Multi-label classification of music into emotions.. In ISMIR, Vol. 8. 325--330.Google Scholar
- K Trohidis, G Tsoumakas, G Kalliris, and I P Vlahavas. 2008b. Multi-Label Classification of Music into Emotions.. In ISMIR. 325--330.Google Scholar
- D Turnbull, L Barrington, D Torres, and G Lanckriet. 2008. Semantic annotation and retrieval of music and sound effects. IEEE Transactions on Audio, Speech, and Language Processing, Vol. 16, 2 (2008), 467--476.Google ScholarDigital Library
- Stephen M Wilson, Istvan Molnar-Szakacs, and Marco Iacoboni. 2007. Beyond superior temporal cortex: intersubject correlations in narrative speech comprehension. Cerebral cortex, Vol. 18, 1 (2007), 230--242.Google Scholar
- Z Xiao, E Dellandréa, W Dou, and L Chen. 2008. What is the Best Segment Duration for Music Mood Analysis ?. In International Workshop on Content-Based Multimedia Indexing, CBMI 2008. 17--24.Google ScholarCross Ref
Index Terms
- A Multimodal View into Music's Effect on Human Neural, Physiological, and Emotional Experience
Recommendations
iSargam: music notation representation for Indian Carnatic music
Indian classical music, including its two varieties, Carnatic and Hindustani music, has a rich music tradition and enjoys a wide audience from various parts of the world. The Carnatic music which is more popular in South India still continues to be ...
Neural discrimination of nonprototypical chords in music experts and laymen: An meg study
At the level of the auditory cortex, musicians discriminate pitch changes more accurately than nonmusicians. However, it is not agreed upon how sound familiarity and musical expertise interact in the formation of pitch-change discrimination skills, that ...
Emotional States Associated with Music: Classification, Prediction of Changes, and Consideration in Recommendation
We present several interrelated technical and empirical contributions to the problem of emotion-based music recommendation and show how they can be applied in a possible usage scenario. The contributions are (1) a new three-dimensional resonance-arousal-...
Comments