James J. Deng
Alfredo Milani
Skip Abstract Section
Abstract
We present several interrelated technical and empirical contributions to the problem of emotion-based music recommendation and show how they can be applied in a possible usage scenario. The contributions are (1) a new three-dimensional resonance-arousal-valence model for the representation of emotion expressed in music, together with methods for automatically classifying a piece of music in terms of this model, using robust regression methods applied to musical/acoustic features; (2) methods for predicting a listener’s emotional state on the assumption that the emotional state has been determined entirely by a sequence of pieces of music recently listened to, using conditional random fields and taking into account the decay of emotion intensity over time; and (3) a method for selecting a ranked list of pieces of music that match a particular emotional state, using a minimization iteration method. A series of experiments yield information about the validity of our operationalizations of these contributions. Throughout the article, we refer to an illustrative usage scenario in which all of these contributions can be exploited, where it is assumed that (1) a listener’s emotional state is being determined entirely by the music that he or she has been listening to and (2) the listener wants to hear additional music that matches his or her current emotional state. The contributions are intended to be useful in a variety of other scenarios as well.
- G. Adomavicius and A. Tuzhilin. 2005. Toward the next generation of recommender systems: A survey of the state-of-the-art and possible extensions. IEEE Transactions on Knowledge and Data Engineering 17, 6, 734--749. DOI:http://dx.doi.org/10.1109/TKDE.2005.99 Google Scholar
Digital Library
- F. Agrafioti, D. Hatzinakos, and A. K. Anderson. 2012. ECG pattern analysis for emotion detection. IEEE Transactions on Affective Computing 3, 1, 102--115. Google ScholarDigital Library
- D. L. Bartlett. 1996. Physiological responses to music and sound stimuli. In Handbook of Music Psychology, Vol. 2, D. Hodges (Ed.). Mmb Music, 343--85.Google Scholar
- E. Bigand, S. Vieillard, F. Madurell, J. Marozeau, and A. Dacquet. 2005. Multidimensional scaling of emotional responses to music: The effect of musical expertise and of the duration of the excerpts. Cognition and Emotion 19, 8, 1113--1139.Google ScholarCross Ref
- G. Borg. 1998. Borg’s Perceived Exertion and Pain Scales. Human Kinetics.Google Scholar
- C. C. Chang and C. J. Lin. 2011. LIBSVM: A library for support vector machines. ACM Transactions on Intelligent Systems and Technology 2, 3, 27. Google ScholarDigital Library
- H.-T. Cheng, Y.-H. Yang, Y.-C. Lin, I.-B. Liao, and H. H. Chen. 2008. Automatic chord recognition for music classification and retrieval. In Proceedings of the 2008 IEEE International Conference on Multimedia and Expo. 1505--1508.Google Scholar
- G. L. Collier. 2007. Beyond valence and activity in the emotional connotations of music. Psychology of Music 35, 1, 110--131.Google ScholarCross Ref
- E. Coutinho and A. Cangelosi. 2011. Musical emotions: Predicting second-by-second subjective feelings of emotion from low-level psychoacoustic features and physiological measurements. Emotion 11, 4, 921.Google ScholarCross Ref
- R. T. Dean and F. Bailes. 2010. Time series analysis as a method to examine acoustical influences on real-time perception of music. Empirical Musicology Review 5, 4, 152--175.Google ScholarCross Ref
- J. J. Deng and C. Leung. 2012. Emotion-based music recommendation using audio features and user playlist. In Proceedings of the 6th International Conference on New Trends in Information Science and Service Science and Data Mining (ISSDM). 796--801.Google Scholar
- J. J. Deng and C. H. C. Leung. 2013. Music retrieval in joint emotion space using audio features and emotional tags. In Advances in Multimedia Modeling. Lecture Notes in Computer Science, Vol. 7732. Springer, 524--534.Google ScholarCross Ref
- J. Dias and A. Paiva. 2005. Feeling and reasoning: A computational model for emotional characters. In Proceedings of the 12th Portuguese Conference on Progress in Artificial Intelligence. 127--140. DOI:http://dx.doi.org/10.1007/11595014_13 Google ScholarDigital Library
- T. Eerola, O. Lartillot, and P. Toiviainen. 2009. Prediction of multidimensional emotional ratings in music from audio using multivariate regression models. In Proceedings of the International Conference on Music Information Retrieval. 621--626.Google Scholar
- T. Eerola and J. K. Vuoskoski. 2010. A comparison of the discrete and dimensional models of emotion in music. Psychology of Music 39, 1, 18--49. http://pom.sagepub.com/cgi/doi/10.1177/0305735610362821Google ScholarCross Ref
- P. Ekman. 1994. Moods, emotions, and traits. In The Nature of Emotions: Fundamental Questions, P. Eckman and R. J. Davidson (Eds). Oxford University Press, 56--58.Google Scholar
- J. R. J. Fontaine, K. R. Scherer, E. B. Roesch, and P. C. Ellsworth. 2007. The world of emotions is not two-dimensional. Psychological Science 18, 12, 1050--1057.Google ScholarCross Ref
- P. Gebhard. 2005. ALMA: A layered model of affect. In Proceedings of the 4th International Joint Conference on Autonomous Agents and Multiagent Systems. 29--36. DOI:http://dx.doi.org/10.1145/1082473.1082478 Google ScholarDigital Library
- H. Gunes, B. Schuller, M. Pantic, and R. Cowie. 2011. Emotion representation, analysis and synthesis in continuous space: A survey. In Proceedings of the IEEE International Conference on Automatic Face and Gesture Recognition and Workshops (FG ’11). 827--834.Google Scholar
- B. Han, S. Rho, R. B. Dannenberg, and E. Hwang. 2009. SMERS: Music emotion recognition using support vector regression.. In Proceedings of the 10th International Society for Music Information Retrieval Conference. 651--656. http://dblp.uni-trier.de/db/conf/ismir/ismir2009.html#HanRDH09.Google Scholar
- B. Han, S. Rho, S. Jun, and E. Hwang. 2010. Music emotion classification and context-based music recommendation. Multimedia Tools and Applications 47, 3, 433--460. Google ScholarDigital Library
- A. Hanjalic. 2006. Extracting moods from pictures and sounds: Towards truly personalized TV. IEEE Signal Processing Magazine 23, 2, 90--100.Google ScholarCross Ref
- J. Healey, R. Picard, and F. Dabek. 1998. A new affect-perceiving interface and its application to personalized music selection. In Proceedings of the 1998 Workshop on Perceptual User Interfaces. 4--6.Google Scholar
- K. Hevner. 1936. Experimental studies of the elements of expression in music. American Journal of Psychology 48, 2, 246--268.Google ScholarCross Ref
- X. Hu, J. S. Downie, and A. F. Ehmann. 2009. Lyric text mining in music mood classification. American Music 183, 5,049, 2--209.Google Scholar
- G. Ilie and W. F. Thompson. 2006. A comparison of acoustic cues in music and speech for three dimensions of affect. Music Perception 23, 4, 319--330.Google ScholarCross Ref
- C. E. Izard and C. Z. Malatesta. 1987. Perspectives on emotional development I: Differential emotions theory of early emotional development. In Handbook of Infant Development (2nd ed.), J. D. Osofsky (Ed.). Wiley, 494--554.Google Scholar
- J. H. Janssen, E. L. van den Broek, and J. H. D. M. Westerink. 2009. Personalized affective music player. In Proceedings of the 3rd International Conference on Affective Computing and Intelligent Interaction and Workshops (ACII ’09). 1--6.Google Scholar
- J. H. Janssen, E. L. van den Broek, and J. H. D. M. Westerink. 2012. Tune in to your emotions: A robust personalized affective music player. User Modeling and User-Adapted Interaction 22, 3, 255--279. Google ScholarDigital Library
- S. Jun, S. Rho, and E. Hwang. 2010. Music retrieval and recommendation scheme based on varying mood sequences. International Journal on Semantic Web and Information Systems 6, 2, 1--16. Google ScholarDigital Library
- P. N. Juslin and P. Laukka. 2004. Expression, perception, and induction of musical emotions: A review and a questionnaire study of everyday listening. Journal of New Music Research 33, 3, 217--238.Google ScholarCross Ref
- P. N. Juslin and J. A. Sloboda. 2001. Music and Emotion: Theory and Research, Vol. 20. Oxford University Press. http://nursinglibrary.org/Portal/main.aspx?pageid=36&sku==80380&ProductPrice==89.5000Google Scholar
- J. Kim. 2008. Emotion recognition based on physiological changes in music listening. IEEE Transactions on Pattern Analysis and Machine Intelligence 30, 12, 2067--2083. Google ScholarDigital Library
- Y. E. Kim, E. M. Schmidt, R. Migneco, B. G. Morton, P. Richardson, J. Scott, J. A. Speck, and D. Turnbull. 2010. Music emotion recognition: A state of the art review. In Proceedings of the International Conference on Music Information Retrieval. 255--266.Google Scholar
- V. J. Konečni. 2008. Does music induce emotion? A theoretical and methodological analysis. Psychology of Aesthetics, Creativity, and the Arts 2, 2, 115.Google ScholarCross Ref
- C. L. Krumhansl. 2002. Music: A link between cognition and emotion. Current Directions in Psychological Science 11, 2, 45--50.Google ScholarCross Ref
- F. F. Kuo, M. F. Chiang, M. K. Shan, and S. Y. Lee. 2005. Emotion-based music recommendation by association discovery from film music. In Proceedings of the 13th Annual ACM International Conference on Multimedia. 507--510. Google ScholarDigital Library
- J. D. Lafferty, A. McCallum, and F. C. N. Pereira. 2001. Conditional random fields: Probabilistic models for segmenting and labeling sequence data. In Proceedings of the 18th International Conference on Machine Learning. 282--289. http://dl.acm.org/citation.cfm?id=645530.655813 Google ScholarDigital Library
- O. Lartillot and P. Toiviainen. 2007. A Matlab toolbox for musical feature extraction from audio. In Proceedings of the 10th International Conference on Digital Audio Effects. 237--244.Google Scholar
- R. W. Levenson. 1988. Emotion and the autonomic nervous system: A prospectus for research on autonomic specificity. In Social Psychophysiology and Emotion: Theory and Clinical Applications, H. Wagner (Ed.). John Wiley and Sons, 17--42.Google Scholar
- Y. P. Lin, C. H. Wang, T. P. Jung, T. L. Wu, S. K. Jeng, J. R. Duann, and J. H. Chen. 2010. EEG-based emotion recognition in music listening. IEEE Transactions on Biomedical Engineering 57, 7, 1798--1806.Google ScholarCross Ref
- C. C. Lu and V. S. Tseng. 2009. A novel method for personalized music recommendation. Expert Systems with Applications 36, 6, 10035--10044. Google ScholarDigital Library
- L. Lu, D. Liu, and H. J. Zhang. 2006. Automatic mood detection and tracking of music audio signals. IEEE Transactions on Audio, Speech, and Language Processing 14, 1, 5--18. Google ScholarDigital Library
- K. F. MacDorman and S. O. C. C. Ho. 2007. Automatic emotion prediction of song excerpts: Index construction, algorithm design, and empirical comparison. Journal of New Music Research 36, 4, 281--299.Google ScholarCross Ref
- J. P. Magalhães and W. B. de Haas. 2011. Functional modelling of musical harmony: An experience report. In Proceedings of the 16th International Conference on Functional Programming. 156--162. Google ScholarDigital Library
- S. Marsella, J. Gratch, and P. Petta. 2010. Computational models of emotion. In A Blueprint for Affective Computing: A Sourcebook and Manual, K. R. Scherer, T. Banziger, and E. Roesch (Eds.). Oxford University Press, 21--46.Google Scholar
- A. Mehrabian. 1980. Basic Dimensions for a General Psychological Theory: Implications for Personality, Social, Environmental, and Developmental Studies. Oelgeschlager, Gunn, and Hain, Cambridge, MA.Google Scholar
- A. Mehrabian. 1996. Pleasure-arousal-dominance: A general framework for describing and measuring individual differences in temperament. Current Psychology 14, 4, 261--292.Google ScholarCross Ref
- L. B. Meyer. 1956. Emotion and Meaning in Music. University of Chicago Press.Google Scholar
- M. Muller and S. Ewert. 2010. Towards timbre-invariant audio features for harmony-based music. IEEE Transactions on Audio, Speech, and Language Processing 18, 3, 649--662. Google ScholarDigital Library
- N. Orio. 2006. Music Retrieval: A Tutorial and Review, Vol. 1. Now Pub. Google ScholarDigital Library
- A. Ortony, G. L. Clore, and A. Collins. 1990. The Cognitive Structure of Emotions. Cambridge University Press.Google Scholar
- R. W. Picard. 1997. Affective Computing. MIT Press, Cambridge, MA, 167--170. Google ScholarDigital Library
- R. W. Picard, E. Vyzas, and J. Healey. 2001. Toward machine emotional intelligence: Analysis of affective physiological state. IEEE Transactions on Pattern Analysis and Machine Intelligence 23, 10, 1175--1191. Google ScholarDigital Library
- T. Pohle, E. Pampalk, and G. Widmer. 2005. Evaluation of frequently used audio features for classification of music into perceptual categories. In Proceedings of the International Workshop on Content-Based Multimedia Indexing.Google Scholar
- P. Pudil, F. J. Ferri, J. Novovicova, and J. Kittler. 1994. Floating search methods for feature selection with nonmonotonic criterion functions. In Proceedings of the 12th International Conference on Pattern Recognition. 279--283.Google Scholar
- J. A. Russell. 1980. A circumplex model of affect. Journal of Personality and Social Psychology 39, 6, 1161.Google ScholarCross Ref
- K. R. Scherer. 2004. Which emotions can be induced by music? What are the underlying mechanisms? And how can we measure them? Journal of New Music Research 33, 3, 239--251.Google Scholar
- K. R. Scherer and J. S. Oshinsky. 1977. Cue utilization in emotion attribution from auditory stimuli. Motivation and Emotion 1, 4, 331--346.Google ScholarCross Ref
- K. R. Scherer. 1993. Neuroscience projections to current debates in emotion psychology. Cognition and Emotion 7, 1, 1--41.Google ScholarCross Ref
- K. R. Scherer, M. R. Zentner, and A. Schacht. 2002. Emotional states generated by music: An exploratory study of music experts. Musicae Scientiae 5, 1, 149--171.Google ScholarCross Ref
- U. Schimmack and R. Rainer. 2002. Experiencing activation: Energetic arousal and tense arousal are not mixtures of valence and activation. Emotion 2, 4, 412.Google ScholarCross Ref
- E. M. Schmidt and Y. E. Kim. 2010. Prediction of time-varying musical mood distributions from audio. In Proceedings of the 11th International Society for Music Information Retrieval Conference. 465--470.Google Scholar
- E. M. Schmidt and Y. E. Kim. 2011. Modeling musical emotion dynamics with conditional random fields. In Proceedings of the 12th International Society for Music Information Retrieval Conference. 777--782.Google Scholar
- E. M. Schmidt, D. Turnbull, and Y. E. Kim. 2010. Feature selection for content-based, time-varying musical emotion regression. In Proceedings of the International Conference on Multimedia Information Retrieval. 267--274. Google ScholarDigital Library
- E. Schubert. 1999. Measuring emotion continuously: Validity and reliability of the two-dimensional emotion-space. Australian Journal of Psychology 51, 3, 154--165.Google ScholarCross Ref
- E. Schubert. 2004. Modeling perceived emotion with continuous musical features. Music Perception 21, 4, 561--585.Google ScholarCross Ref
- C. E. Seashore. 1923. Measurements on the expression of emotion in music. Proceedings of the National Academy of Sciences of the United States of America 9, 9, 323.Google ScholarCross Ref
- M. C. Sezgin, B. Gunsel, and G. Karabulut Kurt. 2012. Perceptual audio features for emotion detection. EURASIP Journal on Audio, Speech, and Music Processing 2012, 1, 16.Google ScholarCross Ref
- B. R. Steunebrink, M. Dastani, and J.-J. C. Meyer. 2008. A formal model of emotions: Integrating qualitative and quantitative aspects. In Proceedings of the 18th European Conference on Artificial Intelligence. 256--260. http://dl.acm.org/citation.cfm?id=1567281.1567340 Google ScholarDigital Library
- J. M. Talarico, K. S. LaBar, and D. C. Rubin. 1994. Emotional intensity predicts autobiographical memory experience. Memory and Cognition 32, 7, 1118--1132.Google ScholarCross Ref
- R. E. Thayer. 1989. The Biopsychology of Mood and Arousal. Oxford University Press.Google Scholar
- M. E. Tipping. 2001. Sparse Bayesian learning and the relevance vector machine. Journal of Machine Learning Research 1, 211--244. Google ScholarDigital Library
- M. Tkalcic, A. Kosir, and J. Tasic. 2011. Affective recommender systems: The role of emotions in recommender systems. In Proceedings of the RecSys 2011 Workshop on Human Decision Making in Recommender Systems. 9--13.Google Scholar
- G. Tzanetakis and P. Cook. 2000. Marsyas: A framework for audio analysis. Organised Sound 4, 3, 169--175. Google ScholarDigital Library
- G. D. Webster and C. G. Weir. 2005. Emotional responses to music: Interactive effects of mode, texture, and tempo. Motivation and Emotion 29, 1, 19--39.Google ScholarCross Ref
- G. Wijnalda, S. Pauws, F. Vignoli, and H. Stuckenschmidt. 2005. A personalized music system for motivation in sport performance. Pervasive Computing 4, 3, 26--32. Google ScholarDigital Library
- S. Yan, D. Xu, B. Zhang, H. J. Zhang, Q. Yang, and S. Lin. 2007. Graph embedding and extensions: A general framework for dimensionality reduction. IEEE Transactions on Pattern Analysis and Machine Intelligence 29, 1, 40--51. Google ScholarDigital Library
- Y. H. Yang and H. H. Chen. 2011. Ranking-based emotion recognition for music organization and retrieval. IEEE Transactions on Audio, Speech, and Language Processing 19, 4, 762--774. Google ScholarDigital Library
- Y. H. Yang, Y. C. Lin, Y. F. Su, and H. H. Chen. 2008. A regression approach to music emotion recognition. IEEE Transactions on Audio, Speech, and Language Processing 16, 2, 448--457. Google ScholarDigital Library
- M. S. M. Yik, J. A. Russell, and L. F. Barrett. 1999. Structure of self-reported current affect: Integration and beyond. Journal of Personality and Social Psychology 77, 3, 600.Google ScholarCross Ref
- M. Zentner, D. Grandjean, and K. R. Scherer. 2008. Emotions evoked by the sound of music: Characterization, classification, and measurement. Emotion 8, 4, 494.Google ScholarCross Ref
Index Terms
- Emotional States Associated with Music: Classification, Prediction of Changes, and Consideration in Recommendation
Recommendations
An algorithm for music recommendation based on the user's musical preferences and desired emotions
MUM '18: Proceedings of the 17th International Conference on Mobile and Ubiquitous MultimediaStudies show that music is an effective form of emotional induction and can change the emotional behavior of the user. In view of this, this article seeks to create a "selection" algorithm that includes a subset of songs chosen by the user, while taking ...
Modeling Affective Responses to Music Using Audio Signal Analysis and Physiology
Music, Mind, and EmbodimentAbstractA key issue in designing personalized music affective applications is to find effective ways to direct emotion by music selection with appropriate combination of acoustic features. The aim of this study is to understand the dynamic relationships ...
Are you emotional or depressed? Learning about your emotional state from your music using machine learning
Music plays an important role in our society and has applications broader than just entertainment and pleasure due to its social and physiological effects. There has been recent interest in music, and two active research topics are music information ...
Comments