Learning word vectors for sentiment analysis

ABSTRACT

Unsupervised vector-based approaches to semantics can model rich lexical meanings, but they largely fail to capture sentiment information that is central to many word meanings and important for a wide range of NLP tasks. We present a model that uses a mix of unsupervised and supervised techniques to learn word vectors capturing semantic term--document information as well as rich sentiment content. The proposed model can leverage both continuous and multi-dimensional sentiment information as well as non-sentiment annotations. We instantiate the model to utilize the document-level sentiment polarity annotations present in many online documents (e.g. star ratings). We evaluate the model using small, widely used sentiment and subjectivity corpora and find it out-performs several previously introduced methods for sentiment classification. We also introduce a large dataset of movie reviews to serve as a more robust benchmark for work in this area.

References

1. C. O. Alm, D. Roth, and R. Sproat. 2005. Emotions from text: machine learning for text-based emotion prediction. In Proceedings of HLT/EMNLP, pages 579--586.
2. A. Andreevskaia and S. Bergler. 2006. Mining Word-Net for fuzzy sentiment: sentiment tag extraction from WordNet glosses. In Proceedings of the European ACL, pages 209--216.
3. Y. Bengio, R. Ducharme, P. Vincent, and C. Jauvin. 2003. a neural probabilistic language model. Journal of Machine Learning Research, 3:1137--1155, August.
4. D. M. Blei, A. Y. Ng, and M. I. Jordan. 2003. Latent dirichlet allocation. Journal of Machine Learning Research, 3:993--1022, May.
5. J. Boyd-Graber and P. Resnik. 2010. Holistic sentiment analysis across languages: multilingual supervised latent Dirichlet allocation. In Proceedings of EMNLP, pages 45--55.
6. R. Collobert and J. Weston. 2008. A unified architecture for natural language processing. In Proceedings of the ICML, pages 160--167.
7. S. Deerwester, S. T. Dumais, G. W. Furnas, T. K. Landauer, and R. Harshman. 1990. Indexing by latent semantic analysis. Journal of the American Society for Information Science, 41:391--407, September.
8. R. E. Fan, K. W. Chang, C. J. Hsieh, X. R. Wang, and C. J. Lin. 2008. LIBLINEAR: A library for large linear classification. The Journal of Machine Learning Research, 9:1871--1874, August.
9. J. R. Finkel and C. D. Manning. 2009. Joint parsing and named entity recognition. In Proceedings of NAACL, pages 326--334.
10. A. B. Goldberg and J. Zhu. 2006. Seeing stars when there aren't many stars: graph-based semi-supervised learning for sentiment categorization. In TextGraphs: HLT/NAACL Workshop on Graph-based Algorithms for Natural Language Processing, pages 45--52.
11. T. Jay. 2000. Why We Curse: A Neuro-Psycho-Social Theory of Speech. John Benjamins, Philadelphia/Amsterdam.
12. D. Kaplan. 1999. What is meaning? Explorations in the theory of Meaning as Use. Brief version --- draft 1. Ms., UCLA.
13. A. Kennedy and D. Inkpen. 2006. Sentiment classification of movie reviews using contextual valence shifters. Computational Intelligence, 22:110--125, May.
14. F. Li, M. Huang, and X. Zhu. 2010. Sentiment analysis with global topics and local dependency. In Proceedings of AAAI, pages 1371--1376.
15. C. Lin and Y. He. 2009. Joint sentiment/topic model for sentiment analysis. In Proceeding of the 18th ACM Conference on Information and Knowledge Management, pages 375--384.
16. J. Martineau and T. Finin. 2009. Delta tfidf: an improved feature space for sentiment analysis. In Proceedings of the 3rd AAAI International Conference on Weblogs and Social Media, pages 258--261.
17. A. Mnih and G. E. Hinton. 2007. Three new graphical models for statistical language modelling. In Proceedings of the ICML, pages 641--648.
18. G. Paltoglou and M. Thelwall. 2010. A study of information retrieval weighting schemes for sentiment analysis. In Proceedings of the ACL, pages 1386--1395.
19. B. Pang and L. Lee. 2004. A sentimental education: sentiment analysis using subjectivity summarization based on minimum cuts. In Proceedings of the ACL, pages 271--278.
20. B. Pang and L. Lee. 2005. Seeing stars: exploiting class relationships for sentiment categorization with respect to rating scales. In Proceedings of ACL, pages 115--124.
21. B. Pang, L. Lee, and S. Vaithyanathan. 2002. Thumbs up? sentiment classification using machine learning techniques. In Proceedings of EMNLP, pages 79--86.
22. C. Potts. 2007. The expressive dimension. Theoretical Linguistics, 33:165--197.
23. B. Snyder and R. Barzilay. 2007. Multiple aspect ranking using the good grief algorithm. In Proceedings of NAACL, pages 300--307.
24. M. Steyvers and T. L. Griffiths. 2006. Probabilistic topic models. In T. Landauer, D McNamara, S. Dennis, and W. Kintsch, editors, Latent Semantic Analysis: A Road to Meaning.
25. J. Turian, L. Ratinov, and Y. Bengio. 2010. Word representations: A simple and general method for semi-supervised learning. In Proceedings of the ACL, page 384394.
26. P. D. Turney and P. Pantel. 2010. From frequency to meaning: vector space models of semantics. Journal of Artificial Intelligence Research, 37:141--188.
27. H. Wallach, D. Mimno, and A. McCallum. 2009. Rethinking LDA: why priors matter. In Proceedings of NIPS, pages 1973--1981.
28. C. Whitelaw, N. Garg, and S. Argamon. 2005. Using appraisal groups for sentiment analysis. In Proceedings of CIKM, pages 625--631.
29. T. Wilson, J. Wiebe, and R. Hwa. 2004. Just how mad are you? Finding strong and weak opinion clauses. In Proceedings of AAAI, pages 761--769.

Index Terms

• Applied computing

  • Arts and humanities

    • Language translation

• Computing methodologies

  • Artificial intelligence

    • Natural language processing

  • Machine learning