Content Attention Model for Aspect Based Sentiment Analysis

ABSTRACT

Aspect based sentiment classification is a crucial task for sentiment analysis. Recent advances in neural attention models demonstrate that they can be helpful in aspect based sentiment classification tasks, which can help identify the focus words in human. However, according to our empirical study, prevalent content attention mechanisms proposed for aspect based sentiment classification mostly focus on identifying the sentiment words or shifters, without considering the relevance of such words with respect to the given aspects in the sentence. Therefore, they are usually insufficient for dealing with multi-aspect sentences and the syntactically complex sentence structures. To solve this problem, we propose a novel content attention based aspect based sentiment classification model, with two attention enhancing mechanisms: sentence-level content attention mechanism is capable of capturing the important information about given aspects from a global perspective, whiles the context attention mechanism is responsible for simultaneously taking the order of the words and their correlations into account, by embedding them into a series of customized memories. Experimental results demonstrate that our model outperforms the state-of-the-art, in which the proposed mechanisms play a key role.

References

1. Yossi Adi, Einat Kermany, Yonatan Belinkov, Ofer Lavi, and Yoav Goldberg. 2017. Fine-grained analysis of sentence embeddings using auxiliary prediction tasks. In Proc. of the 5th ICLR. CoRR, Toulon, France.
2. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In Proc. of the 3rd International Conference on Learning Representations. CoRR, Scottsdale, USA.
3. Yoshua Bengio, Aaron Courville, and Pascal Vincent. 2013. Representation learning: A review and new perspectives. IEEE Transactions on Pattern Analysis and Machine Intelligence 35, 8 (2013), 1798--1828.
4. Peng Chen, Zhongqian Sun, Lidong Bing, and Wei Yang. 2017. Recurrent attention network on memory for aspect sentiment analysis. In Proc. of the 2017 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Copenhagen, Denmark, 463--472.
5. Zhiyuan Chen, Arjun Mukherjee, and Bing Liu. 2014. Aspect extraction with automated prior knowledge learning. In Proc. of the 52nd Annual Meeting of the Association for Computational Linguistics. ACL, Baltimore, Maryland, 347--358.
6. Kyunghyun Cho, Bart van Merriënboer, Dzmitry Bahdanau, and Yoshua Bengio. 2014. On the properties of neural machine translation: encoder-decoder ap- proaches. In Proc. of 8th Workshop on Syntax, Semantics and Structure in Statistical Translation. Association for Computational Linguistics, Doha, Qatar, 103--111.
7. Li Dong, Furu Wei, Chuanqi Tan, Duyu Tang, Ming Zhou, and Ke Xu. 2014. Adaptive recursive neural network for target-dependent twitter sentiment classification. In Proc. of the 52nd Annual Meeting of the Association for Computational Linguistics (Short Papers). ACL, Baltimore, Maryland, 49--54.
8. Alex Graves, Greg Wayne, Malcolm Reynolds, Tim Harley, Ivo Danihelka, Agnieszka Grabska-Barwi'ska, Sergio Gómez Colmenarejo, Edward Grefenstette, Tiago Ramalho, John Agapiou, Adrià Puigdomènech Badia, Karl Moritz Hermann, Yori Zwols, Georg Ostrovski, Adam Cain, Helen King, Christopher Summerfield, Phil Blunsom, Koray Kavukcuoglu, and Demis Hassabis. 2016. Hybrid computing using a neural network with dynamic external memory. Nature 538, 7626 (2016), 471--476.
9. Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. In Advances in Neural Information Processing Systems 28. Curran Associates, Inc., Montréal, Canada, 1693--1701.
10. Ozan Irsoy and Claire Cardie. 2014. Opinion mining with deep recurrent neural networks. In Proc. of the 2014 Conference on Empirical Methods in Natural Language Processing. ACL, Doha, Qatar, 720--728.
11. Long Jiang, Mo Yu, Ming Zhou, Xiaohua Liu, and Tiejun Zhao. 2011. Target-dependent twitter sentiment classification. In Proc. of the 49th Annual Meeting of the Association for Computational Linguistics. ACL, Portland, Oregon, USA, 151--160.
12. Yoon Kim, Carl Denton, Luong Hoang, and Alexander M Rush. 2017. Structured attention networks. In Proc. of the 5th International Conference on Learning Representations. CoRR, Toulon, France.
13. Svetlana Kiritchenko, Xiaodan Zhu, Colin Cherry, and Saif M. Mohammad. 2014. NRC-Canada-2014: Detecting aspects and sentiment in customer reviews. In Proc. of the 8th International Workshop on Semantic Evaluation. Dublin, Ireland, 437--442.
14. Ankit Kumar, Ozan Irsoy, Peter Ondruska, Mohit Iyyer, James Bradbury, Ishaan Gulrajani, Victor Zhong, Romain Paulus, and Richard Socher. 2016. Ask me anything: Dynamic memory networks for natural language processing. In Proc. of The 33rd International Conference on Machine Learning, Vol. 48. PMLR, New York, NY, USA, 1378--1387.
15. Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. 2015. Deep learning. Nature 521, 7553 (2015), 436--444.
16. Bing Liu. 2012. Sentiment analysis and opinion mining. Synthesis Lectures on Human Language Technologies 5, 1 (2012), 1--167.
17. Yue Lu and Chengxiang Zhai. 2008. Opinion integration through semi-supervised topic modeling. In Proc. of the 17th International Conference on World Wide Web. ACM, Beijing, China, 121--130.
18. Minh-Thang Luong, Hieu Pham, and Christopher D Manning. 2015. Effective approaches to attention-based neural machine translation. In Proc. of the 2015 Conference on Empirical Methods in Natural Language Processing. ACL, Lisbon, Portugal, 1412--1421.
19. Dehong Ma, Sujian Li, Xiaodong Zhang, and Houfeng Wang. 2017. Interactive attention networks for aspect-level sentiment classification. In Proc. of the Twenty- Sixth IJCAI. Melbourne, Australia, 4068--4074.
20. Qiaozhu Mei, Xu Ling, Matthew Wondra, Hang Su, and ChengXiang Zhai. 2007. Topic sentiment mixture: modeling facets and opinions in weblogs. In Proc. of the 16th international conference on World Wide Web. ACM, Banff, Alberta, Canada, 171--180.
21. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In Advances in Neural Information Processing Systems 26. Curran Associates, Inc., Lake Tahoe, USA, 3111--3119.
22. Samaneh Moghaddam and Martin Ester. 2013. The FLDA model for aspect-based opinion mining: addressing the cold start problem. In Proc. of the 22nd International Conference on WWW. ACM, Rio de Janeiro, Brazil, 909--918.
23. Thien Hai Nguyen and Kiyoaki Shirai. 2015. PhraseRNN: Phrase recursive neural network for aspect-based sentiment analysis. In Proc. of the 2015 Conference on Empirical Methods in Natural Language Processing. ACL, Lisbon, Portugal, 2509--2514.
24. Jeffrey Pennington, Richard Socher, and Christopher D Manning. 2014. Glove: Global vectors for word representation. In Proc. of the 2014 Conference on Empirical Methods in Natural Language Processing. ACL, Doha, Qatar, 1532--1543.
25. Maria Pontiki, Dimitris Galanis, John Pavlopoulos, Harris Papageorgiou, Ion Androutsopoulos, and Suresh Manandhar. 2014. Semeval-2014 task 4: Aspect based sentiment analysis. In Proc. of the 8th International Workshop on Semantic Evaluation. Dublin, Ireland, 27--35.
26. Soujanya Poria, Erik Cambria, and Alexander Gelbukh. 2016. Aspect extraction for opinion mining with a deep convolutional neural network. Knowledge-Based Systems 108, Supplement C (2016), 42--49.
27. Guang Qiu, Bing Liu, Jiajun Bu, and Chun Chen. 2011. Opinion word expansion and target extraction through double propagation. Computational Linguistics 37, 1 (2011), 9--27.
28. Tim Rocktäschel, Edward Grefenstette, Karl Moritz Hermann, Tomás Kociský, and Phil Blunsom. 2016. Reasoning about entailment with neural attention. In Proc. of the 4th ICLR. San Juan, Puerto Rico.
29. Sebastian Ruder, Parsa Ghaffari, and John G Breslin. 2016. A hierarchical model of reviews for aspect-based sentiment analysis. In Proc. of the 2016 Conference on Empirical Methods in Natural Language Processing. ACL, Austin, Texas, USA, 999--1005.
30. Alexander M. Rush, Sumit Chopra, and Jason Weston. 2015. A neural attention model for sentence summarization. In Proc. of the 2015 Conference on EMNLP. ACL, Lisbon, Portugal, 379--389.
31. Minjoon Seo, Aniruddha Kembhavi, Ali Farhadi, and Hannaneh Hajishirzi. 2017. Bi-directional attention flow for machine comprehension. In Proc. of the 5th International Conference on Learning Representations. CoRR, Toulon, France.
32. Richard Socher, Danqi Chen, Christopher D Manning, and Andrew Y. Ng. 2013. Reasoning with neural tensor networks for knowledge base completion. In NIPS 2013. Curran Associates, Inc., Lake Tahoe, USA, 926--934.
33. Kaisong Song, Ling Chen, Wei Gao, Shi Feng, Daling Wang, and Chengqi Zhang. 2016. PerSentiment: A personalized sentiment classification system for microblog users. In Proc. of the 25th International Conference Companion on World Wide Web. International World Wide Web Conferences Steering Committee, Montréal, Canada, 255--258.
34. Sainbayar Sukhbaatar, Arthur Szlam, Jason Weston, and Rob Fergus. 2015. End-to-end memory networks. In Advances in Neural Information Processing Systems 28. Curran Associates, Inc., Montréal, Canada, 2440--2448.
35. Yaming Sun, Lei Lin, Duyu Tang, Nan Yang, Zhenzhou Ji, and Xiaolong Wang. 2015. Modeling mention, context and entity with neural networks for entity disambiguation. In Proc. of the Twenty-Fourth International Joint Conference on Artificial Intelligence. Buenos Aires, Argentina, 1333--1339.
36. Duyu Tang, Bing Qin, Xiaocheng Feng, and Ting Liu. 2016. Effective LSTMs for target-dependent sentiment classification. In Proc. of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers. Osaka, Japan, 3298--3307.
37. Duyu Tang, Bing Qin, and Ting Liu. 2016. Aspect level sentiment classification with deep memory network. In Proc. of the 2016 Conference on Empirical Methods in Natural Language Processing. ACL, Austin, Texas, USA, 214--224.
38. Duyu Tang, Furu Wei, Nan Yang, Zhou Ming, Ting Liu, and Bing Qin. 2014. Learning sentiment-specific word embedding for twitter sentiment classification. In Proc. of the 52nd Annual Meeting of the Association for Computational Linguistics. ACL, Baltimore, Maryland, 1555--1565.
39. Duy-Tin Vo and Yue Zhang. 2015. Target-dependent twitter sentiment classification with rich automatic features. In Proc. of the 24th International Joint Conference on Artificial Intelligence. Buenos Aires, Argentina, 1347--1353.
40. Shuai Wang, Zhiyuan Chen, and Bing Liu. 2016. Mining aspect-specific opinion using a holistic lifelong topic model. In Proc. of the 25th International Conference on World Wide Web. International World Wide Web Conferences Steering Committee, Montréal, Canada, 167--176.
41. Yequan Wang, Minlie Huang, Xiaoyan Zhu, and Li Zhao. 2016. Attention-based LSTM for aspect-level sentiment classification. In Proc. of the 2016 Conference on Empirical Methods in Natural Language Processing. ACL, Austin, Texas, USA, 606--615.
42. Jason Weston, Sumit Chopra, and Antoine Bordes. 2015. Memory networks. In Proc. of the 3rd International Conference on Learning Representations. CoRR, Scottsdale, Arizona, USA.
43. Meishan Zhang, Yue Zhang, and Duy-Tin Vo. 2016. Gated neural networks for targeted sentiment analysis. In Proc. of the Thirtieth AAAI. AAAI Press, Phoenix, Arizona, USA, 3087--3093.

Index Terms

• Computing methodologies

  • Artificial intelligence

    • Natural language processing

      • Information extraction

  • Machine learning

    • Machine learning approaches

      • Neural networks

• Information systems

  • Information retrieval

    • Retrieval tasks and goals

      • Sentiment analysis