Stance classification of multi-perspective consumer health information

ABSTRACT

While search engines are effective in answering direct factual questions such as, 'What are the symptoms of a disease X?', they are not so effective in addressing complex consumer health queries, which do not have a single definitive answer, such as, 'Is treatment X effective for disease Y?'. Instead, the users are presented with a vast number of search results with often contradictory perspectives and no definitive answer. We denote such queries as Multi-Perspective Consumer Health Information (MPCHI) queries for which there is no single 'Yes or No' answer. While ascertaining the credibility of the claims requires domain expertise, an efficient categorization of the search results according to their stance (support or oppose) to the queries will help the searcher in decision making. Hence, this paper focuses on the problem of stance classification for MPCHI data at sentence level, presenting a new data set for MPCHI queries. Unlike typical debate or argumentative text, the linguistic characteristics of MPCHI is quite different, with extensive use of scientific formal language and absence of opinion bearing words. Hence, such inherently different characteristic of MPCHI text requires going beyond traditional Bag of Words (BoW) features for stance classification. Hence, we propose using a rich non-traditional set of features such as medical semantic relations, stance vectors, sentiment polarity, textual entailment, and study their impact on MPCHI stance classification using an SVM and a neural network classifier. We find that using novel non-traditional features improves MPCHI stance classification performance over traditional BoW model by 24% for the SVM classifier, and 44% for the neural network classifier respectively, for the best feature combination.

References

1. Ameeta Agrawal and Aijun An. 2012. Unsupervised emotion detection from text using semantic and syntactic relations. In Web Intelligence and Intelligent Agent Technology (WI-IAT), 2012 IEEE/WIC/ACM International Conferences on, Vol. 1. IEEE, 346--353.
2. Rakesh Agrawal, Sridhar Rajagopalan, Ramakrishnan Srikant, and Yirong Xu. 2003. Mining Newsgroups Using Networks Arising from Social Behavior. In Proceedings of the 12th International Conference on World Wide Web (WWW '03). ACM, New York, NY, USA, 529--535.
3. Pranav Anand, Marilyn Walker, Rob Abbott, Jean E. Fox Tree, Robeson Bowmani, and Michael Minor. 2011. Cats Rule and Dogs Drool!: Classifying Stance in Online Debate. In Proceedings of the 2Nd Workshop on Computational Approaches to Subjectivity and Sentiment Analysis (WASSA '11). Association for Computational Linguistics, Stroudsburg, PA, USA, 1--9. http://dl.acm.org/citation.cfm?id=2107653.2107654
4. Alexandra Balahur, Zornitsa Kozareva, and Andrés Montoyo. 2009. Determining the Polarity and Source of Opinions Expressed in Political Debates. In Proceedings of the 10th International Conference on Computational Linguistics and Intelligent Text Processing (CICLing '09). Springer-Verlag, Berlin, Heidelberg, 468--480.
5. O. Biran and O Rambow. 2011. Identifying Justifications in Written Dialogs,. In Proceedings of the 2011 IEEE Fifth International Conference on Semantic Computing.
6. Official Google Blog. 2015. Google Health Information Knowledge Graph. (2015). https://googleblog.blogspot.co.uk/2015/02/health-info-knowledge-graph.html
7. Clinton Burfoot, Steven Bird, and Timothy Baldwin. 2011. Collective Classification of Congressional Floor-debate Transcripts. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies - Volume 1 (HLT '11). Association for Computational Linguistics, Stroudsburg, PA, USA, 1506--1515. http://dl.acm.org/citation.cfm?id=2002472.2002655
8. Jacob Cohen. 1968. Weighted kappa: Nominal scale agreement provision for scaled disagreement or partial credit. Psychological bulletin 70, 4 (1968), 213.
9. Janez Demšar. 2006. Statistical comparisons of classifiers over multiple data sets. Journal of Machine learning research 7, Jan (2006), 1--30.
10. Shiri Dori-Hacohen. 2015. Controversy Detection and Stance Analysis. In Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR '15). ACM, New York, NY, USA, 1057--1057.
11. Shiri Dori-Hacohen and James Allan. 2013. Detecting controversy on the web. In Proceedings of the 22nd ACM international conference on Conference on information & knowledge management (CIKM '13). ACM, New York, NY, USA, 1845--1848.
12. Shiri Dori-Hacohen and James Allan. 2015. Automated Controversy Detection on the Web. In Proceedings of the 37th European Conference on Information Retrieval. 423--434. ECIR submission.
13. Shiri Dori-Hacohen, Elad Yom-Tov, and James Allan. 2015. Navigating Controversy as a Complex Search Task. In ECIR 15. Electronic proceedings only.
14. Adam Faulkner. 2012. Automated Classification of Stance in Student Essays: An Approach Using Stance Target Information and the Wikipedia Link-Based Measure. In Proceedings of the Twenty-Seventh International Artificial Intelligence Research Conference.
15. W. Ferreira and A Vlachos. 2016. Emergent: a novel data-set for stance classification.. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics.
16. Kazi Saidul Hasan and Vincent Ng. 2013. Stance Classification of Ideological Debates: Data, Models, Features, and Constraints. In Proceedings of the Sixth International Joint Conference on Natural Language Processing. 1348--1356.
17. Myungha Jang and James Allan. 2016. Improving Automated Controversy Detection on the Web. In Proceedings of the 39th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR '16). ACM, New York, NY, USA, 865--868.
18. Tony Mullen and Robert Malouf. 2008. Taking sides: User classification for informal online political discourse. Internet Research 18 (2008), 177--190.
19. Akiko Murakami and Rudy Raymond. 2010. Support or Oppose?: Classifying Positions in Online Debates from Reply Activities and Opinion Expressions. In Proceedings of the 23rd International Conference on Computational Linguistics: Posters (COLING '10). Association for Computational Linguistics, Stroudsburg, PA, USA, 869--875. http://dl.acm.org/citation.cfm?id=1944566.1944666
20. SEBASTIAN PAD?!, TAE-GIL NOH, ASHER STERN, RUI WANG, and ROBERTO ZANOLI. 2015. Design and realization of a modular architecture for textual entailment. Natural Language Engineering 21 (3 2015), 167--200. Issue 02.
21. Thomas C. Rindflesh and Marcelo Fiszman. 2003. The Interaction of Domain Knowledge and Linguistic Structure in Natural Language Processing: Interpreting Hypernymic Propositions in Biomedical Text. J. of Biomedical Informatics 36, 6 (Dec. 2003), 462--477.
22. Parinaz Sobhani, Diana Inkpen, and Stan Matwin. 2015. From argumentation mining to stance classification. NAACL HLT 2015 (2015).
23. Swapna Somasundaran and Janyce Wiebe. 2009. Recognizing Stances in Online Debates. In Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP: Volume 1 - Volume 1 (ACL '09). Association for Computational Linguistics, Stroudsburg, PA, USA, 226--234. http://dl.acm.org/citation.cfm?id=1687878.1687912
24. Swapna Somasundaran and Janyce Wiebe. 2010. Recognizing Stances in Ideological On-line Debates. In Proceedings of the NAACL HLT 2010 Workshop on Computational Approaches to Analysis and Generation of Emotion in Text (CAAGET '10). Association for Computational Linguistics, Stroudsburg, PA, USA, 116--124. http://dl.acm.org/citation.cfm?id=1860631.1860645
25. Dhanya Sridhar, Lise Getoor, and Marilyn Walker. 2014. Collective Stance Classification of Posts in Online Debate Forums. In ACL Joint Workshop on Social Dynamics and Personal Attributes in Social Media.
26. Matt Thomas, Bo Pang, and Lillian Lee. 2006. Get out the Vote: Determining Support or Opposition from Congressional Floor-debate Transcripts. In Proceedings of the 2006 Conference on Empirical Methods in Natural Language Processing (EMNLP '06). Association for Computational Linguistics, Stroudsburg, PA, USA, 327--335. http://dl.acm.org/citation.cfm?id=1610075.1610122
27. Marilyn A. Walker, Pranav Anand, Robert Abbott, and Ricky Grant. 2012. Stance Classification Using Dialogic Properties of Persuasion (NAACL HLT '12). Association for Computational Linguistics, Stroudsburg, PA, USA, 592--596. http://dl.acm.org/citation.cfm?id=2382029.2382124
28. Yi-Chia Wang and Carolyn P. Rosé. 2010. Making Conversational Structure Explicit: Identification of Initiation-response Pairs Within Online Discussions. In 2010 North American Chapter of the Association for Computational Linguistics (HLT '10). Association for Computational Linguistics, Stroudsburg, PA, USA, 673--676. http://dl.acm.org/citation.cfm?id=1857999.1858096
29. WebMD. 2016. WebMD information on link between vaccine and autism. (2016). http://www.webmd.com/children/vaccines/features/vaccine-linked-to-autism#1
30. Ainur Yessenalina, Yisong Yue, and Claire Cardie. 2010. Multi-level Structured Models for Document-level Sentiment Classification. In Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing (EMNLP '10). Association for Computational Linguistics, Stroudsburg, PA, USA, 1046--1056. http://dl.acm.org/citation.cfm?id=1870658.1870760
31. Elad Yom-Tov and Luis Fernández-Luque. 2014. Information is in the eye of the beholder: Seeking information on the MMR vaccine through an Internet search engine. In AMIA 2014, American Medical Informatics Association Annual Symposium, Washington, DC, USA, November 15-19, 2014. http://knowledge.amia.org/56638-amia-1.1540970/t-004-1.1544972/f-004-1.1544973/a-226-1.1544995/a-227-1.1544992
32. Charles Zaiontz. 2016. Real Statistics Using Excel. (2016). http://www.real-statistics.com/statistics-tables/wilcoxon-signed-ranks-table/