Amy X. Zhang
Scott Appling
David Karger
ABSTRACT
The proliferation of misinformation in online news and its amplification by platforms are a growing concern, leading to numerous efforts to improve the detection of and response to misinformation. Given the variety of approaches, collective agreement on the indicators that signify credible content could allow for greater collaboration and data-sharing across initiatives. In this paper, we present an initial set of indicators for article credibility defined by a diverse coalition of experts. These indicators originate from both within an article's text as well as from external sources or article metadata. As a proof-of-concept, we present a dataset of 40 articles of varying credibility annotated with our indicators by 6 trained annotators using specialized platforms. We discuss future steps including expanding annotation, broadening the set of indicators, and considering their use by platforms and the public, towards the development of interoperable standards for content credibility.
- Jason Abbruzzese. 2017. Facebook is going to do something about those terrible ads on your website. (May 2017). http://mashable.com/2017/05/10/facebookcrackdown-bad-ads-news-feed/Google Scholar
- Ahmer Arif, John J Robinson, Stephanie A Stanek, Elodie S Fichet, Paul Townsend, Zena Worku, and Kate Starbird. 2017. A Closer Look at the Self-Correcting Crowd: Examining Corrections in Online Rumors. In Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing. ACM, ACM, 2 Penn Plaza, Suite 701, New York, NY 10121-0701., 155--168. Google ScholarDigital Library
- Ahmer Arif, Kelley Shanahan, Fang-Ju Chou, Yoanna Dosouto, Kate Starbird, and Emma S Spiro. 2016. How information snowballs: Exploring the role of exposure in online rumor propagation. In Proceedings of the 19th ACM Conference on Computer-Supported Cooperative Work & Social Computing. ACM, ACM, 2 Penn Plaza, Suite 701, New York, NY 10121-0701., 466--477. Google ScholarDigital Library
- Leticia Bode and Emily K Vraga. 2015. In related news, that was wrong: The correction of misinformation through related stories functionality in social media. Journal of Communication 65, 4 (2015), 619--638.Google ScholarCross Ref
- David B. Buller and Judee K. Burgoon. 1996. Interpersonal Deception Theory. Communication Theory 6, 3 (1996), 203--242.Google ScholarCross Ref
- Carlos Castillo, Marcelo Mendoza, and Barbara Poblete. 2011. Information credibility on twitter. In Proceedings of the 20th international conference on World wide web. ACM, ACM, 2 Penn Plaza, Suite 701, New York, NY 10121-0701., 675--684. Google ScholarDigital Library
- Michael A. Caulfield. 2017. Go Upstream to the Find the Source. (Jan 2017). https://webliteracy.pressbooks.com/chapter/go-upstream-to-find-the-source/Google Scholar
- Yimin Chen, Niall J Conroy, and Victoria L Rubin. 2015. Misleading online content: Recognizing clickbait as false news. In Proceedings of the 2015 ACM on Workshop on Multimodal Deception Detection. ACM, ACM, 2 Penn Plaza, Suite 701, New York, NY 10121-0701., 15--19. Google ScholarDigital Library
- Giovanni Luca Ciampaglia, Prashant Shiralkar, Luis M. Rocha, Johan Bollen, Filippo Menczer, and Alessandro Flammini. 2015. Computational Fact Checking from Knowledge Networks. PLOS ONE 10, 6 (06 2015), 1--13.Google Scholar
- Andrew J Flanagin and Miriam J Metzger. 2000. Perceptions of Internet information credibility. Journalism & Mass Communication Quarterly 77, 3 (2000), 515--540.Google ScholarCross Ref
- William B. Frakes. 1986. Information and misinformation: An investigation of the notions of information, misinformation, informing, and misinforming. Journal of the American Society for Information Science 37, 1 (1986), 48--49.Google Scholar
- William K. Frankena. 1939. The naturalistic fallacy. Mind 48, 192, Article 4 (1939), 13 pages.Google Scholar
- Daniel Funke. 2017. It's been a year since Facebook partnered with factcheckers. How's it going (Dec. 2017). Retrieved January 5, 2018 from https: //www.poynter.org/news/its-been-year- facebook-partnered- fact-checkershows-it-goingGoogle Scholar
- Cecilie Gaziano and Kristin McGrath. 1986. Measuring the concept of credibility. Journalism quarterly 63, 3 (1986), 451--462.Google ScholarCross Ref
- lucas graves and Tom glaisyer. 2012. The Fact-Checking Universe in Spring 2012: An Overview. The New America Foundation, Washington, DC, USA. https: //www.issuelab.org/resource/the-fact-checking-universe-in-spring-2012-anoverview.htmlGoogle Scholar
- Jennifer D. Greer. 2003. Evaluating the Credibility of Online Information: A Test of Source and Advertising Influence. Mass Communication and Society 6, 1 (2003), 11--28.Google ScholarCross Ref
- Manish Gupta, Peixiang Zhao, and Jiawei Han. 2012. Evaluating event credibility on twitter. In Proceedings of the 2012 SIAM International Conference on Data Mining. SIAM, SIAM, 3600 Market Street, 6th Floor | Philadelphia, PA 19104--2688 USA, 153--164.Google ScholarCross Ref
- John B. Horrigan and John Gramlich. 2017. Many Americans, especially blacks and Hispanics, are hungry for help as they sort through information. (Nov 2017). http://www.pewresearch.org/fact- tank/2017/11/29/many-americansespecially-blacks-and-hispanics-are-hungry- for-help-as- they-sort- throughinformation/Google Scholar
- IREX.org. 2017. Ukrainians' self-defense against disinformation: What we learned from Learn to Discern. (June 2017). Retrieved January 5, 2018 from https:// www.irex.org/insight/ukrainians-self-defense-against-disinformation-whatwe-learned-learn-discernGoogle Scholar
- Alice Marwick and Rebecca Lewis. 2017. Media Manipulation and Disinformation Online. Report. Data & Society Research Institute. https://edoc.coe.int/en/mediafreedom/7495-information-disorder-toward-an-interdisciplinary-frameworkfor-research-and-policy-making.htmlGoogle Scholar
- Aaron M. McCright and Riley E. Dunlap. 2011. The Politicization of Climate Change and Polarization in the American Public's Views of Global Warming, 2001--2010. Sociological Quarterly 52, 2 (2011), 155--194.Google ScholarCross Ref
- Marcelo Mendoza, Barbara Poblete, and Carlos Castillo. 2010. Twitter Under Crisis: Can we trust what we RT. In Proceedings of the first workshop on social media analytics. ACM, ACM, 2 Penn Plaza, Suite 701, New York, NY 10121-0701., 71--79. Google ScholarDigital Library
- Panagiotis Takas Metaxas, Samantha Finn, and Eni Mustafaraj. 2015. Using twittertrails.com to investigate rumor propagation. In Proceedings of the 18th ACM Conference Companion on Computer Supported Cooperative Work & Social Computing. ACM, ACM, 2 Penn Plaza, Suite 701, New York, NY 10121-0701., 69--72. Google ScholarDigital Library
- Miriam J. Metzger, Andrew J. Flanagin, Keren Eyal, Daisy R. Lemus, and Robert M. Mccann. 2003. Credibility for the 21st Century: Integrating Perspectives on Source, Message, and Media Credibility in the Contemporary Media Environment. Annals of the International Communication Association 27, 1 (2003), 293--335.Google ScholarCross Ref
- Hans K Meyer, Doreen Marchionni, and Esther Thorson. 2010. The journalist behind the news: credibility of straight, collaborative, opinionated, and blogged --news--. American Behavioral Scientist 54, 2 (2010), 100--119.Google ScholarCross Ref
- Tanushree Mitra and Eric Gilbert. 2015. CREDBANK: A Large-Scale Social Media Corpus With Associated Credibility Annotations. In Ninth International AAAI Conference on Web and Social Media. ACM, ACM, 2 Penn Plaza, Suite 701, New York, NY 10121-0701., Article 10582, 10 pages.Google Scholar
- Meredith Ringel Morris, Scott Counts, Asta Roseway, Aaron Hoff, and Julia Schwarz. 2012. Tweeting is believing: understanding microblog credibility perceptions. In Proceedings of the ACM 2012 conference on Computer Supported Cooperative Work. ACM, ACM, 2 Penn Plaza, Suite 701, New York, NY 10121-0701., 441--450. Google ScholarDigital Library
- Lucia Moses. 2016. The underbelly of the internet: How content ad networks fund fake news. (Nov. 2016). Retrieved January 5, 2018 from https://digiday.com/ media/underbelly-internet-fake-news-gets-funded/Google Scholar
- Ryosuke Nagura, Yohei Seki, Noriko Kando, and Masaki Aono. 2006. A Method of Rating the Credibility of News Documents on the Web. In Proceedings of the 29th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR '06). ACM, New York, NY, USA, Article 1148316, 2 pages. Google ScholarDigital Library
- Daniel J Okeefe. 2002. Persuasion: Theory and research. Vol. 2. Sage, Los Angeles, CA.Google Scholar
- Will Oremus. 2016. Only You Can Stop the Spread of Fake News. http: //www.slate.com. (December 2016).Google Scholar
- Gordon Pennycook and David G Rand. 2017. Assessing the effect of disputed warnings and source salience on perceptions of fake news accuracy. Technical Report. SSRN.Google Scholar
- Peter Pirolli, Evelin Wollny, and Bongwon Suh. 2009. So you know youre getting the best possible information: a tool that increases Wikipedia credibility. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, ACM, 2 Penn Plaza, Suite 701, New York, NY 10121-0701., 1505--1508. Google ScholarDigital Library
- Kashyap Popat, Subhabrata Mukherjee, Jannik Strötgen, and Gerhard Weikum. 2016. Credibility assessment of textual claims on the web. In Proceedings of the 25th ACM International on Conference on Information and Knowledge Management. ACM, ACM, 2 Penn Plaza, Suite 701, New York, NY 10121-0701., 2173--2178. Google ScholarDigital Library
- Martin Potthast, Sebastian Köpsel, Benno Stein, and Matthias Hagen. 2016. Clickbait detection. In European Conference on Information Retrieval. Springer, Springer International Publishing, Gewerbestrasse 11, 6330 Cham, Switzerland, 810--817.Google ScholarCross Ref
- Aditya Ranganathan, Daniel Kim, Nick Adams, and Saul Perlmutter et al. 2017. Crowdsourcing Credibility: A Citizen-Science Approach to NewsLiteracy via Public Editor. Technical Report. University of Berkeley. https: //northwestern.app.box.com/s/77ekftnfp0w8ixxkivkgodqubwhaumyvGoogle Scholar
- Hannah Rashkin, Eunsol Choi, Jin Yea Jang, Svitlana Volkova, and Yejin Choi. 2017. Truth of varying shades: Analyzing language in fake news and political fact-checking. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing. ACL, 209 N. Eighth Street, Stroudsburg PA 18360, USA, 2921--2927.Google ScholarCross Ref
- Jacob Ratkiewicz, Michael Conover, Mark Meiss, Bruno Gonçalves, Snehal Patil, Alessandro Flammini, and Filippo Menczer. 2011. Truthy: mapping the spread of astroturf in microblog streams. In Proceedings of the 20th international conference companion on World wide web. ACM, ACM, 2 Penn Plaza, Suite 701, New York, NY 10121-0701., 249--252. Google ScholarDigital Library
- Marta Reacasens, Cristian Danescu-Niculescu-Mizil, and Dan Jurafsky. 2013. Linguistic Models for Analyzing and Detecting Biased Language. In 51st Annual Meeting of the Association for Computational Linguistics. ACL, ACL, 209 N. Eighth Street, Stroudsburg PA 18360, USA, 1650--1659.Google Scholar
- Paul Resnick, Samuel Carton, Souneil Park, Yuncheng Shen, and Nicole Zeffer. 2014. Rumorlens: A system for analyzing the impact of rumors and corrections in social media. In Proc. Computational Journalism Conference. ACM, ACM, 2 Penn Plaza, Suite 701, New York, NY 10121-0701., 5.Google Scholar
- Soo Young Rieh and David R. Danielson. 2007. Credibility: A multidisciplinary framework. Annual Review of Information Science and Technology 41, 1 (2007), 307--364. Google ScholarDigital Library
- Christine Schmidt. 2017. This project aims to de-flatten digital publishing by matching the best content with premium ads. (Nov 2017). http: / /www.niemanlab.org / 2017 / 11 / this - project - aims - to - de - flatten - digital - publishing-by-matching-the-best-content-with-premium-ads/Google Scholar
- Mike Schmierbach and Anne Oeldorf-Hirsch. 2012. A little bird told me, so I didnt believe it: Twitter, credibility, and issue perceptions. Communication Quarterly 60, 3 (2012), 317--337.Google ScholarCross Ref
- Per O Seglen. 1997. Why the impact factor of journals should not be used for evaluating research. BMJ: British Medical Journal 314, 7079 (1997), 498.Google ScholarCross Ref
- Kai Shu, Amy Sliva, Suhang Wang, Jiliang Tang, and Huan Liu. 2017. Fake News Detection on Social Media: A Data Mining Perspective. SIGKDD Explor. Newsl. 19, 1, Article 3137600 (Sept. 2017), 15 pages. Google ScholarDigital Library
- Henry Silverman and Lin Huang. 2017. News Feed FYI: Fighting Engagement Bait on Facebook. (Dec 2017). https://newsroom.fb.com/news/2017/12/newsfeed-fyi-fighting-engagement-bait-on-facebook/Google Scholar
- Kate Starbird. 2017. Examining the Alternative Media Ecosystem Through the Production of Alternative Narratives of Mass Shooting Events on Twitter. In ICWSM. ACM, ACM, 2 Penn Plaza, Suite 701, New York, NY 10121-0701., 230--239.Google Scholar
- S Shyam Sundar. 1998. Effect of source attribution on perception of online news stories. Journalism & Mass Communication Quarterly 75, 1 (1998), 55--68.Google ScholarCross Ref
- Lauren Vogel. 2017. Viral misinformation threatens public health. Canadian Medical Association Journal 189, 50, Article E1567 (Dec 2017), 1 pages. https: //www.ncbi.nlm.nih.gov/pmc/articles/PMC5738254/Google ScholarCross Ref
- Claire Wardle and Hossein Derakhshan. 2017. Information disorder: Toward an interdisciplinary framework for research and policy making. Coucil of Europe Report DGI(2017)09. Council of Europe. https://edoc.coe.int/en/media-freedom/7495- information- disorder- toward-an-interdisciplinary- framework- for- researchand-policy-making.htmlGoogle Scholar
- Sam Wineburg and Sarah McGrew. 2017. Lateral Reading: Reading Less and Learning More When Evaluating Digital Information. Technical Report Working Paper No. 2017-A1. Stanford History Education Group.Google Scholar
- You Wu, Pankaj K Agarwal, Chengkai Li, Jun Yang, and Cong Yu. 2017. Computational Fact Checking through Query Perturbations. ACM Transactions on Database Systems (TODS) 42, 1 (2017), 4. Google ScholarDigital Library
- Kenneth C.C. Yang. 2007. Factors influencing Internet users-- perceived credibility of news-related blogs in Taiwan. Telematics and Informatics 24, 2 (2007), 69--85. Google ScholarDigital Library
- Wenlin Yao, Zeyu Dai, Ruihong Huang, and James Caverlee. 2017. Online Deception Detection Refueled by Real World Data Collection. In Proceedings of the International Conference Recent Advances in Natural Language Processing, RANLP 2017. INCOMA Ltd., Varna, Bulgaria, 793--802.Google Scholar
Index Terms
- A Structured Response to Misinformation: Defining and Annotating Credibility Indicators in News Articles
Recommendations
On the Misinformation Beat: Understanding the Work of Investigative Journalists Reporting on Problematic Information Online
CSCWJournalists are increasingly investigating and reporting on problematic online content such as misinformation, disinformation, and conspiracy theories, leading to the creation of a new misinformation beat. The process of collecting, analyzing, and ...
Combating Misinformation by Sharing the Truth: a Study on the Spread of Fact-Checks on Social Media
AbstractMisinformation on social media has become a horrendous problem in our society. Fact-checks on information often fall behind the diffusion of misinformation, which can lead to negative impacts on society. This research studies how different factors ...
Combating Misinformation in Bangladesh: Roles and Responsibilities as Perceived by Journalists, Fact-checkers, and Users
CSCWThere has been a growing interest within CSCW community in understanding the characteristics of misinformation propagated through computational media, and the devising techniques to address the associated challenges. However, most work in this area has ...
Comments