Ronald E. Robertson
Skip Abstract Section
Skip Supplemental Material SectionAbstract
There is a growing consensus that online platforms have a systematic influence on the democratic process. However, research beyond social media is limited. In this paper, we report the results of a mixed-methods algorithm audit of partisan audience bias and personalization within Google Search. Following Donald Trump's inauguration, we recruited 187 participants to complete a survey and install a browser extension that enabled us to collect Search Engine Results Pages (SERPs) from their computers. To quantify partisan audience bias, we developed a domain-level score by leveraging the sharing propensities of registered voters on a large Twitter panel. We found little evidence for the "filter bubble'' hypothesis. Instead, we found that results positioned toward the bottom of Google SERPs were more left-leaning than results positioned toward the top, and that the direction and magnitude of overall lean varied by search query, component type (e.g. "answer boxes"), and other factors. Utilizing rank-weighted metrics that we adapted from prior work, we also found that Google's rankings shifted the average lean of SERPs to the right of their unweighted average.
Supplemental Material
Available for Download
- Hunt Allcott and Matthew Gentzkow. 2017. Social media and fake news in the 2016 election. Journal of Economic Perspectives 31, 2 (May 2017), 211--236.Google Scholar
Cross Ref
- AllSides. 2018. Media Bias Ratings. AllSides. (2018). https://www.allsides.com/media-bias/media-bias-ratingsGoogle Scholar
- Eytan Bakshy, Solomon Messing, and Lada A Adamic. 2015. Exposure to ideologically diverse news and opinion on Facebook. Science 348, 6239 (2015), 1130--1132.Google Scholar
- Nicholas J. Belkin. 1980. Anamolous States of Knowledge as a Basis for Information Retrieval. Canadian Journal of Information and Library Sciences 5 (1980), 133--143.Google Scholar
- James R Beniger. 1987. Personalization of mass media and the growth of pseudo-community. Communication research 14, 3 (1987), 352--371.Google Scholar
- W. Lance Bennett and Jarol B. Manheim. 2006. The One-Step Flow of Communication. The ANNALS of the American Academy of Political and Social Science 608, 1 (2006), 213--232.Google ScholarCross Ref
- Shlomo Berkovsky, Jill Freyne, and Harri Oinas-Kukkonen. 2012. Influencing individually: fusing personalization and persuasion. ACM Transactions on Interactive Intelligent Systems (TiiS) 2, 2 (2012), 9. Google ScholarDigital Library
- Edelman Berland. 2017. 2017 Edelman Trust Barometer. (2017). http://www.edelman.com/trust2017/ Accessed: 2017-03-07.Google Scholar
- Edward L Bernays. 1928. Propaganda. Ig publishing.Google Scholar
- Robert M Bond, Christopher J Fariss, Jason J Jones, Adam DI Kramer, Cameron Marlow, Jaime E Settle, and James H Fowler. 2012. A 61-million-person experiment in social influence and political mobilization. Nature 489, 7415 (2012), 295--298.Google Scholar
- Erik Borra and Ingmar Weber. 2012. Political Insights: Exploring Partisanship in Web Search Queries. First Monday 17, 7 (July 2012).Google ScholarCross Ref
- Ceren Budak, Sharad Goel, and Justin M Rao. 2016. Fair and balanced? Quantifying media bias through crowdsourced content analysis. Public Opinion Quarterly 80, S1 (2016), 250--271.Google ScholarCross Ref
- Michael Buhrmester, Tracy Kwang, and Samuel D Gosling. 2011. Amazon's Mechanical Turk a new source of inexpensive, yet high-quality, data? Perspectives on Psychological Science 6, 1 (2011), 3--5.Google ScholarCross Ref
- Aylin Caliskan, Joanna J Bryson, and Arvind Narayanan. 2017. Semantics derived automatically from language corpora contain human-like biases. Science 356, 6334 (2017), 183--186.Google Scholar
- Dana R. Carney, John T. Jost, Samuel D. Gosling, and Jeff Potter. 2008. The Secret Lives of Liberals and Conservatives: Personality Profiles, Interaction Styles, and the Things They Leave Behind. Political Psychology 29, 6 (Dec. 2008), 807--840.Google ScholarCross Ref
- Kalyani Chadha and Rob Wells. 2016. Journalistic Responses to Technological Innovation in Newsrooms: An Exploratory Study of Twitter Use. Digital Journalism 4, 8 (Nov. 2016), 1020--1035.Google ScholarCross Ref
- Chun-Fang Chiang and Brian Knight. 2011. Media Bias and Influence: Evidence from Newspaper Endorsements. The Review of Economic Studies 78, 3 (2011), 795--820.Google ScholarCross Ref
- Inc comScore. 2017. comScore Explicit Core Search Query Report (Desktop Only). (2017). https://www.comscore. com/Insights/Rankings Accessed: 2017-02--12.Google Scholar
- Kareem Darwish, Walid Magdy, and Tahar Zanouda. 2017. Trump vs. Hillary: What Went Viral During the 2016 US Presidential Election. In International Conference on Social Informatics. Springer, 143--161.Google Scholar
- Stefano DellaVigna and Ethan Kaplan. 2007. The Fox News effect: Media bias and voting. The Quarterly Journal of Economics 122, 3 (2007), 1187--1234.Google ScholarCross Ref
- Nicholas Diakopoulos, Daniel Trielli, Jennifer Stark, and Sean Mussenden. 2018. I Vote For-How Search Informs Our Choice of Candidate. In Digital Dominance: The Power of Google, Amazon, Facebook, and Apple, M. Moore and D. Tambini (Eds.). 22.Google Scholar
- James N Druckman and Michael Parkin. 2005. The impact of media bias: How editorial slant affects voters. Journal of Politics 67, 4 (2005), 1030--1049.Google ScholarCross Ref
- William H. Dutton and Grant Blank. 2013. Cultures of the Internet: The Internet in Britain: Oxford Internet Survey 2013 Report. Technical Report. Oxford Internet Institute, University of Oxford. http://oxis.oii.ox.ac.uk/wp-content/ uploads/2014/11/OxIS-2013.pdfGoogle Scholar
- William H. Dutton, Bianca Christin Reisdorf, Elizabeth Dubois, and Grant Blank. 2017. Search and Politics: A Cross-National Survey. (2017).Google Scholar
- William H. Dutton, Bianca Christin Reisdorf, Elizabeth Dubois, and Grant Blank. 2017. Search and Politics: The Uses and Impacts of Search in Britain, France, Germany, Italy, Poland, Spain, and the United States. (2017).Google Scholar
- Dina ElBoghdady. 2013-04--23T09:52--500. Market Quavers after Fake AP Tweet Says Obama Was Hurt in White House Explosions. Washington Post (2013-04--23T09:52--500).Google Scholar
- Robert Epstein. 2018. Taming Big Tech: The Case for Monitoring. Hacker Noon. (May 2018). https://hackernoon. com/taming-big-tech-5fef0df0f00dGoogle Scholar
- Robert Epstein and Ronald E Robertson. 2015. The search engine manipulation effect (SEME) and its possible impact on the outcomes of elections. Proceedings of the National Academy of Sciences 112, 33 (2015), E4512--E4521.Google ScholarCross Ref
- Robert Epstein and Ronald E Robertson. 2017. A Method for Detecting Bias in Search Rankings, with Evidence of Systematic Bias Related to the 2016 Presidential Election. Technical Report White Paper no. WP-17-02. American Institute for Behavioral Research and Technology. 5 pages.Google Scholar
- Robert Epstein, Ronald E. Robertson, David Lazer, and Christo Wilson. 2017. Suppressing the search engine manipulation effect (SEME). Proceedings of the ACM: Human-Computer Interaction 1, 42 (2017). Issue CSCW. Google ScholarDigital Library
- S. Fortunato, A. Flammini, F. Menczer, and A. Vespignani. 2006. Topical Interests and the Mitigation of Search Engine Bias. Proceedings of the National Academy of Sciences 103, 34 (Aug. 2006), 12684--12689.Google ScholarCross Ref
- Adrien Friggeri, Lada A Adamic, Dean Eckles, and Justin Cheng. 2014. Rumor Cascades. In ICWSM.Google Scholar
- Gallup. 2017. Presidential Approval Ratings -- Donald Trump. (2017). http://news.gallup.com/poll/203198/ presidential-approval-ratings-donald-trump.aspx Accessed: 2017--10-08.Google Scholar
- Matthew Gentzkow and Jesse M Shapiro. 2010. What drives media slant? Evidence from US daily newspapers. Econometrica 78, 1 (2010), 35--71.Google ScholarCross Ref
- Alan S Gerber, Dean Karlan, and Daniel Bergan. 2009. Does the media matter? A field experiment measuring the effect of newspapers on voting behavior and political opinions. American Economic Journal: Applied Economics 1, 2 (2009), 35--52.Google ScholarCross Ref
- Tarleton Gillespie. 2014. The Relevance of Algorithms. In Media Technologies: Essays on Communication, Materiality, and Society, Tarleton Gillespie, Pablo J. Boczkowski, and Kirsten A. Foot (Eds.). The MIT Press, Cambridge, 167--194.Google Scholar
- Sharad Goel, Ashton Anderson, Jake Hofman, and Duncan J. Watts. 2015. The Structural Virality of Online Diffusion. Management Science 62, 1 (July 2015), 180--196.Google Scholar
- Sharad Goel, Duncan J. Watts, and Daniel G. Goldstein. 2012. The Structure of Online Diffusion Networks. ACM Press, 623. Google ScholarDigital Library
- Google. 2015. Tweets take flight in the Google app. Google Blog. (2015). https://googleblog.blogspot.com/2015/05/ tweets-take-flight-in-google-app.htmlGoogle Scholar
- Google. 2016. Google Trends: Year in Search. Google Trends. (2016). https://trends.google.com/trends/yis/2016/ GLOBALGoogle Scholar
- Google. 2017. Google Chrome Privacy Notice. (2017). https://www.google.com/chrome/browser/privacy/ #browser-modes Accessed: 2017-04-01. {42} Jeffrey Gottfried and Elisa Shearer. 2016. News use across social media platforms. Pew Research Center. (2016). http://www.journalism.org/2016/05/26/news-use-across-social-media-platforms-2016/Google Scholar
- Tim Groseclose and Jeffrey Milyo. 2005. A measure of media bias. The Quarterly Journal of Economics 120, 4 (2005), 1191--1237.Google ScholarCross Ref
- Andrew Guess, Brendan Nyhan, Benjamin Lyons, and Jason Reifler. 2018. Avoiding the Echo Chamber About Echo Chambers. Technical Report. Knight Foundation.Google Scholar
- Aniko Hannak, Piotr Sapie?y'ski, Arash Molavi Kakhki, Balachander Krishnamurthy, David Lazer, Alan Mislove, and Christo Wilson. 2013. Measuring Personalization of Web Search. In Proceedings of the 22nd International World Wide Web Conference. Google ScholarDigital Library
- Itai Himelboim, Stephen McCreery, and Marc Smith. 2013. Birds of a Feather Tweet Together: Integrating Network and Content Analyses to Examine Cross-Ideology Exposure on Twitter. Journal of Computer-Mediated Communication 18, 2 (Jan. 2013), 40--60. Google ScholarDigital Library
- William Hobbs, Lisa Friedland, Kenneth Joseph, Oren Tsur, Stefan Wojcik, and David Lazer. 2017. "Voters of the Year": 19 Voters Who Were Unintentional Election Poll Sensors on Twitter. In ICWSM.Google Scholar
- Lucas D Introna and Helen Nissenbaum. 2000. Shaping the Web: Why the Politics of Search Engines Matters. The Information Society 16 (2000), 169--185.Google ScholarCross Ref
- Shanto Iyengar and Kyu S Hahn. 2009. Red Media, Blue Media: Evidence of Ideological Selectivity in Media Use. Journal of Communication 59, 1 (March 2009), 19--39.Google ScholarCross Ref
- Mohit Iyyer, Peter Enns, Jordan Boyd-Graber, and Philip Resnik. 2014. Political Ideology Detection Using Recursive Neural Networks. In Association for Computational Linguistics.Google Scholar
- Shan Jiang and Christo Wilson. 2018. Linguistic Signals under Misinformation and Fact-Checking: Evidence from User Comments on Social Media. Proceedings of the ACM: Human-Computer Interaction 2 (2018). Issue CSCW. Google ScholarDigital Library
- Kenneth Joseph, Lisa Friedland, William Hobbs, Oren Tsur, and David Lazer. 2017. ConStance: Modeling Annotation Contexts to Improve Stance Classification. arXiv preprint arXiv:1708.06309 (2017).Google Scholar
- Cecilia Kang, Tiffany Hsu, Kevin Roose, Natasha Singer, and Matthew Rosenberg. 2018. Mark Zuckerberg Testimony: Day 2 Brings Tougher Questioning. The New York Times (April 2018).Google Scholar
- Elihu Katz and Paul Felix Lazarsfeld. 1955. Personal influence, The part played by people in the flow of mass communications. The Free Press.Google Scholar
- Matthew Kay, Cynthia Matuszek, and Sean A. Munson. 2015. Unequal Representation and Gender Stereotypes in Image Search Results for Occupations. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems. ACM Press, 3819--3828. Google ScholarDigital Library
- Chloe Kliman-Silver, Aniko Hannak, David Lazer, Christo Wilson, and Alan Mislove. 2015. Location, Location, Location: The Impact of Geolocation on Web Search Personalization. In Proceedings of the 2015 ACM Conference on Internet Measurement. {57} Juhi Kulshrestha, Motahhare Eslami, Johnnatan Messias, Muhammad Bilal Zafar, Saptarshi Ghosh, Krishna P Gummadi, and Karrie Karahalios. 2017. Quantifying search bias: Investigating sources of bias for political searches in social media. In 20th ACM Conference on Computer-Supported Cooperative Work and Social Computing (CSCW 2017). {58} Harold D Lasswell. 1938. Propaganda technique in the world war. P. Smith, New York, NY. Google ScholarDigital Library
- David Lazer, Alex (Sandy) Pentland, Lada Adamic, Sinan Aral, Albert Laszlo Barabasi, Devon Brewer, Nicholas Christakis, Noshir Contractor, James Fowler, Myron Gutmann, Tony Jebara, Gary King, Michael Macy, Deb Roy, and Marshall Van Alstyne. 2009. Life in the Network: The Coming Age of Computational Social Science. Science (New York, N.Y.) 323, 5915 (Feb. 2009), 721--723.Google Scholar
- David M. J. Lazer, Matthew A. Baum, Yochai Benkler, Adam J. Berinsky, Kelly M. Greenhill, Filippo Menczer, Miriam J. Metzger, Brendan Nyhan, Gordon Pennycook, David Rothschild, Michael Schudson, Steven A. Sloman, Cass R. Sunstein, Emily A. Thorson, Duncan J. Watts, and Jonathan L. Zittrain. 2018. The Science of Fake News. Science 359, 6380 (March 2018), 1094--1096.Google ScholarCross Ref
- Huyen Le, Zubair Shafiq, and Padmini Srinivasan. 2017. Scalable News Slant Measurement Using Twitter. In ICWSM. 4.Google Scholar
- S. C. Matz, M. Kosinski, G. Nave, and D. J. Stillwell. 2017. Psychological Targeting as an Effective Approach to Digital Mass Persuasion. Proceedings of the National Academy of Sciences (Nov. 2017), 201710966.Google Scholar
- Connor McMahon, Isaac Johnson, and Brent Hecht. 2017. The Substantial Interdependence of Wikipedia and Google: A Case Study on the Relationship Between Peer Production Communities and Information Technologies. (2017).Google Scholar
- Miller McPherson, Lynn Smith-Lovin, and James M Cook. 2001. Birds of a Feather: Homophily in Social Networks. Annual Review of Sociology 27, 1 (Aug. 2001), 415--444.Google ScholarCross Ref
- Solomon Messing and Sean J. Westwood. 2014. Selective Exposure in the Age of Social Media: Endorsements Trump Partisan Source Affiliation When Selecting News Online. Communication Research 41, 8 (Dec. 2014), 1042--1063.Google ScholarCross Ref
- P Takis Metaxas and Yada Pruksachatkun. 2017. Manipulation of Search Engine Results during the 2016 US Congressional Elections. In ICIW.Google Scholar
- Ronald B Mincy. 1993. The Urban Institute audit studies: their research and policy context. Clear and Convincing Evidence: Measurement of Discrimination in America (1993), 165--86.Google Scholar
- Amy Mitchell, Jeffrey Gottfried, Jocelyn Kiley, and Katerina Eva Matsa. 2014. Political Polarization & Media Habits. Pew Research Center's Journalism Project. (Oct. 2014).Google Scholar
- Nic Newman, David A. L. Levy, and Rasmus Kleis Nielsen. 2017. Reuters Institute Digital News Report 2017. SSRN Electronic Journal (2017).Google Scholar
- Safiya Umoja Noble. 2018. Algorithms of Oppression: How Search Engines Reinforce Racism. NYU Press.Google Scholar
- Brendan Nyhan. 2012. Does the US Media Have a Liberal Bias? Perspectives on Politics 10, 03 (Sept. 2012), 767--771.Google ScholarCross Ref
- Stefan Palan and Christian Schitter. 2017. Prolific.ac - subject pool for online experiments. Journal of Behavioral and Experimental Finance (2017).Google Scholar
- Eli Pariser. 2011. The filter bubble: What the Internet is hiding from you. Penguin UK. Google ScholarDigital Library
- Souneil Park, Minsam Ko, Jungwoo Kim, Ying Liu, and Junehwa Song. 2011. The politics of comments: predicting political orientation of news stories with commenters' sentiment patterns. In Proceedings of the ACM 2011 conference on Computer supported cooperative work. ACM, 113--122. Google ScholarDigital Library
- Eyal Peer, Laura Brandimarte, Sonam Samat, and Alessandro Acquisti. 2017. Beyond the Turk: alternative platforms for crowdsourcing behavioral research. Journal of Experimental Social Psychology 70 (2017), 153--163.Google ScholarCross Ref
- Richard E Petty, Joseph R Priester, and Pablo Brinol. 2009. Mass media attitude change: Implications of the elaboration likelihood model of persuasion (3rd ed.). Vol. 2. Lawrence Erlbaum Associates, Mahwah, NJ, 155--198.Google Scholar
- Media Insight Project. 2014. The Personal News Cycle: How Americans Choose to Get Their News. Media Insight Project. (March 2014). https://www.americanpressinstitute.org/publications/reports/survey-research/ how-americans-get-news/Google Scholar
- Kristen Purcell, Joanna Brenner, and Lee Rainie. 2012. Search engine use 2012. Technical Report. Pew Research Center's Internet and American Life Project.Google Scholar
- Advanced Web Ranking. 2017. CTR study. (2017). https://www.advancedwebranking.com/cloud/ctrstudy.Google Scholar
- Marta Recasens, Cristian Danescu-Niculescu-Mizil, and Dan Jurafsky. 2013. Linguistic Models for Analyzing and Detecting Biased Language. In 51st Annual Meeting of the Association for Computational Linguistics. ACL, 1650--1659.Google Scholar
- Filipe N Ribeiro, Lucas Henrique, Fabricio Benevenuto, Abhijnan Chakraborty, Juhi Kulshrestha, Mahmoudreza Babaei, and Krishna P Gummadi. 2018. Media Bias Monitor: Quantifying Biases of Social Media News Outlets at Large-Scale. (2018), 10.Google Scholar
- Christoph Riedl, Johannes Bjelland, Geoffrey Canright, Asif Iqbal, Kenth Engo-Monsen, Taimur Qureshi, Pal Roe Sundsoy, and David Lazer. 2018. Product Diffusion through On-Demand Information-Seeking Behaviour. Journal of The Royal Society Interface 15, 139 (Feb. 2018), 20170751.Google ScholarCross Ref
- Rainer Riemann, Claudia Grubich, Susanne Hempel, Susanne Mergl, and Manfred Richter. 1993. Personality and attitudes towards current political topics. Personality and Individual Differences 15, 3 (1993).Google Scholar
- Ronald E Robertson, David Lazer, and Christo Wilson. 2018. Auditing the Personalization and Composition of Politically-Related Search Engine Results Pages. In WWW 2018: The 2018 Web Conference. ACM, Lyon, France, 11. Google ScholarDigital Library
- Christian Sandvig, Kevin Hamilton, Karrie Karahalios, and Cedric Langbort. 2014. Auditing algorithms: Research methods for detecting discrimination on internet platforms. In Proceedings of "Data and Discrimination: Converting Critical Concerns into Productive Inquiry", a Productivereconference at the 64th Annual Meeting of the International Communication Association.Google Scholar
- Barry Schwartz. 2015. Google Officially Expands Twitter Into Desktop Search Results. Search Engine Land. (2015). https://searchengineland.com/google-officially-expands-twitter-into-desktop-search-results-228723Google Scholar
- David O. Sears and Jonathan L. Freedman. 1967. Selective Exposure to Information: A Critical Review. Public Opinion Quarterly 31, 2 (1967), 194.Google ScholarCross Ref
- Pawel Sobkowicz. 2017. Social Simulation Models at the Ethical Crossroads. Science and Engineering Ethics (Nov. 2017), 1--15. {89} Ruihua Song, Zhenxiao Luo, Ji-Rong Wen, Yong Yu, and Hsiao-Wuen Hon. 2007. Identifying Ambiguous Queries in Web Search. In Proceedings of the 16th International Conference on World Wide Web (WWW '07). ACM, New York, NY, USA, 1169--1170. https://doi.org/10.1145/1242572.1242749 Google ScholarDigital Library
- Betsy Sparrow, Jenny Liu, and Daniel M Wegner. 2011. Google effects on memory: Cognitive consequences of having information at our fingertips. Science 333, 6043 (2011), 776--778.Google ScholarCross Ref
- Jennifer A Stark and Nicholas Diakopoulos. 2017. Using Baselines for Algorithm Audits. In European Conference on Data and Computational Journalism. 3.Google Scholar
- Danny Sullivan. 2016. Google Now Handles at Least 2 Trillion Searches per Year. Search Engine Land. (May 2016). https://searchengineland.com/google-now-handles-2--999-trillion-searches-per-year-250247Google Scholar
- New York Times. 1995. Edward Bernays, 'Father of Public Relations' And Leader in Opinion Making, Dies at 103. (1995).Google Scholar
- Francesca Tripodi. 2018. Searching for Alternative Facts: Analyzing Scriptural Inference in Conservative News Practices. Data & Society. (May 2018).Google Scholar
- Liwen Vaughan and Mike Thelwall. 2004. Search Engine Coverage Bias: Evidence and Possible Causes. Information Processing & Management 40, 4 (July 2004), 693--707. Google ScholarDigital Library
- Soroush Vosoughi, Deb Roy, and Sinan Aral. 2018. The Spread of True and False News Online. Science 359, 6380 (March 2018), 1146--1151.Google ScholarCross Ref
- Duncan J. Watts and David M. Rothschild. 2017. Don't blame the election on fake news. Blame it on the media. Columbia Journalism Review. (2017). https://www.cjr.org/analysis/fake-news-media-election-trump.phpGoogle Scholar
- Felix Ming Fai Wong, Chee Wei Tan, Soumya Sen, and Mung Chiang. 2016. Quantifying political leaning from tweets, retweets, and retweeters. IEEE transactions on knowledge and data engineering 28, 8 (2016), 2158--2172.Google Scholar
- Daniel Xiaodan Zhou, Paul Resnick, and Qiaozhu Mei. 2011. Classifying the Political Leaning of News Articles and Users from User Votes.. In ICWSM.Google Scholar
- Jonathan L Zittrain. 2014. Engineering an election. Harvard Law Review Forum 127, 8 (2014), 335--341.Google Scholar
Index Terms
- Auditing Partisan Audience Bias within Google Search
Recommendations
Auditing the Personalization and Composition of Politically-Related Search Engine Results Pages
WWW '18: Proceedings of the 2018 World Wide Web ConferenceSearch engines are a primary means through which people obtain information in today»s connected world. Yet, apart from the search engine companies themselves, little is known about how their algorithms filter, rank, and present the web to users. This ...
Auditing the Partisanship of Google Search Snippets
WWW '19: The World Wide Web ConferenceThe text snippets presented in web search results provide users with a slice of page content that they can quickly scan to help inform their click decisions. However, little is known about how these snippets are generated or how they relate to a user's ...
Comments