Ronald E. Robertson
ABSTRACT
Search engines are a primary means through which people obtain information in today»s connected world. Yet, apart from the search engine companies themselves, little is known about how their algorithms filter, rank, and present the web to users. This question is especially pertinent with respect to political queries, given growing concerns about filter bubbles, and the recent finding that bias or favoritism in search rankings can influence voting behavior. In this study, we conduct a targeted algorithm audit of Google Search using a dynamic set of political queries. We designed a Chrome extension to survey participants and collect the Search Engine Results Pages (SERPs) and autocomplete suggestions that they would have been exposed to while searching our set of political queries during the month after Donald Trump»s Presidential inauguration. Using this data, we found significant differences in the composition and personalization of politically-related SERPs by query type, subjects» characteristics, and date.
- Ahmed Allam, Peter Johannes Schulz, and Kent Nakamoto. 2014. The impact of search engine selection and sorting criteria on vaccination beliefs and attitudes: two experiments manipulating Google output. Journal of Medical Internet Research 16, 4 (2014), e100.Google Scholar
Cross Ref
- Hunt Allcott and Matthew Gentzkow. 2017. Social media and fake news in the 2016 election. Journal of Economic Perspectives 31, 2 (May 2017), 211--236.Google ScholarCross Ref
- Solomon E Asch. 1946. Forming impressions of personality. The Journal of Abnormal and Social Psychology 41, 3 (1946), 258--290.Google ScholarCross Ref
- Eytan Bakshy, Solomon Messing, and Lada A Adamic. 2015. Exposure to ideologically diverse news and opinion on Facebook. Science 348, 6239 (2015), 1130--1132.Google Scholar
- Andrea Ballatore, Mark Graham, and Shilad Sen. 2017. Digital hegemonies: the localness of search engine results. Annals of the American Association of Geographers (2017), 1--22.Google Scholar
- Adam J Berinsky, Gregory A Huber, and Gabriel S Lenz. 2012. Evaluating online labor markets for experimental research: Amazon. com's Mechanical Turk. Political Analysis 20, 3 (2012), 351--368.Google ScholarCross Ref
- Edelman Berland. 2017. 2017 Edelman Trust Barometer. http://www.edelman. com/trust2017/. (2017). Accessed: 2017-03-07.Google Scholar
- SEO Blog. 2014. How to Turn Off Google Personalized Search Results. http: //www.seoblog.com/2014/09/turn-google-personalized-search-results/. (2014). Accessed: 2017--10-01.Google Scholar
- Michael Buhrmester, Tracy Kwang, and Samuel D Gosling. 2011. Amazon's Mechanical Turk a new source of inexpensive, yet high-quality, data? Perspectives on Psychological Science 6, 1 (2011), 3--5.Google ScholarCross Ref
- Georg Buscher, Edward Cutrell, and Meredith Ringel Morris. 2009. What do you see when you're surfing?: using eye tracking to predict salient regions of web pages. In Proceedings of the SIGCHI conference on human factors in computing systems. ACM, 21--30. Google ScholarDigital Library
- Traffic Generation Cafe. 2017. Google Incognito: How to Search Google Without Being Tracked. https://trafficgenerationcafe.com/ search-engine-ranking-tip-incognito-search/. (2017). Accessed: 2017--10-01.Google Scholar
- Le Chen, Alan Mislove, and Christo Wilson. 2015. Peeking Beneath the Hood of Uber. In Proceedings of the 2015 ACM Conference on Internet Measurement. Google ScholarDigital Library
- Le Chen, Alan Mislove, and Christo Wilson. 2016. An Empirical Analysis of Algorithmic Pricing on Amazon Marketplace. In Proceedings of the 25th International World Wide Web Conference. Google ScholarDigital Library
- Chitika. 2012. Ask.com Has The Most Long-Winded Searchers, Report Says. Technical Report. Chitika. http://searchengineland.com/ ask-com-has-the-most-long-winded-searchers-report-says-109202Google Scholar
- Inc comScore. 2017. comScore Explicit Core Search Query Report (Desktop Only). https://www.comscore.com/Insights/Rankings. (2017). Accessed: 2017-02--12.Google Scholar
- Nick Diakopoulos. 2013. Algorithmic defamation: The case of the shameless autocomplete. http://www.nickdiakopoulos.com/2013/08/06/ algorithmic-defamation-the-case-of-the-shameless-autocomplete/. (2013).Google Scholar
- Nick Diakopoulos. 2013. Sex, Violence, and Autocomplete Algorithms: Methods and Context. http://www.nickdiakopoulos.com/2013/08/01/ sex-violence-and-autocomplete-algorithms-methods-and-context/. (2013).Google Scholar
- Nicholas Diakopoulos. 2014. Algorithmic accountability reporting: On the investigation of black boxes. Tow Center for Digital Journalism, Columbia University (2014).Google Scholar
- Dimitar Dimitrov, Philipp Singer, Florian Lemmerich, and Markus Strohmaier. 2016. Visual positions of links and clicks on wikipedia. In Proceedings of the 25th International Conference Companion on World Wide Web. International World Wide Web Conferences Steering Committee, 27--28. Google ScholarDigital Library
- Hermann Ebbinghaus. 1913. Memory: A contribution to experimental psychology. Number 3. University Microfilms.Google ScholarCross Ref
- Robert Epstein and Ronald E Robertson. 2015. The search engine manipulation effect (SEME) and its possible impact on the outcomes of elections. Proceedings of the National Academy of Sciences 112, 33 (2015), E4512--E4521.Google ScholarCross Ref
- Robert Epstein, Ronald E. Robertson, David Lazer, and Christo Wilson. 2017. Suppressing the search engine manipulation effect (SEME). Proceedings of the ACM: Human-Computer Interaction 1, 42 (2017). Issue 2. Google ScholarDigital Library
- Eyal Ert and Aliza Fleischer. 2016. Mere Position Effect in Booking Hotels Online. Journal of Travel Research 55, 3 (2016), 311--321.Google ScholarCross Ref
- Motahhare Eslami, Amirhossein Aleyasen, Karrie Karahalios, Kevin Hamilton, and Christian Sandvig. 2015. Feedvis: A path for exploring news feed curation algorithms. In Proceedings of the 18th ACM Conference Companion on Computer Supported Cooperative Work & Social Computing. Google ScholarDigital Library
- Gallup. 2017. Presidential Approval Ratings -- Donald Trump. http://news.gallup. com/poll/203198/presidential-approval-ratings-donald-trump.aspx. (2017). Accessed: 2017--10-08.Google Scholar
- Florian Geigl, Kristina Lerman, Simon Walk, Markus Strohmaier, and Denis Helic. 2016. Assessing the Navigational Effects of Click Biases and Link Insertion on the Web. In Proceedings of the 27th ACM Conference on Hypertext and Social Media. ACM, 37--47. Google ScholarDigital Library
- David F Gleich, Paul G Constantine, Abraham D Flaxman, and Asela Gunawardana. 2010. Tracking the random surfer: empirically measured teleportation parameters in PageRank. In Proceedings of the 19th International Conference on World Wide Web. ACM, 381--390. Google ScholarDigital Library
- Google. 2017. Google Chrome Privacy Notice. https://www.google.com/chrome/ browser/privacy/#browser-modes. (2017). Accessed: 2017-04-01.Google Scholar
- Google. 2017. Search using autocomplete. https://support.google.com/websearch/ answer/106230. (2017). Accessed: 2017-04-01.Google Scholar
- Andrew Guess, Brendan Nyhan, and Jason Reifler. 2018. Selective Exposure to Misinformation: Evidence from the consumption of fake news during the 2016 US presidential campaign. (2018).Google Scholar
- Xunhua Guo, Mingyue Zhang, Chenyue Yang, et al. 2016. Order Effects in Online Product Recommendation: A Scenario-based Analysis. In Proceedings of the 22nd Americas Conference on Information Systems.Google Scholar
- Aniko Hannak, Piotr Sapiezynski, Arash Molavi Kakhki, Balachander Krishnamurthy, David Lazer, Alan Mislove, and Christo Wilson. 2013. Measuring Personalization of Web Search. In Proceedings of the 22nd International World Wide Web Conference. Google ScholarDigital Library
- Aniko Hannak, Gary Soeller, David Lazer, Alan Mislove, and Christo Wilson. 2014. Measuring Price Discrimination and Steering on E-commerce Web Sites. In Proceedings of the 2014 ACM Conference on Internet Measurement. Google ScholarDigital Library
- Robin M Hogarth and Hillel J Einhorn. 1992. Order effects in belief updating: The belief-adjustment model. Cognitive Psychology 24, 1 (1992), 1--55.Google ScholarCross Ref
- Adrianne Jeffries. 2017. Google's featured snippets are worse than fake news. https://theoutline.com/post/1192/ google-s-featured-snippets-are-worse-than-fake-news. (2017). Accessed: 2017--10-08.Google Scholar
- Chloe Kliman-Silver, Aniko Hannak, David Lazer, Christo Wilson, and Alan Mislove. 2015. Location, Location, Location: The Impact of Geolocation on Web Search Personalization. In Proceedings of the 2015 ACM Conference on Internet Measurement. Google ScholarDigital Library
- Certified Knowledge. 2012. Ask.com Has The Most Long-Winded Searchers, Report Says. Technical Report. Certified Knowledge. http://certifiedknowledge.org/ blog/are-search-queries-becoming-even-more-unique-statistics-from-google/Google Scholar
- Juhi Kulshrestha, Motahhare Eslami, Johnnatan Messias, Muhammad Bilal Zafar, Saptarshi Ghosh, Krishna P Gummadi, and Karrie Karahalios. 2017. Quantifying search bias: Investigating sources of bias for political searches in social media. In 20th ACM Conference on Computer-Supported Cooperative Work and Social Computing (CSCW 2017). Google ScholarDigital Library
- David M. J. Lazer, Matthew A. Baum, Yochai Benkler, Adam J. Berinsky, Kelly M. Greenhill, Filippo Menczer, Miriam J. Metzger, Brendan Nyhan, Gordon Pennycook, David Rothschild, Michael Schudson, Steven A. Sloman, Cass R. Sunstein, Emily A. Thorson, Duncan J. Watts, and Jonathan L. Zittrain. 2018. The Science of Fake News. Science 359, 6380 (March 2018), 1094--1096.Google ScholarCross Ref
- Kristina Lerman and Tad Hogg. 2014. Leveraging position bias to improve peer recommendation. PloS one 9, 6 (2014), e98914.Google ScholarCross Ref
- Xiaolu Lu, Alistair Moffat, and J Shane Culpepper. 2016. The effect of pooling and evaluation depth on IR metrics. Information Retrieval Journal 19, 4 (2016), 416--445. Google ScholarDigital Library
- Ramona Ludolph, Ahmed Allam, and Peter J Schulz. 2016. Manipulating Google's Knowledge Graph box to counter biased information processing during an online search on vaccination: application of a technological debiasing strategy. Journal of Medical Internet research 18, 6 (2016).Google ScholarCross Ref
- Connor McMahon, Isaac Johnson, and Brent Hecht. 2017. The Substantial Interdependence of Wikipedia and Google: A Case Study on the Relationship Between Peer Production Communities and Information Technologies. (2017).Google Scholar
- Nicco Mele, David Lazer, Matthew Baum, Nir Grinberg, Lisa Friedland, Kenneth Joseph, Will Hobbs, and Carolina Mattsson. 2017. Combating fake news: An agenda for research and action. (2017).Google Scholar
- P Takis Metaxas and Yada Pruksachatkun. 2017. Manipulation of Search Engine Results during the 2016 US Congressional Elections. In ICIW.Google Scholar
- Peter J. Meyers. 2013. 101 Google Answer Boxes: A Journey into the Knowledge Graph. https://moz.com/blog/ 101-google-answer-boxes-a-journey-into-the-knowledge-graph. (2013). Accessed: 2017--10-08.Google Scholar
- Jakub Mikians, László Gyarmati, Vijay Erramilli, and Nikolaos Laoutaris. 2012. Detecting price and search discrimination on the Internet. In Proceedings of the 11th ACM Workshop on Hot Topics in Networks. Google ScholarDigital Library
- Jakub Mikians, László Gyarmati, Vijay Erramilli, and Nikolaos Laoutaris. 2013. Crowd-assisted search for price discrimination in e-commerce: First results. In Proceedings of the Ninth ACM Conference on Emerging Networking Experiments and Technologies. Google ScholarDigital Library
- Ronald B Mincy. 1993. The Urban Institute audit studies: their research and policy context. Clear and Convincing Evidence: Measurement of Discrimination in America (1993), 165--86.Google Scholar
- Amy Mitchell, Jeffrey Gottfried, Elisa Shearer, and Kristine Lu. 2017. How Americans Encounter, Recall and Act Upon Digital News. Technical Report. Pew Research Center.Google Scholar
- Alistair Moffat and Justin Zobel. 2008. Rank-biased precision for measurement of retrieval effectiveness. ACM Transactions on Information Systems (TOIS) 27, 1 (2008), 2. Google ScholarDigital Library
- Motherboard. 2017. Go Ahead, Ask Google ?What Happened to the Dinosaurs?. https://motherboard.vice.com/en_us/article/pga4wg/ go-ahead-ask-google-what-happened-to-the-dinosaurs. (2017). Accessed: 2017--10-08.Google Scholar
- Moz. 2012. Face-off - 4 Ways to De-personalize Google. https://moz.com/blog/ face-off-4-ways-to-de-personalize-google. (2012). Accessed: 2017--10-01.Google Scholar
- Sean A Munson, Stephanie Y Lee, and Paul Resnick. 2013. Encouraging Reading of Diverse Political Viewpoints with a Browser Widget.. In ICWSM.Google Scholar
- Bennet B Murdock. 1962. The serial position effect of free recall. Journal of Experimental Psychology 64, 5 (1962), 482--488.Google ScholarCross Ref
- Jamie Murphy, Charles Hofacker, and Richard Mizerski. 2006. Primacy and recency effects on clicking behavior. Journal of Computer-Mediated Communication 11, 2 (2006), 522--535.Google ScholarCross Ref
- Nic Newman, Richard Fletcher, Antonis Kalogeropoulos, David A. L. Levy, and Rasmus Kleis Nielsen. 2017. Reuters Institute Digital News Report 2017. Technical Report. Reuters Institute for the Study of Journalism.Google Scholar
- Helen Nissenbaum. 1996. Accountability in a computerized society. Science and Engineering Ethics 2, 1 (1996), 25--42.Google ScholarCross Ref
- Alamir Novin and Eric Meyers. 2017. Making sense of conflicting science information: Exploring bias in the search engine result page. In Proceedings of the 2017 Conference on Human Information Interaction and Retrieval. ACM, 175--184. Google ScholarDigital Library
- Bing Pan, Helene Hembrooke, Thorsten Joachims, Lori Lorigo, Geri Gay, and Laura Granka. 2007. In google we trust: Users' decisions on rank, position, and relevance. Journal of Computer-Mediated Communication 12, 3 (2007), 801--823.Google ScholarCross Ref
- Eli Pariser. 2011. The filter bubble: What the Internet is hiding from you. Penguin UK. Google ScholarDigital Library
- Parse.ly. 2017. The authority report: How audiences find articles, by topic. Technical Report. Parse.ly. http://learn.parsely.com/rs/314-EBB-255/images/ authority-report-13.pdfGoogle Scholar
- Frank Pasquale. 2015. The black box society: The secret algorithms that control money and information. Harvard University Press, Cambridge, MA. Google Scholar
- Eyal Peer, Laura Brandimarte, Sonam Samat, and Alessandro Acquisti. 2017. Beyond the Turk: alternative platforms for crowdsourcing behavioral research. Journal of Experimental Social Psychology 70 (2017), 153--163.Google ScholarCross Ref
- Eyal Peer, Joachim Vosgerau, and Alessandro Acquisti. 2014. Reputation as a sufficient condition for data quality on Amazon Mechanical Turk. Behavior research methods 46, 4 (2014), 1023--1031.Google Scholar
- Kristen Purcell, Joanna Brenner, and Lee Rainie. 2012. Search engine use 2012. Technical Report. Pew Research Center's Internet and American Life Project.Google Scholar
- Advanced Web Ranking. 2017. CTR Study February 2017. https://www. advancedwebranking.com/cloud/ctrstudy. (2017). Accessed: 2017-04-01.Google Scholar
- Christian Sandvig, Kevin Hamilton, Karrie Karahalios, and Cedric Langbort. 2014. Auditing algorithms: Research methods for detecting discrimination on internet platforms. In Proceedings of 'Data and Discrimination: Converting Critical Concerns into Productive Inquiry?, a Productivereconference at the 64th Annual Meeting of the International Communication Association.Google Scholar
- Gary Soeller, Karrie Karahalios, Christian Sandvig, and Christo Wilson. 2016. MapWatch: Detecting and Monitoring International Border Personalization on Online Maps. In Proceedings of the 25th International World Wide Web Conference. Google ScholarDigital Library
- Lauren Starling. 2013. How to remove a word from Google autocomplete. (2013). http://www.laurenstarling.org/ how-to-remove-a-word-from-google-autocomplete/Google Scholar
- Danny Sullivan. 2017. Google's 'One True Answer' problem when featured snippets go bad. https://searchengineland.com/ googles-one-true-answer-problem-featured-snippets-270549. (2017). Accessed: 2017--10-08.Google Scholar
- Sander van der Linden, Anthony Leiserowitz, Seth Rosenthal, and Edward Maibach. 2017. Inoculating the public against misinformation about climate change. Global Challenges 1, 2 (2017).Google Scholar
- Soroush Vosoughi, Deb Roy, and Sinan Aral. 2018. The Spread of True and False News Online. Science 359, 6380 (March 2018), 1146--1151.Google ScholarCross Ref
- William Webber, Alistair Moffat, and Justin Zobel. 2010. A similarity measure for indefinite rankings. ACM Transactions on Information Systems (TOIS) 28, 4 (2010), 20. Google ScholarDigital Library
- Wiideman. 2010. Beat the autocomplete - A study of Google auto-suggest. (2010). https://www.wiideman.com/research/google-autocomplete/study-resultsGoogle Scholar
Index Terms
- Auditing the Personalization and Composition of Politically-Related Search Engine Results Pages
Recommendations
Auditing the Partisanship of Google Search Snippets
WWW '19: The World Wide Web ConferenceThe text snippets presented in web search results provide users with a slice of page content that they can quickly scan to help inform their click decisions. However, little is known about how these snippets are generated or how they relate to a user's ...
Auditing Partisan Audience Bias within Google Search
There is a growing consensus that online platforms have a systematic influence on the democratic process. However, research beyond social media is limited. In this paper, we report the results of a mixed-methods algorithm audit of partisan audience bias ...
What users see - Structures in search engine results pages
This paper investigates the composition of search engine results pages. We define what elements the most popular web search engines use on their results pages (e.g., organic results, advertisements, shortcuts) and to which degree they are used for ...
Comments