Through the Google Goggles: Sociopolitical Bias in Search Engine Design by Alejandro M. Diaz Submitted to the Program in Science, Technology and Society in Partial Fulfillment of the Requirements for Interdisciplinary Honors at Stanford University May 2005 © 2005 Alejandro Diaz All rights reserved The author hereby grants to Stanford University permission to reproduce and to distribute publicly paper and electronic copies of this thesis document in whole or in part. Signature of Author ...............……………………....................................................................................… Program in Science, Technology and Society May 23, 2005 Through the Google Goggles: Sociopolitical Bias in Search Engine Design by Alejandro M. Diaz Submitted to the Program in Science, Technology and Society in Partial Fulfillment of the Requirements for Interdisciplinary Honors ABSTRACT: As much of our knowledge, news, and discourse moves online and to the Web in particular, search engines are increasingly becoming the “gatekeepers” of cyberspace. What’s more, a single search engine—Google—now handles the majority of Web queries. Google directs hundreds of millions of users towards some content and not others, towards some sources and not others. As with all gatekeepers (e.g., television networks), if we believe in the principles of deliberative democracy—and especially if we believe that the Web is an open, “democratic” medium—then we should expect our search engines to disseminate a broad spectrum of information on any given topic. But unlike most other gatekeepers, the information disseminated through modern search engines is not explicitly chosen and written by journalists, editors, and producers. It is instead largely determined by a complex system of algorithms, hardware, and software. The varied designs for search technologies encode certain values about what sort of content is “important,” “relevant,” or “authoritative.” In this thesis, following a hybrid approach that incorporates both media studies and STS theories, we will look at the biases of, motivations for, use of, and resistance to the Google search engine. It is hoped that through this analysis, we might start to uncover the sociopolitics of search. Thesis Supervisor: Professor Robert McGinn Title: Chair, Program for Science, Technology, and Society Acknowledgements I’d like to thank my advisor and mentor, Professor Robert McGinn, for supporting my work during this two-year journey, for giving me the wonderful opportunity to TA, and for teaching me all I know about STS. I am grateful to Professor Aneesh for leading the Honors College (back in 2003) and for keeping STS novel and exciting. Many thanks to Professor Winograd for his insightful comments. And to my new advisor, Professor Turner: bless you for letting me turn in that paper late. This wouldn’t be a Stanford honors thesis if I didn’t thank the Undergraduate Research Program (URP) for supporting this endeavor. Many thanks to my friends—Rebecca, Joanna, Brett, Cheryl, Tamara, Logan, Katherine, Dawn, Saleem, Leonard, Tess—for distracting me. It’s your fault this took so long. To Iceland: your music is what kept me writing. Finally, thanks to my Dad, for always being so proud of me. To my Mom: I owe it all to you. Including those five-hundred-dollars I had to pay the library because you mailed my thesis research back to Stanford and didn’t tape the box properly. Table of Contents 1. Introduction ……………………………………………………………………………….………. 1 2. Technologies of Bias ………………………………………………………………………...…. 5 Frameworks, Foundations, and a Review of the Literature 3. Democracy, the Media, and Search Engines ………………………….………………. 18 The Prospects for a ‘Democratic’ Cyberspace 4. Making Sense of Search ……………………………………………………………………….. 51 A Technical Overview of the Web, Search Engines, and Google 5. The Politics of PageRank …………………………………………………………….……..…. 62 Do the Rich Get Richer? 6. The Transparency Dilemma ……………………………………………………………….... 95 How Hidden Heuristics, Commercial Optimizers, and Manual Exclusion Silently Shape Google’s Results 7. Advertising and “Mixed Motives” ………………………………………………………..…. 109 Exploring Search Engine (Hyper)commercialization 8. Googleopoly ………………………………………………………………………………………… 128 Consolidation, Concentration, and the Future of Search 9. Conclusion …………………………………………………………………………………………… 158 Appendix I: The Random Surfer Model ………………………………………………… A-1 Appendix II: Quantifying the Role of PageRank ……………………………………… A-4 Bibliography 1 Introduction Daniel Brandt is a bit of a conspiracy theorist. Lately, he’s been busy speculating about the dark side of Google on his web site, google-watch.org. On a series of pages—including one entitled “Spooks on Board at Google”—Brandt links the search engine, through the inventor of the Segway scooter, with the FBI, the Bush Administration, and the NSA.1 Here he readily uses phrases like “government surveillance,” “big brother,” “spies in Washington,” and “top- secret security clearance.” Brandt argues that because Google maintains detailed logs of each user’s search terms, tracks IP addresses, and is a virtual monopoly on internet search, it holds private information about the online activities of almost every Internet user. This information, he argues, can be used and misused by Google, corporations, and especially post-9/11 government agencies. But Brandt has another bone to pick with Google. In a separate document on his site, he takes issue with PageRank, the proprietary algorithm at the heart of the search engine’s ranking system.2 Calling PageRank Google’s “original sin,” he argues that it is “uniquely tyrannical,” favoring large, corporate, and established sites over new, independent, and often more relevant pages. Because Google orders its results by PageRank, “it’s frequently the case that a…perfectly relevant page […] will get buried in the rankings because it isn’t sufficiently popular.”3 In addition, it allows corporations with greater budgets to optimize their sites for better placement among the results. He argues that these factors, combined with Google’s overwhelming market share, effectively silence smaller sites and minority opinion on the Internet. With a pinch of alarmist flare, he calls for FTC to regulation of “advertising agencies Introduction 1 that parade as search engines.”4 He even likens Google’s monopoly and behavior to that of Microsoft, forecasting “world domination again … this time with cute colored letters.”5 Put lightly, Brandt’s argument seems a bit of a stretch. Google’s meteoric rise was not the result of advertising, corporate maneuvers, or illicit games. It became popular, it appears, because it simply provides what users want. And so, Brandt has become a bit of an outcast in the Web community. On the forums of the technology site ArsTechnica, for instance, users express little sympathy for his concerns:6 Google rocks. Anyone who says otherwise should be shot. There are two reasons that Google's status should not be concerning: (1) They've been good, so far. (2) Anyone can do it. Everyone talks like Google controls the internet, but obviously, Google's users give it that control. Anyone with bandwidth and server space (ie $$) could compete if they wanted. No monopolies here. because in an ideal world if you want to know where to find an ATM, you should get a lecture about how all the banks are corrupt and stealing your money... Someone please tell me the guy this article is about doesn't actually believe what he says... I use google [sic] all the time, without ever worrying about it turning to the dark side. As long as it's free, I'm gonna stick with it. Users interviewed for a 2005 study by the Pew Internet and American Life Project expressed similar satisfaction and trust in Google, defending their exclusive reliance on the site relatively simple ways:7 Google is clean, fast and thorough. I use Google, it is fast and comprehensive. Google is the search engine I use 98% of the time. I use it almost exclusively because it is fast and accurate. I go directly to vendor sites when I have that option. I use Google because it gives me better searches. It's fast and what I'm looking for is almost always in the top page of results. Some users have even gone so far as to write the company beaming “thank you letters”: I wanted to thank you for such a great product. If google came in a box, I’d eat it for breakfast. If Google were a waitress, I would leave 20 percent. If Google was next to me on a plane, I would let it have the window seat. How do you do it? How can you be that good and that fast? It’s like magic.8 Introduction 2 Despite these positive sentiments, we should not write off Brandt’s rant quite so swiftly. If Google does systematically disfavor new and underrepresented voices after all, then many broad social ideals—of competition, free speech, and deliberative democracy—are potentially undermined. Such a bias is problematic even if it is neither intentional nor premeditated, and even if popular use seems to condone it. As we increasingly turn to Google for information, we must ask ourselves how the search engine is influencing what we do and do not read when we go online. Google and other search engines are, quite clearly, in part responsible for guiding us towards a particular sources of information, and away from others. Search engines therefore play a key role in mediating the interaction between Web users and content authors. These three primary stakeholders—the user, the search engine, and the sites to be searched—may have competing interests, and a negotiation among them is carried out with each search. The user, on the one hand, is interested in finding the most “relevant” information. Web pages, in contrast, are all vying to appear towards the top of the result list—even if they are not the most relevant sources for the user. Search engines are ultimately responsible for reconciling these competing interests, but they themselves may have a (commercial) stake in satisfying the largest number of searches as quickly as possible while selling as much advertising space as they can.