The London School of Economics and Political Science Search Engine Bias The Structuration of Traffic on the World-Wide Web Elizabeth Jane Van Couvering A thesis submitted to The Department Media and Communications London School of Economics and Political Science for the degree of Doctor of Philosophy London, December 2009 1 Declaration I certify that the thesis I have presented for examination for the MPhil/PhD degree of the London School of Economics and Political Science is solely my own work other than where I have clearly indicated that it is the work of others (in which case the extent of any work carried out jointly by me and any other person is clearly identified in it). The copyright of this thesis rests with the author. Quotation from it is permitted, provided that full acknowledgement is made. This thesis may not be reproduced without the prior written consent of the author. I warrant that this authorization does not, to the best of my belief, infringe the rights of any third party. 2 Abstract Search engines are essential components of the World Wide Web; both commercially and in terms of everyday usage, their importance is hard to overstate. This thesis examines the question of why there is bias in search engine results – bias that invites users to click on links to large websites, commercial websites, websites based in certain countries, and websites written in certain languages. In this thesis, the historical development of the search engine industry is traced. Search engines first emerged as prototypical technological startups emanating from Silicon Valley, followed by the acquisition of search engine companies by major US media corporations and their development into portals. The subsequent development of pay-per-click advertising is central to the current industry structure, an oligarchy of virtually integrated companies managing networks of syndicated advertising and traffic distribution. The study also shows a global landscape in which search production is concentrated in and caters for large global advertising markets, leaving the rest of the world with patchy and uneven search results coverage. The analysis of interviews with senior search engine engineers indicates that issues of quality are addressed in terms of customer service and relevance in their discourse, while the analysis of documents, interviews with search marketers, and participant observation within a search engine marketing firm showed that producers and marketers had complex relationships that combine aspects of collaboration, competition, and indifference. The results of the study offer a basis for the synthesis of insights of the political economy of media and communication and the social studies of technology tradition, emphasising the importance of culture in constructing and maintaining both local structures and wider systems. In the case of search engines, the evidence indicates that the culture of the technological entrepreneur is very effective in creating a new mega- business, but less successful in encouraging a debate on issues of the public good or public responsibility as they relate to the search engine industry. Abstract 3 Acknowledgements This thesis is dedicated to my readers, past, present and future. It is a pleasure to thank those who have helped me in this work. First and foremost I would like to thank my husband, Henrik Örnebring, who has been an unwavering and uncomplaining source of support and my daughter Charlotte, who has given me perspective. My interviewees and informants, without whom this work would not have been possible, have my sincere gratitude. Thanks to my colleagues at the LSE who have helped me so much while working on their own projects by listening, reading and commenting: Evangelia Berdou, Ellen Helsper, Zoe Sujon, David Brake, Shani Orgad, Anita Howarth, Patrick McCurdy and Yukie Hori (who also generously helped to translate figures from Nielsen NetRatings Japanese home page). I also owe a debt of gratitude to the more senior scholars at the LSE, especially Sonia Livingstone, Nick Couldry, Terhi Rantanen, Damian Tambini and the late Roger Silverstone, for criticism and encouragement in equal measures. Thanks also go to Jean Morris, whose administrative support was always prompt, efficient, and pleasant. I would like to thank my parents who inspired me to begin this work and, particularly, my mother, Judith Harris, who has been a wonderful support and confidante throughout the work. And finally I would like to specially thank my supervisor, Robin Mansell, since I have benefited very greatly from her scholarship, her help and her commitment to me and to this project from the very first. Thank you all very much. 4 Search Engine Bias: The Structuration of Traffic on the World-Wide Web Table of Contents I. Bias in Internet Search Engines................................................... 9 1.1 Introduction...................................................................................................9 1.2 Researching search engines and identifying a gap.................................. 10 1.3 Investigating bias in internet search engines........................................... 18 1.4 Conclusion.................................................................................................. 27 II. The Dynamics of Technological Structuration ........................ 29 2.1 Introduction................................................................................................ 29 2.2 Bias as a normative conflict ...................................................................... 31 2.3 Technology as a social practice................................................................. 36 2.4 Conflict in context ..................................................................................... 50 2.5 Conceptual framework.............................................................................. 59 2.6 Conclusion.................................................................................................. 63 III. Follow the Results....................................................................... 64 3.1 Introduction................................................................................................ 64 3.2 Development of the research question ................................................... 65 3.3 Research design.......................................................................................... 67 3.4 Level A: Agents and their actions ............................................................ 71 3.5 Level B: Structures..................................................................................... 83 3.6 Alternative research designs...................................................................... 88 3.7 Conclusion.................................................................................................. 90 IV. The History of the Internet Search Engine ...............................91 4.1 Introduction................................................................................................ 91 4.2 The development of the search engine industry .................................... 93 4.3 Technological entrepreneurs (1994-1997)............................................... 95 4.4 Portals and vertical integration (1997-2001) ......................................... 100 4.5 Syndication and consolidation (2002-?)................................................. 114 4.6 Conclusion................................................................................................ 125 V. Finding the Centre..................................................................... 129 5.1 Introduction.............................................................................................. 129 5.2 The four country cases............................................................................ 132 5.3 Search Engines in Japan.......................................................................... 134 5.4 Search Engines in Germany ................................................................... 137 5.5 Search Engines in China ......................................................................... 138 5.6 Search Engines in South Africa.............................................................. 143 5.7 International search engine roduction................................................... 147 5.8 Conclusions .............................................................................................. 158 VI. Is Relevance Relevant?................................................................161 6.1 Introduction.............................................................................................. 161 6.2 The market schema.................................................................................. 163 6.3 The science-technology schema............................................................. 166 5 6.4 The war schema ....................................................................................... 169 6.5 The strategic use of technological schemas .......................................... 171 6.6 The difficulty of articulating the public good ....................................... 177 6.7 Conclusions .............................................................................................. 179 VII. Ranking Highly............................................................................181 7.1 Introduction.............................................................................................. 181 7.2