TheInformation Society, 16:169 –185, 2000 Copyright c 2000T aylor& Francis 0197-2243/° 00$12.00+ .00

Shaping the Web: Whythe Politics ofSearch Engines Matters

Lucas D.Introna LondonSchool of Economics,London, United Kingdom

Helen Nissenbaum University Center for HumanV alues,Princeton University, Princeton,New Jersey,USA

Enhancedby the technologyof the WorldWide W eb,it This articleargues that searchengines raise not merely technical has becomean integral part ofthe ever-expandingglobal issuesbut alsopolitical ones. Our studyof searchengines suggests media system, movingonto center stage ofmedia politics thatthey systematically exclude (in somecases by design and in alongsidetraditional broadcastmedia— television andra- some,accidentally ) certainsites and certaintypes of sitesin favor dio.Enthusiasts ofthe “newmedium” have heralded it as ofothers,systematically giving prominence to some at theexpense ademocratizingforce that will givevoice to diverse so- ofothers. We argue that such biases,which would leadto a nar- cial, economic,and cultural groups,to members ofsociety rowingof theWeb’ s functioningin society,run counterto thebasic notfrequentlyheard in the publicsphere. It will empower architectureof the Web aswell as tothevalues and idealsthat have the traditionally disempowered,giving them access both fueledwidespread support for its growth and development.We to typically unreachablenodes of powerand to previously considerways of addressingthe politics of searchengines, raising inaccessible troves ofinformation. doubts whether,in particular,themarket mechanism could serve asanacceptablecorrective. Toscholars oftraditional media,these optimistic claims must havea ringof familiarity, echoingsimilar optimistic predictions concerningthe democratizingand empower- Keywords searchengines, bias, values in design,W orldWide W eb, ingcapacities ofbothradio and television. Instead ofthe digitaldivide, information access expectedpublic gains andfulŽ lment ofdemocratic pos- sibilities, instead ofthe spreadingof access andpower, however,the gains,the power,and the access werecon- TheInternet,no longermerely ane-mail andŽ le-sharing solidated inthe handsof afewdominant individuals and system, has emergedas adominantinteractive medium. institutions. Inthe wordsof acclaimed media critic Robert McChesney (1999, p. 1), Received17 July1997; accepted 24 November1998. TheAmerican media system is spinning out of control Weareindebted to many colleagues for commenting on andques- ina hyper-commercializedfrenzy. Fewer than ten transna- tioningearlier versions of this article: audiences at the conference tionalmedia conglomerates dominate much of our media; “ComputerEthics: A PhilosophicalEnquiry, ”London;members of the fewerthan two dozen account for the overwhelming major- seminarsat the Kennedy School of Government,Harvard University, ityof our newspapers, magazines, Ž lms,television, radio, and andtheCenter for Arts and Cultural Policy Studies, Princeton Univer- books.With every aspect of ourmedia culture now fair game sity;Steven T epper,Eszter Hargittai, Phil Agre; and Rob Kling and re- forcommercial exploitation, we can look forward to the full- viewers for TheInformation Society .Wearegrateful to LeeGiles, Brian scalecommercialization of sports, arts, and education, the LaMacchia,Andrea LaPaugh (andmembers of her graduate seminar ), disappearanceof notionsof publicservice from public dis- andAndrew T omkinsfor technical guidance, and to our able research course,and the degeneration of journalism,political cover- assistantsMichael Cohen and Sayumi T akahashi.H. Nissenbaumac- age,and children’ s programmingunder commercial pressure. knowledgesthe invaluable support of theNational Science Foundation throughgrant SBR-9806234. McChesney’s work (1993, 1997b) traces—in verysub- Addresscorrespondence to Helen Nissenbaum, University Center tle andconvincing detail— how commercial interests were forHuman V alues,Princeton University, Princeton, NJ 08544-1013, woveninto the veryŽ berof the modernmedia networks USA. E-mail:[email protected] throughlegislation, market mechanisms, andthe like. 169 170 L.D.INTRONA AND H.NISSENBAUM

Thesemoves progressively pushed out and silenced the see irrefutable signs ofgradual centralization andcom- publicservice agenda,which was verycentral tothe vision mercialization ofguidingforces. Like McChesney, we are ofthe early pioneersin the Želd—McChesney’ s (1993) particularly concernedwith the waythese competingin- historical accountof radio is verytelling in this regard. terests (centralized commercial vs.decentralized public ) His central argument,historically grounded,is that the may,early on,be woven in, or out, of the veryŽ berof fundamentalcourse of media is determinedprimarily by media networks.Search engines constitute aparticularly howthey’ re ownedand operated .Most U.S.communica- telling venuefor this competition.And prospects, as seen tion media—going back to AMradioin the 1920s—have fromthe perspective ofthe time ofwriting this article, do followedthis path: AtŽrst, whenthey do notseem com- notlook good for broad public interests. mercially viable,they are developedby the nonproŽt, non- Searchengines constitute apowerfulsource of access commercial sector. Whentheir proŽt-making potential andaccessibility within the Web.Access, alreadya thorny emerges,however, the corporatesector starts colonizing issue, is the subject ofmuch scholarship andresearch the media,and through a variety ofmechanisms, usually (Golding,1994; Hoffman & Novak,1998; Pollack, 1995; its dominanceof politicians, muscles outthe rest andtakes Schiller, 1995 ),as well as alengthyreport by the Na- over.McChesney argues that this pattern is seen inthe tional Telecommunicationsand Information Administra- cases ofFMradio,in UHFtelevision, andto some extent tion (NTIA), Falling Throughthe Net .Focusingon social, in satellite andcable. economic,and racial factors, these worksshow how access Onthe prospects ofthe Internet,there are divergentpre- tothe Webis preconŽgured in subtle butpolitically impor- dictions. Some,like DanSchiller (1995) andMcChesney, tant ways,resulting in exclusionof signiŽ cant voices.It inuenced by their knowledgeof othermedia, anticipate is notenough, however, to worryabout overcoming these asimilar narrowingof prospects forthe Internet.They traditional barriers, to focusonly on the grantingof entry pointto the commitment ofthe UnitedStates to private tothe media space ofthe Web.It is notenoughif, as wear- ownershipof communications technology as the single gue,the space itself is distorted in favorof those wealthyin most important andconsistent historical policyposition technical oreconomicresources throughthe mechanism that inuenced the courseof telecommunications devel- ofbiased search engines.The politics ofsearch engines opment.And this same commitment is clearly evidentin thus represents the broaderstruggle tosustain the demo- the rhetoric ofthe political foundationsof the Internet, cratic potential oftraditional media,the Internet,and the namely,the fact that ofŽve“values”that Vice-President WorldWide W ebin particular. GoreidentiŽ ed as onesthat shoulddeŽ ne and guide the Ina statistical studyof Websearch engines,S. Lawrence developmentof the GlobalInternet Infrastructure (GII), andC. L.Giles (1999) estimated that noneof the search the Žrst onelisted was “private investment” (OfŽ ce of the enginesthey studied, taken individually, index more than Vice President, 1995 ).Schiller asks, “Whatis the likeli- 16%of the total indexableW eb,which they estimate to hoodof robustadherence to . . . elemental democratic pre- consist of800million pages.Combining the results ofthe scription, whenthe character ofthe networkdevelopment search enginesthey studied, they estimated the coverage is nowall-too-evidently tobe given mainly as afunction toincrease toapproximately 42%. This conŽrms the prim- ofunrestrained corporate ambition andprivate design?” itive impressions ofmanyusers, namely,that the Webis (Schiller, 1995,p. 6 ).Others,like MarkPoster (1995), almost inconceivablylarge, and also that search engines offera contrasting view,arguing that the distinctly “post- onlyvery partially meet the desperate needfor an effective modern”nature of the Internet,with its capacity to dis- wayof Žndingthings. 1 Whenjudging what the producers seminate material rather thancentralize it, will discourage ofsearch engineshave accomplished so far,optimists, fo- the endowmentof authority— both academic andpoliti- cusingon the half-full portionof the cup,may legitimately cal. Its development,therefore, is unlikelyto mirror that marvel atthe progressin Websearch technologiesand at ofpreviousmedia. the sheer bulkof pagesthat are successfully found.In this Thebroader debate about the dualpossibilities of article, however,we are concernedwith the half-empty media—tobe democratizingor tobe colonized by special- portionof the cup:the portionsof the Webthat remain ized interests at the expenseof the publicgood— inspires hiddenfrom view. andmotivates this article onthe politics ofsearch engines. Thepurpose of this article is not,however, to bemoan Thegeneral position we defend,and illustrate inthis one the generaldifŽ culties ofbuildingcomprehensive search case, is that althoughthe Internet andthe Weboffer excit- engines,nor to highlightthe technologicaldifŽ culties that ingprospects forfurthering the publicgood, the beneŽts must surely impose limits onthe rangeof scopeand cov- are conditional,resting precariouslyon a numberof po- eragethat eventhe best search enginescan achieve. Our litical, economic,and technical factors. FollowingPoster, concern,rather, is withthe waysthat developers,design- weare buoyedby clear instances wherethe Weband In- ers, andproducersof search engineswill direct these tech- ternet haveserved broad political andends. But wealso nologicallimitations, the inuences that maycome into POLITICS OFSEARCH ENGINES 171 playin determiningany systematic inclusions andex- signers ofthe search engines’databases nor,explicitly, by clusions, the wide-rangingfactors that dictate systematic some otherauthority, but rather theyare “deduced”from prominencefor some sites, dictating systematic invisibil- Webpages themselves inthe process ofindexing.In a par- ity forothers. These, we think,are political issues. 2 They ticular Webpage a keywordcan be anyof the following: are important becausewhat people (the seekers) are able Actual keywordsindicated bythe Web-pagede- to Žndonthe Webdetermines whatthe Webconsists of • signer in anHTML metatag as follows: . • All orsome ofthe wordsappearing in the title ABRIEFAND SELECTIVETECHNICAL OVERVIEW that is indicated bythe HTML. search enginesis beyondthe scopeof this article, 3 we high- TheŽrst Xwordsin a Webpage (possiblyexclud- light aspects ofsearch enginesthat weconsiderrelevant • ingstop words 8 ). to ourdiscussion oftheir politics. Webrie y discuss the All the wordsin the Webpage (possiblyexcluding natureof the connectionbetween search enginesand W eb • stop words). pages,the process bywhichthis relationship is established, andhow this relationship affects the producers (or own- Most search enginesuse at least some ofthe wordsin ers) ofWebpages wishing to havetheir pagesrecognized. the title tag ofthe Webpage as the relevant keywords Web-pageproviders seeking recognition from search en- forindexing purposes. 9 It is obviouslyimportant forW eb- gines fortheir Webpages must focuson two key tasks: pageproducers as well as seekers toknow what words on a (a) beingindexed and (b) achievinga rankingin the top particular Webpage are seen askeywordsby the indexing 10–20search results displayed. 4 software ofsearch engines.Thus, one might naturally ask: Howdoes a search enginego aboutcreating its database andwhat does it store in it? OnBeingIndexed Theanswer to this questiondepends on which of ba- Havinga pageindexed, the essential Žrst stage ofbe- sically twocategories (andwithin these categories, the ingrecognized by search engines,is extremely important. furthersubcategories ) the search engineŽ ts. Onecategory Withoutmuch exaggeration one could say that toexist is includes directory-basedsearch enginessuch as Yahoo! tobe indexed by a search engine.If aWebpage is not andAliweb. In this category,the vast majority ofthe pages in the indexof a search engine,a personwishing to ac- indexedare manuallysubmitted to the search engines’ed- cess it must knowthe complete UniformResource Loca- itors byWebmasters (andother creators ofWebpages ).10 tor (URL)—also knownas the Webpage address— such Theother category includes search enginesthat automat- as http://is.lse.ac.uk/lucas/cepe98.htmlfor the CEPE’98 ically harvest URLsby means ofspiders (also referredto conference. 5 Since there is norigid standardfor produc- asrobotsor softbots ).Amongthe most well-knownsearch ingURLs, they are notobvious or even logical in the enginesŽ tting this categoryare Alta Vista, Lycos,and waywe tend to thinkthat the addresses ofour physi- Hotbot. cal homesare logical. 6 Sometimes the Internet domain- Inthe case ofdirectory-basedsearch engines,Web-page namestructure mayhelp, such as “ac.uk”or “edu”for an creators submit URLsto the search enginesfor possible academic institution in the UnitedKingdom or United inclusion into their databases. Ifyouwantedyour page rec- States. However,for most searches wedo not have any ognizedby Yahoo!,for example, you would submit your idea ofthe URLsinvolved. 7 URLandbackground information to ahumaneditor, who This is wheresearch enginesenter the picture.They wouldreview the pageand decide whether or nottosched- create amapof the Webby indexingW ebpagesaccording ule yourpage for indexing. If yourpage is scheduledfor to keywordsand then create enormousdatabases that link indexing,it wouldbe retrieved bythe indexingsoftware, pagecontent to keywordsto URLs.When aseeker ofinfor- whichwould parse 11 the pageand index it accordingto mation submits akeyword (or phrase)—presumably,one the keywords (content) foundin the page.For directory- that best captures his orher interest— the search- basedsearch engines,therefore, human gatekeepers hold enginedatabase returns to the seeker alist ofURLslinked the keyto inclusion intheir indexeddatabases. Atthe time to that keyword,ideally includingall those that are relevant ofthe writing this article, there is aconsiderablebacklog, tothe seeker’s interest. It is important to notethat search so this process cantake upto six monthsfrom the time of enginesuse the notionof a keyword (i.e.,that whichis submission to the time ofinclusion. indexedand henceused for searching ) in arather minimal Webowners wishing to havetheir pagesindexed must sense. Keywordsare notdetermined a priori bythe de- surely wonderwhat criteria these humaneditors use to 172 L.D.INTRONA AND H.NISSENBAUM decidewhether or notto indextheir pages.This is amajor enginecrawl algorithms means wecan only try to infer boneof contention,especially foranyone contesting these the character ofthese algorithms fromsearch enginese- decision criteria. WithY ahoo!,for example, representa- lection patterns—an inexact exercise. tives say that theyuse criteria ofrelevancy (Phua, 1998). Wehavelearned something of the natureof spider algo- Theexact natureof these criteria, however,is notwidely rithms froma paperon efŽcient crawlingby Cho, Garcia- knownor publiclydisseminated and,evidently, these cri- Molina,and Page, 12 presentedat the WWW7conference teria are notconsistently appliedby the variouseditors. As (Choet al., 1998 ).This paper,which discusses commonly aresult, youmay have your page rejected (withoutnotiŽ - usedmetrics fordetermining the “importance”of aWeb cation) andwould not knowwhat to doto get it accepted. pageby crawlingspiders, provideskey insights relevant to DannySullivan, the editor of SearchEngine W atch , be- the main claims ofourarticle. Because ofits signiŽcance, lieves that the base success rate forany submitted page’s wediscuss it herein some detail. Choet al. (1998, p. 1) beinglisted with Yahoo!is approximately25%. Two fac- write: tors that seem to increase the chancesof beinglisted are the numberof links (to andfrom a givensite— also referredto Givena WebpageP ,wecan deŽne theimportance of the as inlinks andoutlinks ) andhow full aparticular category page, I(P),inone of thefollowing ways . . . : happensto be.When editors feel theyneed more references 1. Similarityto a DrivingQuery Q . AqueryQ drivesthe withina category,they lower the entrybarriers. Defending crawlingprocess , and I(P) isdeŽ ned to bethetextual their approach,representatives ofY ahoo!maintain they similaritybetween P and Q . . . . list whatusers want,arguing that if users werenot Žnding 2. BacklinkCount .Thevalue of I (P) isthe number of relevant informationthey would cease usingY ahoo!. (We linksto P thatappear over the entireweb. We use IB (P) ) torefer to this importance metric. Intuitively,a page returnto this formof responselater . With Aliweb,a very Pthatis linked to by many pages is more important small site in comparisonto its competitors, users submit thanone thatis seldom referenced .Onthe web, IB (P) supplemental informationabout their Web-pagecontent isuseful for ranking query results, giving end-users andkeywords as awayof helpingthe indexingsoftware pagesthat are more likely to be of general interest. improvethe quality ofits indexingand hence provide bet- Notethat evaluating IB (P) requirescounting backlinks ter search results. Representatives ofAliweb emphasize overthe entire web. A crawlermay estimate this value that theydo notprovide comprehensive coverage; rather, with IB0 (P),thenumber of links to P thathave been theyemphasize high-qualitysearch results. Because this is seen so far. asmall site, it is still able toindex most ofits submissions. 3. PageRank. The IB(P) metrictreats all links equally. Asit becomeslarger, it may,like its competitors, needto Thus,a linkfrom the Y ahoo!home page counts the same establish criteria forinclusion andexclusion. asalinkfrom some individual’ s homepage. However , ( Beingindexed by search enginesthat automatically har- sincethe Y ahoo!home page is more important it has amuchhigher IB count ),itwouldmake sense to value ( vest URLsis amatter ofbeing visited bya spider also thatlink more highly. The P ageRankbacklink metric, called robot,crawler, softbot, agent, etc. ).Spiders usually IR(P),recursivelydeŽ nes the importance of a page start crawlingfrom a historical list ofURLs, especially tobe the weighted sum of the backlinks to it . Such documentswith manylinks elsewhere,such as server lists, ametrichas been found to be veryuseful in ranking “What’s New”pages, and other popular sites onthe Web. resultsof user queries [Page 1998.2]. We use IR0 (P) Softwarerobots crawl the Web—that is, automatically tra- forthe estimated value of IR (P) whenwe have only a verse the Web’s hypertextstructure— Žrst retrieving adoc- subsetof pages available. umentand then recursively retrieving all documentsthat 4. LocationMetric . The IL(P) importanceof page P isa are referenced (linkedby otherURLs ) in the original doc- functionof itslocation, not of itscontents. If URL u ument.W ebowners interested inhaving their pagesin- leadsto P,thenIL (P) isa functionof u. Forexample, dexedmight wish theyhad access to details concerning URLs endingwith “ .com”may be deemed more useful thanURLs withother endings, or URLs containing the routes spiders followwhen they crawl, which sites thestring “ home”may be moreof interestthan other theyfavor, which they visit andhow often, which not, and URLs. Anotherlocation metric that is sometimes used soforth. This, however, is acomplicated technical subject, considersURLs withfewer slashes more useful than andthe details are steadfastly guardedas trade secrets by thosewith more slashes. Allthese examples are local the respective search enginecompanies. From our experi- metricssince they can be evaluated simply by looking enceand discussions withthose involvedin the Želd,we atthe URLs.” [emphasisadded ] wouldcontend with some certainty that spiders are guided byaset ofcriteria that steer them ina systematic wayto The Similarity toa Driving QueryQ metric uses a select certain types ofsites andpages and not select oth- queryterm orstring (Q)—suchas “holidaycottages, ”for ers. However,the blackouton information about search example—as the basic heuristic forcrawling. This means POLITICS OFSEARCH ENGINES 173 that the spider doesnot need to make a decision about products,free software utilities, access to exclusive infor- importancesince it will bedirected in its search bythe mation,and so forth.Obviously, not all Web-pagecreators querystring itself. Forour discussion, this metric is of haveequal access to the resources orthe incentive to in- minorsigniŽ cance. 13 Thereal issue emerges whenthe duceothers to link to them. crawlingspider must “decide”importance without the The LocationMetric useslocation informationfrom the use ofa submitted queryterm. This is wherethe other URLtodetermine “nextsteps” in the crawl.“ Forexam- metrics playthe dominantrole. The Backlink metric uses ple,URLs ending with ‘.com’may be deemed more use- the backlink (or inlink) countas its importanceheuristic. ful thanURLs with otherendings, or URLscontaining the Thevalue of the backlinkcount is the numberof links string ‘home’may be moreof interest thanother URLs. ” to the pagethat appearover the entire Web—for exam- Eventhough the authorsdo notindicate whatthey see as ple,the numberof links overthe entire Webthat refer to moreimportant, one can assume that these decisions are http://www.ibm.com.The assumption hereis that “apage madewhen crawl heuristics are set fora particular spider. that is linkedto by many [other] pagesis moreimpor- It maytherefore be ofgreat signiŽcance “ whereyou are tant thanone that is seldom referenced.”Obviously,this located”as to howimportant youare seen to be.With the is averyreasonable heuristic. 14 Weknowfrom academic URLas the basis ofdecision making,many things can aid research that it is wise to lookat the “canonical”works youin catching the attention ofthe crawlingspider, such that are referredto— or cited in academic language—by ashavingthe right domainname, being located in the root manyother authors. We know also, however,that notall directory,and so forth.From this discussion oncrawling topics necessarily havecanons. Furthermore, although in metrics wecanconclude that pageswith manybacklinks, some Želds asmall numberof citations maymake a par- especially backlinksfrom other pages with highbacklink ticular worka canon,in otherŽ elds it takes avast num- counts,which are at locations seen asuseful orimportant berof citations to reachcanonical status. Thus,the Back- to the crawlingspider, will becometargets forharvesting. link heuristic wouldtend to crawl andgather the large Anothercriterion that seems to guidespiders is breadth topics/Želds (suchas “sharewarecomputer games” ) since ordepthof representation.If aspider’s algorithm favors aneven relatively unimportantsite in this bigŽ eld will breadth (rather thandepth ),it wouldvisit moresites (or beseen as moreimportant— have relatively moreback- hosts) butindex them onlypartially. Inthe case ofbig links orinlinks—than an actually important site in asmall sites suchas America Online (AOL),Geocities, andso Ž eld (suchas “the local communityservices information” forth,spiders will indexthem at arate ofapproximately page),whichwould have relatively less backlinksor in- 10–15%.15 If yoursite is hostedon AOL or anotherbig links. Theessential pointis that the large Želds determine site, there is agoodchance that it will notbe included. the measure,or threshold,of importance—through sheer Anotherreason that asite, andso all the pageson that volumeof backlinks—in waysthat wouldtend to pushout server,may be excludedfrom search enginedatabases is the equallyimportant small Želds. (Wereturnto this issue that the owner/Webmaster ofthat server has excludedspi- later, in ourmarket discussion. ). ders throughthe robotexclusion standard by means ofa With the PageRank metric, this problemis exacerbated. “robots.txt”Ž le. 16 This is oftendone because requests for Instead oftreating all links equally,this heuristic gives pagesfrom spiders maysigniŽ cantly increase the loadon prominenceto backlinksfrom other important pages— aserver andreduce the level ofservice toall otherusers. pageswith highbacklinkcounts. Thus, “ since [a link from] CNN,forexample, excludes all spiders fromits site, 17 the Yahoo!home page is moreimportant (it has a much as domany sites that offerfree Web-pagespace. 18 It is higher IB [backlink] count),it wouldmake sense to value also important to notethat the harvestingspiders ofthe that link morehighly. ”Inthe analogyof academic papers, search engineswe lookedat process onlyHTML Žles and ametric like this wouldimply that aparticular paperis inparticular HTMLtags. If important informationon your evenmore important if referredto byothers whoare al- Website is in otherformats, suchas Acrobat (pdf ) Ž les or readyseen asimportant—by othercanons. More simply, representedby a graphic (gif ) Žle, this informationcould youare important if others whoare alreadyseen as impor- belost in the indexingprocess. 19 tant indicate that youare important.The problemwith the Havingsaid all ofthis, it oughtto be acknowledged Backlink and PageRank metrics is that theyassume that that most spider-basedsearch enginesdo also allow au- backlinksare areliable indication ofimportanceor rele- tonomoussubmissions byW ebmasters/designers.Soft- vance.In those cases whereauthors of pagescreate links wareis available that automatically generates the required toother pages they see as valuable,this assumption maybe electronic formats andfacilitates submission to anum- true.There are, however, many organizations that actively berof search enginessimultaneously. Usingthis routehas cultivate backlinksby inducingW eb-pagecreators to add hadvery mixed results, accordingto the Webmasters we alinkto their pagethrough incentives suchas discounts on spoke to. 174 L.D.INTRONA AND H.NISSENBAUM

OnBeingRanked tinue to struggle with the issue ofwhat precisely to ex- tract fromthe enormousbulk of possibilities fora given Indexingis butone hurdleto clear forthe creators ofWeb search.20 pageswho strive forrecognition through search engines. Most rankingalgorithms ofsearch enginesuse boththe Havingbeen successful inthe indexinggame, their con- positionand the frequencyof keywordsas abasis fortheir cernshifts toranking. Many observe that to benoticed rankingheuristics (Pringle et al., 1998 ).Accordingly,a byapersondoing a search,a Webpagehas to beranked documentwith ahighfrequency of keywordsin the be- amongthe top10 to 20listed ashits. Because most search ginningof adocumentis seen as morerelevant (relative enginesdisplay the 10most relevant hits onthe Žrst page tothe keywordentered ) thanone with a lowfrequency ofthe search results, Webdesigners jealously covetthose lowerdown in the document.Other ranking schemes, like 10or20top slots. Theimportance of rankingis regularly the heuristic usedby Lycos,are basedon so-called inlink discussed byleadingauthors in the Želd ofW eb-site pro- popularity.The popularity score ofaparticular site is cal- motion: culated basedon the total numberof othersites thatcontain ( Thereis competition for those top tenseats. There is seri- links to thatsite also refer to backlinkvalue, discussed ear- ouscompetition. People are trying to take away the top spots lier).Highlink popularity leads toan improvedranking. everyday. They are always trying to Ž ne-tuneand tweaktheir Aswith the crawl metrics discussed earlier, onesees the HTML codeand learn the next little trick. The best players standardor thresholdof relevancebeing set bythe bigsites evenknow dirty ways to “bumpoff ” theircompetition while at the expenseof equallyrelevant small sites. protectingtheir own sites (Anderson& Henderson,1997 ). Thedesire andbattle forranking have generated a Želd ofknowledgecalled search enginedesign, which teaches Althoughwe havenot foundlarge-scale empirical stud- howto designa Webpagein orderto optimize its rank- ies measuringthe effects ofranking on the behaviorof ingand combines these teachings with software to as- seekers, weobserve anecdotally that seekers are likely sess its rankingpotential. Onone end of the spectrum, to lookdown a list andthen cease lookingwhen they practices that makereasonable use ofprima facie rea- Žnda “hit.”A studyof travel agents usingcomputer- sonableheuristics helpdesigners tooptimize their Web ized airline reservations systems, whichshowed an over- pages’expected rankings when they are legitimately rel- whelminglikelihood that theywould select aight from evantto the personsearching. On the otherend of the the Žrst screenful ofsearch results, is suggestive ofwhat spectrum,some schemes allow Webdesigners to manipu- wemight expectamong W ebusers at large (Friedman & late, ortrick, the heuristics—schemes suchas relevancy (or Nissenbaum,1996 ).Indeed,if this werenot the case it keyword) spamming,21 whereW eb-pagedesigners “trick” wouldbe difŽ cult to see whyW ebmasters are goingto all the rankingalgorithm into rankingtheir pageshigher than the effort to get into the Žrst screen—and there is signiŽ- theydeserve to beranked by means ofkeyword stuff- cant evidencethat theydo, indeed, take it veryseriously. ing,invisible text, tiny text, andso forth.Such spam- Nowit maybe that it is notonly the Žrst screen butthe mingactivities doublypunish the innocent.If, for exam- secondand third screen aswell. Nevertheless, eventhough ple,you design a Webpage with a fewgraphic images wecannotsay withoutfurther research exactly wherethis at the beginning,followed somewhere toward the middle line may be (andit mayvary with topic,type of searcher, with text, youwould be severely “punished”by the algo- and so forth),wecanpropose that it doesmatter whether rithm bothbecause key terms are positionedrelatively low youare in the Žrst fewscreens rather thanmuch lower downon the pageand also becauseyou would be compet- downin the order.Onecould also arguesuch a position ingfor rank with those less, as it were,scrupulous in their froman information-overload point of view; weshall not designs. pursueit here (Wurman,1989 ). Outof this strange rankingwarfare has emergedan im- Relevancyranking is anenormously difŽ cult task. Some possible situation: Search-engineoperators are loath to researchers workingon search technologiesargue that rel- giveout details oftheir rankingalgorithms forfear that evancyranking is currentlythe greater challengefacing spammers will use this knowledgeto trick them. 22 Yet, search enginesand that developmentsin technical know- ethical Web-pagedesigners canlegitimately defenda need howand sheer capacity to Žndand index sites has not to knowhow to designfor, or indicate relevancyto, the nearlybeen matched by the technical capacity toresolve rankingalgorithm so that those whosearch Žndwhat is relevancyranking. Besides the engineeringchallenges, ex- genuinelyrelevant to their searches. 23 perts must struggle with the challengeof approximatinga Beyondthe challengeof second-guessing ranking al- complexhuman value (relevancy) witha computeralgo- gorithms,there mayyet beanother, more certain, method rithm. Inother words, according to these experts,while ofgetting results. Someproducers of W ebsites pursue weseem to bemastering the coverageissue, wecon- otherways of elevating their ranking,ways that are outside POLITICS OFSEARCH ENGINES 175

TABLE 1 Summaryof criteria forindexing and ranking

Perspective Reasonfor exclusion

Searchengine: Indexing Directory-typesearch engines (1) Thehuman editor does not include your submission on thebasis of criteria notgenerally known and apparently inconsistently applied. Automatic-harvesting-typesearchengines (1) Sitenot visited because of spiderexclusion standard set by theW ebmaster. (2) Sitenot in the crawl path of the spider (notsufŽ ciently rich in backlinks ). (3) Partof alarge (often free) sitethat is onlypartially indexed. (4) Documentsdon’ t conformto HTMLstandard (pdf,gif, etc. ). Ranking (intop 10 when relevant ) (1) Didnot buy the keyword or top spot. (2) Nothigh in inlinkpopularity (from andto site ). (3) Relevantkeywords not in metatag or title. (4) Keywordspammers have pushed you down. (5) Importantparts of your title are stop words. (6) Yourpages have been altered (dumped off) throughunethical practices byyourcompetitors. Seeker:Finding appropriate content (1) Usingonly one search engine (sometimesa defaultthat user is unawareof ). (2) Inappropriateuse of searchcriteria. ofthe technical fray: Theytry tobuy them. This subject InT able 1,wesummarize the main points ofourdescrip- is anespecially sensitive one,and representatives ofsev- tion,showing some ofthe wayssearch enginedesigners eral major search enginesindignantly deny that theysell andoperators commonly make choices aboutwhat to in- search positions. Recently,however, in a much-publicized cludein and exclude from their databases. Thesechoices move,Alta Vista andDoublclick have invited advertis- are embeddedin human-interpreted decision criteria, in ers tobid for positions intheir topslots (Hansell, 1999 ). crawl heuristics, andin rankingalgorithms. Yahoo!sells prominenceindirectly byallowingWeb own- ers to payfor express indexing.This allows them to move Implications aheadin the 6-monthqueue. Another method for buying prominence—less controversial butnot unproblematic— Wemay wonder how all this affects the natureof W eb allows Webownersto buykeywords for purposes of banner users’experiences. Based onwhatwe havelearned so far ads.Amazon Books, for example, has acomprehensivear- aboutthe waysearch engineswork, we wouldpredict that rangementwith Yahoo!,and Barnes &Noblehas onewith informationseekers onthe Web,whose experiences are Lycos.If aseeker submits asearch toY ahoo!with the term mediated throughsearch engines,are most likely to Žnd “book”in it, ora term with anamethat correspondsto an popular,large sites whosedesigners haveenough technical author’s nameor booktitle inthe Amazondatabase, the savvyto succeed in the rankinggame, and especially those seeker wouldget the Amazonbanner (and URL) on his or sites whoseproprietors are able to payfor various means of hersearch result screen.This is also true formany other improvingtheir site’s positioning.Seekers are less likely companiesand products. to Žndless popular,smaller sites, includingthose that are Thebattle forranking is foughtnot only between search notsupported by knowledgeableprofessionals. 25 When a enginesand Web masters/ designers butalso amongorgani- search doesyield these sites, theyare likely to havelower zations wishingfor prominence. There is sufŽcient evi- prominencein rankings. denceto suggest that the Žerce competition forboth pres- Thesepredictions are,of course,highly general and will enceand prominence in alisting has led to practices such varyconsiderably according to the keywordsor phrases asoneorganization’ s retrieving acompetitor’s Webpage, with whicha seeker initiates asearch,and this, in turn, editing it sothat it will notdo well in the ranking,and is likely to beaffected bythe seeker’s competencewith resubmitting it as anupdated submission, oroneorganiza- search engines.The nature of experiences of information tion’s buyinga competitor’s nameas akeywordand then seekers will also varyaccording to the search engines havingthe Žrst organization’s bannerand URL displayed theychoose. Some users mayactively seek onesearch whena search is doneon that keyword. 24 engineover others, but some will simply, andperhaps 176 L.D.INTRONA AND H.NISSENBAUM unknowingly,use adefault engineprovided by institu- ketplace as amarketplace in search engineswith seekers as tions orInternet service providers (ISPs).26 We are un- the buyers.This strategy doesnot, as far as wehavebeen likely toŽ ndmuchrelief fromthese robustirregularities in able totell, alter the substantive outcomesof the particular meta search engineslike Metacrawler, AskJeeves, andDe- issues wehavechosen to highlight. brieŽng becausethey base their results onexisting search Wedonotdispute the basic fact ofthe matter, namely enginesand normally accomplish their task byrecogniz- that amarketplace forsearch engines (andseekers, if you ingonly higher-order search keysrather thanŽ rst-order will) is possible. It is also possible that sucha market, engines.27 Wenotefurther that notonly are most users reecting discrepant degreesof satisfaction byseekers, unawareof these particular biases, theyseem also to be will result insome search engines ourishingand others unawarethat theyare unaware. failing. Ourdissatisfaction with this forecast is notthat it cannotcome true butwhat it wouldmean, from the perspective ofsocial values andthe social investment in SHOULDWELETTHEMARKETDECIDE? the Internet,if it did.Why, the critic mightask, on what Readers mayŽ ndlittle to troublethem in this description grounds,would we presumeto overridethe wishes ofusers ofsearch engineproclivities. Whatwe havebefore us is an so as theyare cleanly reected in their market choices? evolvingmarketplace in search engines: Weought to let Ourreply to this challenge,which we try tokeep as free producersof search enginesdo whatthey will andlet users fromsentimental prejudices as possible, cites twomain decidefreely whichthey like best. Searchengines whose sources ofconcern.One is that the conditionsneeded for offeringsare skewedeither becausetheir selections are not amarketplace to functionin a democratic andefŽ cient comprehensiveor becausethey prioritize listings accord- wayare simply notmet in the case ofsearch engines.The ingto highest bidwill suffer inthe marketplace.And even otheris ourjudgment that Web-searchmechanisms are if theydo not, the collective preferencesof participants tooimportant to beshapedby the marketplace alone.We shouldnot be second-guessed. As the representatives of discuss eachin turn,the Žrst oneonly brie y . Yahoo!we cited earlier haveargued, users’ reactions must Avirtue frequentlyclaimed bydefenders of the mar- remain the benchmarkof quality: DissatisŽed seekers will ket mechanism is that participants are free to express their defect froman inadequate search engineto anotherthat preferencesthrough the choices theymake among alterna- doesa better jobof indexing and prioritizing. Thuswill tives. Throughtheir choices,incompetent inefŽ cient sup- the best search engines ourish; the poorones will fade pliers are eliminated in favorof competent, efŽ cient suppli- awaydue to lack ofuse. McChesney (1997b, p.12) de- ers. Asmanycritics havepointed out, however, this holds scribes acomparablefaith inthe market mechanism as true onlyfor markets in whichthose whosupply goods it appliedto traditional broadcastmedia: “Inthe United orservices havean equal opportunity to enter the market States, the notionthat commercial broadcastingis the su- andcommunicate with potential customers, andin which periorsystem becauseit embodiesmarket principles is those whodemand goods and services are fully informed closely attached tothe notionthat the market is the only andact in arational manner.Such an ideal market simply ‘democratic’regulatory mechanism, and that this demo- doesnot exist, andthis is especially soin the case ofsearch cratic market is the essence ofAmericanism, patriotism, engines. andall that is goodand true in the world.”BothMcChes- If wefocuson the demandside Žrst, wesee that most ney (1999) andSchiller (1995),however,have criticized users ofthe Weblack critical informationabout alterna- the idea that amedia market best represents democratic tives. Onlya small fraction ofusers understandhow search ideals. Inthe case ofsearch engines,we are,likewise, not engineswork and by whatmeans theyyield their results. It optimistic aboutthe promise ofdevelopment that is shaped is misleading to suggest that these users are meaningfully onlyby a marketplace. expressingpreferences or exercising free choicewhen they As anyonewho has usedsearch enginesknows, the select fromthe alternatives. Thoughwe lack systematic dominantsearch enginesdo not charge seekers forthe empirical evidence,the anecdotalresults ofasking peo- search service. Rather,the arrangementresembles that ple whythey use orprefer one search engineto others ofcommercial television whereadvertisers paytelevision is some versionof “It Žndswhat I’ mlookingfor” and a stations forthe promise ofviewers.Similarly, search en- shrug.Now, if oneis searchingfor a speciŽc productor gines attract paidadvertisements basedon the promise of service, it maybe possible toknow in advance how to search usage.High usage, presumably, garners advertisers determine that onehas indeedfound what one was look- andhigh charges. T osucceed,therefore, search engines ingfor. When searching for information, however, it is must establish areputationfor satisfying seekers’desires difŽ cult (if notimpossible ) to makesuch a conclusiveas- andneeds; this waythey will attract seekers in the Žrst sessment, since the locating ofinformationalso serves to place,and then will keepthem comingback. 28 As a way informone about that whichone is lookingfor. This is ofsimplifying the discussion, however,we refer to the mar- anold information-retrieval problem—often expressed as POLITICS OFSEARCH ENGINES 177

“howdo youknow what you do notknow until youknow Here,there is enormousinequality. Some enter the mar- it”— with whichinformation science scholars havebeen ket alreadywielding vastly greater prowessand economic battling formany years. It seems unlikelythat this would powerthan others. The rich andpowerful clearly canin- bedifferent forsearch engines.In fact, the partiality ofany uencethe tendencies ofsearch engines; their dollars can search attempt (evenif weassume acompetentsearcher ) (andina restricted waydo already ) playa decisive arole in will magnifythis problemin the contextof search engines. whatgets found.For example, of the top100 sites— based Notonly this, wewouldalso claim that users tendto be ontrafŽc— just 6are not.com commercial sites. 30 If we ignorantabout the inherentpartiality present inany search excludeuniversities, NASA,andthe U.S.government,this enginesearch results (as explainedearlier, in the techni- numberdrops to two.One couldreasonably argue that the cal overview ).Theytend to treat search-engineresults the UnitedNations site oughtto generateat least enoughtrafŽ c waythey treat the results oflibrary cataloguesearches. to beonthe list if weconsiderthat Amazonis in position Giventhe vastness ofthe Web,the close guardingof al- 10 and USA Today inposition 35. The cost toa search gorithms,and the abstruseness ofthe technologyto most engineof losing asmall numberof searchingcustomers users, it shouldcome as nosurprise that seekers are unfa- maybe outweighedby the beneŽts ofpanderingto “the miliar, evenunaware, of the systematic mechanisms that masses”and to entities payingfees forthe variousforms of drivesearch engines.Such awareness, we believe,would enhancedvisibility. Wecanexpect, therefore, that at least makea difference.Although here, too, we came across some drift will becausedby those wishingto befound, nosystematic empirical Žndings,we notethat inspheres which,in turn,would further narrow the Želd ofwhat is outside ofthe electronic media,people draw clear and available to seekers ofinformation, association, support, deŽnitive distinctions betweeninformation and recom- andservices. 31 mendationscoming from disinterested, ascomparedwith It maybe useful tothink of the Webas amarket of interested, sources,between impartial adviceas compared markets, instead ofas just onemarket. When we seek,we with advertisement. 29 Andanecdotalexperience bears this are notinterested in informationin general; rather,we are out,as whencustomers learnedthat AmazonBooks, for interested inspeciŽ c informationrelated toour speciŽ c example,had been representing as “friendlyrecommenda- interests andneeds. Seekers might bein the market for tions”what were in reality paidadvertisements. Customers informationabout, for example, packaged tour holidays respondedwith great ire, andAmazon hastily retreated. orcomputerhardware suppliers. Forthese markets, where Theproblem is equallycomplex on the supplyside ofthe weexpect the demandfor information to begreat, we supposedmarket. W ehavealready indicated the complex wouldexpect the competition forrecognition to be great hurdlesthat needto becleared toget listed andranked ap- aswell. Companieswould pay high prices forthe keyword propriately.They all indicate that there simply is nolevel bannersthat will ensurethem the topspot anda search playingŽ eld byanystretch ofthe imagination.It seems will generatemany hits forthe seekers. Incontrast, there clear that the “market will decide”view (problematic in are other,signiŽ cantly smaller markets—for information most cases) is extremely problematic in this context.It is abouta rare medical conditionor aboutthe services ofa also doubtfulthat this canbe resolved to the pointwhere local governmentauthority or community. the market argumentwill becomevalid. Inthis market ofmarkets, there is likely to belittle Thequestionof whether a marketplace in search engines incentive to ensureinclusion ofthese small markets and sufŽciently approximatesa competitive free market is, per- onlya small cost (inloss ofparticipation ) fortheir ex- haps,subordinate to the questionof whetherwe oughtto clusion.Although we donothave empirical evidence,we leave the shapingof search mechanisms tothe marketplace wouldexpect the law ofPareto toapply (see Sen,1985 ). in the Žrst place.We think this wouldbe abadidea. Wecould imagine that ahighpercentage of search re- Developmentsin Websearchingare shapedby twodis- quests (say 80%,for argument’ s sake ) are directed toa tinct forces.One is the collective preferencesof seekers. small percentage (say 20%) ofthe bigmarkets, which Inthe current,commercial model,search engineswishing wouldbe abundantly represented in search results. 32 Only toachieve greatest popularitywould tend to cater toma- asmall percentageof the search requests (say 20%) might jority interests. While markets undoubtedlywould force beaddressed to the large percentage (say 80%) of the adegreeof comprehensivenessand objectivity in listings, smaller markets, whichwould be underrepresented.This there is unlikelyto be muchmarket incentive tolist sites scenario wouldexplain the limited incentive forinclusion ofinterest tosmall groupsof individuals, such as indi- andrelatively lowcost ofexclusion. We Ž ndthis result viduals interested in rare animals orobjects, individuals problematic. workingin narrowand specialized Želds or,for that mat- Amarket enthusiast doesnot Ž ndthis result problem- ter, individuals oflesser economicpower, and so forth.But atic. This is exactly whatthe market is supposedto do;the popularitywith seekers is notthe onlyforce at play.The rangeand nature of choices are supposedto ebband  ow otheris the forceexerted by entities wishingto befound. in responseto the ebband  owof the wants andneeds of 178 L.D.INTRONA AND H.NISSENBAUM market participants—from varieties ofsalad dressings to mercyof the marketplace is notbasedon formal grounds— makes ofautomobiles. Nevertheless, weresist this conclu- orat least, wedonotsee them.W ebase ourcase against sionnot because we are suspicious ofmarkets ingeneral— leavingit to the market onthe particular functionthat we forcars andsalad dressings, theyare Žne—but because see search enginesserving and on the substantive vision maintaining the variety ofoptionson the Webis ofspecial ofthe Webthat wethinksearch engines (andsearch-and- importance.We resist the conclusionbecause we thinkthat retrieval mechanisms moregenerally ) oughtto sustain. We the valueof comprehensive,thorough, and wide-ranging donotargue unconditionally that the trajectory ofsearch access to the Weblies within the categoryof goodsthat enginedevelopment is wrongor politically dangerousin Elizabeth Andersondescribes inher book ValuesinEthics itself, butrather that it underminesa particular, normative andEconomic as goodsthat shouldnot be left entirely (if vision ofthe Webin society. Thosewho donotshare inthis at all) to the marketplace (Anderson,1993 ). visionare unlikelyto be convinced that search enginesare Andersonconstructs anelaborate argumentdefending different (in kind) fromsalad dressings andautomobiles. the claim that there are ethical limitations onthe scope Thecase that search enginesare aspecial, political good ofmarket normsfor a rangeof goods (andservices ). Ab- presumes that the Web,too, is aspecial good. stracting principles fromcases that are likely tobe noncon- troversial inthis regard—for example, friendship, persons, THEFUTUREOF THEWEBASAPUBLICGOOD 33 andpolitical goods (like the vote )—she thenargues that these principles applyto goodsthat are likely to bemore Thethesis wehereelaborate is that search engines,func- controversial in this regard,such as publicspaces, artistic tioningin the manneroutlined earlier, raise political con- endeavor,addictive drugs,and reproductive capacities. For cerns notsimply becauseof the waythey function, but also some goods,such as cars, bottled salad dressings, andso becausethe waythey function seems to beat oddswith on,“ unexaminedwants, ”expressedthrough the market- the compellingideology of the Webas apublicgood. This place,are aperfectly acceptable basis fordistribution. For ideologyportrays the fundamentalnature and ethos ofthe others,including those that AndersonidentiŽ es, market Webas apublicgood of aparticular kind,a rich arrayof normsdo notproperly express the valuations ofaliberal commercial activity, political activity,artistic activity,as- democratic society like ours,which is committed to “free- sociations ofall kinds,communications of all kinds,and dom,autonomy and welfare” (Anderson,1993, p. 141 ). avirtually endless supplyof information. In this regard Althoughit is notessential to ourposition that weuncrit- the Webwas, and is still seen bymany as, ademocratic ically accept the wholeof Anderson’s analysis, weaccept mediumthat cancircumvent the hegemonyof the tradi- at least this: that there are certain goods—ones that Ander- tional media market,even of governmentcontrol. soncalls “political goods,”including among them schools Overthe courseof a decadeor so,computerized net- andpublic places— that must bedistributed notin accor- works—the Internet andnow the Web—have been envi- dancewith market normsbut “ inaccordance with public sionedas agreat publicgood. Those who have held and principles” (Anderson,1993, p. 159 ). promotedthis visionover the courseof, perhaps, a decade Sustaining the 80%of small markets that wouldbe ne- havebased their claims ona combinationof what we have glected bysearch enginesshaped by market forces quali- alreadyachieved and whatthe futurepromises. Forexam- Žesas atask worthyof publicattention. Sustaining afull ple,with onlya fraction ofthe populationin the United rangeof optionshere is notthe same as sustaining afull States linkedto the Internet,Al Gore (1995) promoted rangeof optionsin bottled salad dressings orcars because the vision ofa GlobalInternet Infrastructure.This con- the formerenriches the democratic arena,may serve fun- ceptionof the great publicgood— part reality, part wishful damental interests ofmanyof the neediest members ofour thinking—has grippedpeople from a variety ofsectors, in- society, andmore (onwhichwe elaborate in the nextsec- cludingscholars, engineersand scientists, entrepreneurs, tion).Wemakepolitical decisions to save certain goods andpoliticians. Eachhas highlighteda particular dimen- that mightfall bythe waysidein apurelymarket-driven sion ofthe Web’s promise,some focusingon information, society. Inthis way,we recognizeand save national trea- some oncommunication,some oncommerce,and so on. sures, historic homes,public parks, schools, and so forth. Althoughwe cannot enumerate here all possible public Inthis spirit, wecommit to servinggroups of people, like beneŽts, wehighlighta few. the disabled,even though (and because) weknowthat a Atheme that is woventhroughout most versions ofthe market mechanism wouldnot cater to their needs. (We promise is that the Webcontributes to the publicgood by makespecial accommodationfor nonproŽ t efforts through servingas aspecial kindof publicspace. The W ebearns tax exemptionwithout consideration for popularity. ) We its characterization as publicin manyof the same ways see anequivalent need in the case ofsearch engines. as otherspaces earntheirs, andit contributes tothe pub- Inorder to makethe case convincing,however, we need lic goodfor many of the same reasons.One feature that to introduceinto the picture asubstantive claim, because pushessomething into the realm wecall publicis that it is ourargument against leavingsearch enginesfully to the notprivately owned.The W ebdoes seem tobe publicin POLITICS OFSEARCH ENGINES 179 this sense: Its hardwareand software infrastructure is not means ofaccess to vast amountsof information,the Web whollyowned by anyperson or institution or,for that mat- promises widespreadbeneŽ ts. Inthis so-called informa- ter, byanysingle nation.Arguably, it doesnot even come tion age,being among the information-richis consideredto underthe territorial jurisdiction ofanyexisting sovereign beso important that some,like the philosopherJeroen van state.34 Thereis nocentral orlocated clearinghousethat den Hoven (1994, 1998),haveargued that it makes sense speciŽes orvets contentor regulates overall whohas the toconstrue access toinformation as oneof the Rawlsian right ofaccess. All those whoaccept the technical proto- “primarygoods,” compelling any just society to guarantee cols, conformto technical standards (HTML,forexam- abasic, orreasonable,degree of it to all citizens. Growing ple),andare able to connectto it mayenter the Web.They use ofthe Webas arepositoryfor all mannerof information mayaccess others onthe Weband, unless theytake special (e.g.,government documents, consumer goods, scientiŽ c precautions,they may be accessed. WhenI post myWeb andartistic works,local publicannouncements, etc. ) lends pages,I maymake them available toany of the millions of increasing weightto this prescription.The W eb,according potential browsers,even if, like astreet vendor,I decideto to the vision,is notintendedas avehicle forfurther expand- chargea fee forentry to my page.The collaborative nature ingthe gapbetween haves and have-nots, but for narrowing ofmuchof the activity onthe Webleads to asense ofthe it (see, e.g.,Civille, 1996;Hoffman & Novak,1998 ). Web’s beingnot simply unownedbut collectively owned. Theview of the Internet as apublicgood, as aglob- TheW ebfulŽ lls some ofthe functionsof other tra- ally inclusive, popularmedium, fueled much of the ini- ditional publicspaces— museums, parks, beaches, and tial social andeconomic investment inthe mediumand schools.It serves asamediumfor artistic expression,a its supportingtechnology, convincing progressive politi- space forrecreation, and a place forstoring andexhibiting cians (orthose whowish to appearprogressive ) to support items ofhistorical andcultural importance,and it caned- it with investment andpolitical backing. 35 Thevision has ucate.Beyond these functions,the onethat has earnedit also motivatedidealistic computerscientists andengineers greatest approbationboth as apublicspace anda political tovolunteer energy and expertise towarddeveloping and goodis its capacity asamediumfor intensive communi- promulgatingthe hardwareand software, from the likes cation amongand between individuals andgroups in just ofJonathanPostel, oneof the early builders ofthe Inter- aboutall the permutations that onecan imagine, namely, net,who worked to keep its standards openand free, 36 one-to-one,one-to-many, etc. It is the HydePark Corner to professionals andresearchers volunteeringin efforts to ofthe electronic age,the publicsquare where people may wire schools andhelp build infrastructure in poorerna- gatheras amass orassociate in smaller groups.They may tions. Theseinclusive values werevery much in the minds talk andlisten, theymay plan and organize.They air view- ofcreators ofthe Weblike Tim Berners-Lee: points anddeliberate overmatters ofpublicimportance. Suchspaces, wherecontent is regulatedonly by afewfun- Theuniversality of theW ebincludesthe fact that the in- damental rules, embodythe ideals ofthe liberal democratic formationspace can represent anything from one’ s personal society. privatejottings to apolishedglobal publication. W easpeople Theidea ofthe Webas apublicspace anda forumfor can,with or withoutthe W eb,interact on all scales. By be- political deliberation has fueleddiscussions onteledemoc- inginvolved on everylevel, we ourselves form the ties which weavethe levels together into a sortof consistency,balancing racyfor some time (Abramsonet al., 1988;Arterton, 1987 ). thehomogeneity and the heterogeneity, the harmony and the Thenotion of the publicsphere as aforumin whichcom- diversity.W ecanbe involvedon apersonal,family, town, cor- municatively rational dialoguecan take place unsullied porate,state, national, union, and international levels. Culture byideology has hadone of its strongest proponentsin existsat alllevels, and we should give it aweightedbalanced Habermas (1989).Althoughthere is nouniversal agree- respectat eachlevel. 37 ment amongscholars onthe extent ofthe effect the Web mayhave in the political sphere,several contributorsto the While the promise ofthe Webas apublicspace anda debatehave cited cases in whichthe Webappears to have publicgood continues to galvanizegeneral, political, and hada decisive impact onthe outcome.Douglas Kellner commercial support,many observers and scholars have (1997) gives some examples: Zapatistas in their struggle cautionedthat the goodsare notguaranteed. The beneŽ ts against the Mexicangovernment, the TiananmenSquare ofthe vast electronic landscape,the billions ofgigabytes democracymovement, environmental activists whoex- ofinformation,and the participation ofmillions ofpeople posedMcDonald’ s throughthe McLibel campaign,and the aroundthe worlddepend on a numberof contingencies.Is- Clean Clothes Campaignsupporting attempts ofFilipino suingone such caution, Lewis Branscomb (1996) calls for garmentworkers to expose exploitative workingcondi- political effort toprotect publicinterests against encroach- tions. ingcommercial interests. Heworries aboutthe enormous Wehave not yet mentionedthe perhapsdominant rea- amountof money “ invested in the newbusiness combi- sonfor conceiving of the Webas apublicgood, namely, nations to exploit this consumerinformation market; the its functionas aconveyorof information. As apublic dollars completely swampthe modest investments being 180 L.D.INTRONA AND H.NISSENBAUM madein bringing public services tocitizens andpublic opportunitiesto locate varioustypes ofinformation,indi- institutions” (p. 27),urgingfederal, state, andlocal gov- viduals,and organizations, a narrowingof the full range ernmentto “developand realize the manynon-proŽ t public ofdeliberative as well asrecreational capabilities. If ac- service applications necessary forthe realization ofthe cess tothe Webis understoodas access byseekers toall ‘promise ofNII’ ” (p. 31). ofthese resources,then the outcomeof biased search en- GaryChapman and Marc Rotenberg,writing in 1993 gines amountsto ashrinkingof access to the Web.This onbehalfof the organizationComputer Professionals for perspective,however, does not represent all that is at stake. Social Responsibility, listed anumberof problems that Atstake is access to the Webin the shapeof those,in ad- wouldneed to besolvedbefore the National Information dition,who would like to befound,to beseen andheard. Infrastructure wouldbe capableof servingthe publicin- Marc Raboydescribes this dimensions ofthe newmedium: terest. Ofparticular relevanceto us hereis Chapmanand Rotenberg’s referenceto MarvinSirbu’ s (1992) call for Thenotion of “access”has traditionally meant different “Developmentof standardized methods for information thingsin broadcasting and in telecommunications. In the Žnding:White Pages directories, Yellow Pages,informa- broadcastingmodel, emphasis is placed on theactive receiver, tion indexes.”Withoutan effective means ofŽndingwhat onfree choice, and access refers to the entire range of prod- youneed, the beneŽts ofan information and communi- uctson offer.In thetelecommunications model, emphasis is cation infrastructure like the Webare signiŽcantly dimin- onthe sender, on the capacity to getone’ s messagesout, and accessrefers to themeans of communication. In the new me- ished.We can conjure up analogies: alibrary containingall diaenvironment, public policy will need to promote a new the printedbooks and papers in the worldwithout covers hybridmodel of communication,which combines the social andwithout a catalogue; aglobaltelephone network with- andcultural objectives of bothbroadcasting and telecommu- outa directory; amagniŽcent encyclopedia,haphazardly nications,and provides new mechanisms— drawn from both organizedand lacking a table ofcontents. traditionalmodels— aimed at maximizing equitable access to Searchengines are notthe onlyanswer to this need, servicesand the means of communication for both senders butthey still are the most prominent,the oneto which andreceivers (Raboy,1998, p. 224 ). most users turnwhen they want to explore new territory on the Web.The power, therefore, that search engineswield Thepublic good of the Weblies notmerely in its func- intheir capacity tohighlight and emphasize certain Web tioningas arepositoryfor seekers to Žndthings, but as sites, while makingothers, essentially, disappear,is con- aforumfor those with something (goods,services, view- siderable. If search enginessystematically highlightWeb points,political activism, etc. ) to offer.Thecost ofa biased sites withpopular appeal and mainstream commercial pur- search-and-retrieval mechanism mayeven be greater for pose,as well asWebsites backedby entrenched economic Web-site ownerswishing to befound—the senders.Con- powers,they amplify these presences onthe Webat the ex- sider anexample of just onetype ofcase, someoneseeking penseof others. Many of the neglectedvenues and sources informationabout, say, vacation rentals in the Fiji Islands. ofinformation,suffering from lack oftrafŽc, perhaps ac- Because onerental is all the personneeds, he or she is likely tually disappear,further narrowing the optionsto Web to lookdown a list ofoptionsand stop lookingwhen he or participants. she Žndsit. Thereis noloss tothe seeker evenif it turns out If trends in the designand function of search engines thatlowerdown on the list there are manyother candidates lead to anarrowingof options on the Web—an actual nar- meeting his orhercriteria. Theseeker has foundwhat he or rowingor anarrowingin whatcan be located—the Web she needs.Those who are notfound (becausetheir lower as apublicgood of the particular kindthat manyenvi- rankingdeprives them ofattention orrecognition ) are of- sionedis undermined.The ideal Webserves all people, fering,arguably, just as muchvalue to the seeker.Our loss, notjust some,not merely those in the mainstream. It is in this case is twofold:One is that if continuinginvisibil- precisely the inclusivity andbreadth that energizedmany ity causes optionsto atrophy,the Želd ofopportunity is to thinkthat this technologywould mean not just business thinned;the otheris that manyof those reachingout for as usual in the electronic realm, notmerely anewtool for attention orconnectionare notbeing served by the Web. entrenchedviews andpowers.The ideal Webwouldextend If search mechanisms systematically narrowthe scopeof the possibilities forassociation, wouldfacilitate access to whatseekers mayŽ ndand what sites maybe found,they obscuresources ofinformation,would give voice to many will diminish the overall valueof the Webas apublicforum ofthe typically unheard,and would preserve intensive and andas abroadlyinclusive sourceof information. broadlyinclusive interactivity. Manyhave observed that torealize the visionof the Inconsidering the effects ofa biased indexingand re- Webasademocratizingtechnology or, more generally, as a trieval system, ourattention Žrst was drawnto the seekers. publicgood, we musttake the questionof access seriously. It is fromthe perspective ofseekers that wenotedthe sys- Weagreewith this sentiment butwish to expandwhat the tematic narrowingof Webofferings: Therewould be fewer term covers.Access involvesnot merely acomputerand POLITICS OFSEARCH ENGINES 181 anetworkhookup, as some haveargued, nor, in addition, wrote: “It is notmypurpose to write offwhat has beenor is the skills andknow-how that enableeffective use.Access beingachieved, but deŽ nitely todemand more. ”Wetake implies acomprehensivemechanism forŽ ndingand being asimilar stance in responseto ourstudy of Websearch found.It is in this contextthat weraise the issue ofthe engines. politics ofsearch engines—apolitics that at present seems to pushthe Webinto a drift that doesnot resonate with one Policy ofthe historically drivingideologies. 38 Wealso believe we haveshown why arally tothe market will notsave the day, As aŽrst step wewould demand full andtruthful dis- will notensureour grandpurpose. The question of howto closure ofthe underlyingrules (oralgorithms ) governing achieveit is far harder. indexing,searching, and prioritizing, stated in awaythat is meaningfulto the majority ofWeb users. Obviously, this might helpspammers. However,we wouldargue that SOMECONCLUSIONSAND IMPLICATIONS the impact ofthese unethical practices wouldbe severely Wehave claimed that search-enginedesign is notonly a dampenedif bothseekers andthose wishingto be found technical matter butalso apolitical one.Search engines wereaware of the particular biases inherentin any given are important becausethey provide essential access to the search engine.W ebelieve,on the whole,that informing Webboth to those with somethingto say andoffer and to users will bebetter thanthe status quo,in spite ofthe those wishingto hearand Ž nd.Our concern is with the difŽculties. Thosewho favor a market mechanism would evidenttendency of manyof the leadingsearch enginesto perhapsbe pleased tonote that disclosure wouldmove giveprominence to popular, wealthy, and powerfulsites at uscloser to fulŽlling the criteria ofan ideal competitive the expenseof others.This theydo through the technical market in search engines.Disclosure is astep in the right mechanisms ofcrawling,indexing, and ranking algorithms direction becauseit wouldlead to aclearer graspof what aswell as throughhuman-mediated trading of prominence is at stake in selecting amongthe varioussearch engines, fora fee.As longas this tendencycontinues, we expect whichin turnshould help seekers to makeinformed de- these political effects will becomemore acute asthe Web cisions aboutwhich search enginesto use andtrust. But expands. disclosure byitself maynot sustain andenhance Web of- Weregret this tendencynot because it goesagainst our ferings inthe waywe wouldlike it to—that is, byretaining personalnorms of fair playbut because it underminesa transparencyfor those less popularsites to promoteinclu- substantive ideal—the substantive visionof the Webasan siveness. inclusive democratic space.This ideal Webis notmerely a Themarketplace alone,as wehaveargued, is notade- newcommunications infrastructure offeringgreater band- quate.As apolicystep, wemight,for example, consider width,speed, massive connectivity,and more, but also a publicsupport for developing more egalitarian andin- platform forsocial justice. It promises access tothe kind clusive search mechanisms andfor research intosearch ofinformationthat aids upwardsocial mobility; it helps andmeta-search technologiesthat wouldincrease trans- peoplemake better decisions aboutpolitics, health,educa- parencyand access. Evidently,if weleave the task ofchart- tion,and more. The ideal Webalso facilitates associations ingthe Webinthe handsof commercial interests alone,we andcommunication that couldempower and give voice to will merely mirror existing asymmetries ofpowerin the those who,traditionally, havebeen weaker and ignored. verystructure ofthe Web (McChesney,1999 ). Although Adrift towardpopular, commercially successful institu- these andother policies couldpromise afairer representa- tions, throughthe partial viewoffered by search engines, tion ofWebofferings, a secondkey lies in the technology seriously threatens these prospects.Scrutiny and discus- itself. sion are important responses tothese issues butpolicy and action are also needed—to Žll that half-emptyportion of Values in Design the cup.We offer preliminary suggestions,calling fora combinationof regulation through public policy as well Philosophersof technologyhave recognized the intricate asvalue-consciousdesign innovation. connectionbetween technology and values— social, polit- Thetenorof oursuggestions is enhancement.W edonot ical, andmoral values. 39 Theseideas— that technological see that regulatingand restricting developmentof commer- systems mayembed or embody values— resonate in so- cialsearch enginesis likely toproduce ends that wewould cial andpolitical commentaryon informationtechnology value—as it were,siphoning off from the half-full portion. written byengineers as well as byphilosophers and ex- This courseof action is likely to beneither practically ap- perts incyberlaw (see, e.g.,Friedman, 1997; Lessig, 1999; pealingnor wise, andmight smack ofcultural elitism or Nissenbaum,1998 ).Translating these ideas intopractice paternalism. AmartyaSen (1987, p. 9),commentingon implies that wecan build better systems—that is to say, existing schools ofthought within the Želd ofeconomics, systems that better reect important social values—if we 182 L.D.INTRONA AND H.NISSENBAUM buildthem with anexplicit commitment to values.With skillis necessary to get what you want.W ereturnto thisissue when we this article, the commitment wehope to inspire among discussthe market argument for the developmentof searchengines. the designers andbuilders ofsearch enginetechnology is 2.Winner, L. 1980.Do artifacts have politics? Daedalus 109:121– acommitment to the valueof fairness as well as to the 136. suite ofvalues representedby the ideologyof the Webas 3.For those interested in more detail, the W ebsite http:/ /www. apublicgood. searchenginewatch.comis a goodplace to start. 4.W earethinking here of thetop 10 to20 whenit is a matterof Twotechnical approachesthat appearto beattracting actualrelevancy. We later discuss the issue of spamming. interest are notwithout drawbacks. One would increase 5.One could argue that it is also possible for a Webpage to be segmentation anddiversiŽ cation. Search engines would foundthrough portal sites, which are increasingly popular, though as a becomeassociated withparticular segments ofsociety— matterof fact,we think it wouldbe highly unlikely that a linkwould be bordersdrawn perhaps according to traditional categories establishedthrough a portalsite if it does not meetthe indexing criteria (sports, entertainment, art, andso forth ).Aproblemwith forsearch engines. segmentation overall,however, is that it couldfragment 6.W erealizewe have not listed all the means through which the veryinclusiveness anduniversality ofthe Webthat we pagesmay be found. For example, one may access a pagethrough value.The W ebmayeventually merely mirror the institu- anoutlink from another page. The problem with such means is that tions ofsociety with its baggageof asymmetrical power theydepend on somewhat unpredictable serendipity. One needs also structures, privilege,and so forth. toadd that increasing numbers of alternativesare emerging as viable options,such as portalsites and keyword retrieval via Centraal’ s Real Theother approach is todevelop individualized spiders Name system (http://www.centraal.com ).Nevertheless,the majority of that gooutand search forpages based on individual cri- thosewho access the W ebcontinue to do it through search engines. teria, buildingindividualized databases accordingto in- Thereis no reasonto believethat this would change in the foreseeable 40 dividualneeds. Thereis, however,a signiŽcant “cost” future. in automatic harvestingvia spiders that eventhe existing 7.W enote,for readers who are aware of the debate currently populationof spiders imposes onsystem resources; this ragingover domain names, that an effectivesystem of searchand re- has alreadycaused concern (see Kostner,1995 ). trievalis a constructiveresponse to the debate and would lessen the Thereis muchinteresting workunder way concerning impactof whateverdecisions are made. W earguethat domain names the technologyof search enginesthat could,in principle, areimportant in inverse proportion to the efŽ cacy of availablesearch help: forexample, improving the wayindividual pages mechanisms,for if individuals and institutions can easily be found on indicate relevance (also referredto as metadata ) (see Mar- thebasis of contentand relevancy, there is less at stakein the precise ) 41 formulationof their domain names. In other words, a highlyeffective chiori,1998 ,reŽning overall search enginetechnology, indexingand retrieval mechanism can mitigate the effects of domain- andimproving Web resource presentation andvisualiza- nameassignments. tion (see Hearst, 1997 ) andmeta-search technology (see 8.A stopword is afrequentlyoccurring word such as the,to, and Lawrence& Giles, 1998 ).Althoughimprovements like wethat is excluded because it occurs too often. Stop words are not these might accidentally promotevalues, they hold great- indexed.This is not insigniŽ cant if one considers that the word “ web” est promise as remedies forthe currentpolitics ofsearch isa stopword in Alta Vista. So if youarea companydoing W ebdesign enginesif theyare explicitly guidedby values. W eurge andhave “ Webdesign” in yourtitle, you may not getindexed and will engineersand scientists whoadhere to the ideologyof beranked accordingly. the Web,to its values ofinclusivity, fairness, andscope 9.The