A Deeper Investigation of the Importance of Wikipedia Links to the Success of Search Engines*
Total Page:16
File Type:pdf, Size:1020Kb
A Deeper Investigation of the Importance of Wikipedia Links to the Success of Search Engines* Nicholas Vincent Brent Hecht Northwestern University Northwestern University Evanston, IL, USA Evanston, IL, USA [email protected] [email protected] ABSTRACT patterns, and content outcomes being imperative to the success of Google Search. Conversely, Google is known to be a key source of A growing body of work has highlighted the important role that traffic – both readers and potential editors – to Wikipedia [13]. Wikipedia’s volunteer-created content plays in helping search In this paper, we replicate and extend earlier work that engines achieve their core goal of addressing the information identified the importance of Wikipedia to the success of search needs of millions of people. In this paper, we report the results of engines but was limited in that it focused only on desktop results an investigation into the incidence of Wikipedia links in search for Google and treated those results as a ranked list, i.e. as “ten engine results pages (SERPs). Our results extend prior work by blue links” [2]. As in this earlier work, we collect the first SERP considering three U.S. search engines, simulating both mobile and for a variety of important queries and ask, “How often do desktop devices, and using a spatial analysis approach designed to Wikipedia links appear and where are these links appearing study modern SERPs that are no longer just “ten blue links”. We within SERPs?” However, our work includes three critical find that Wikipedia links are extremely common in important extensions that provide us with a deeper view into the role search contexts, appearing in 67-84% of all SERPs for common and Wikipedia links play in helping search engines achieve their trending queries, but less ofen for medical queries. Furthermore, primary goals. First, we consider three popular search engines – we observe that Wikipedia links ofen appear in “Knowledge Google, Bing, and DuckDuckGo – rather than just Google. Panel” SERP elements and are in positions visible to users without Second, in light of growing search engine use on mobile devices scrolling, although Wikipedia appears less in prominent positions [29], we simulate queries from both desktop and mobile devices. on mobile devices. Our findings reinforce the complementary Third, we built software to allow us to use a spatial approach to notions that (1) Wikipedia content and research has major impact search auditing that is more compatible with modern SERPs, outside of the Wikipedia domain and (2) powerful technologies which have moved away from the traditional “10 blue links”; like search engines are highly reliant on free content created by SERPs now have “knowledge panels” in a second, right-hand volunteers. column, “featured answers”, social media “carousels”, and other elements. We are sharing our code package to support additional CCS CONCEPTS search audits taking this approach.1 • Human-centered computing~Empirical studies in collaborative Relatedly, because search engines are proprietary, opaque, and social computing and subject to frequent changes (e.g. Google very recently underwent a major redesign [21]), it is critical to validate that data KEYWORDS being used for quantitative search auditing analysis accurately Wikipedia, search engines, user-generated content, data as labor reflects how SERPs appear to users. Our spatial approach to SERP * Tis is a pre-print of a paper accepted to the non-archival track of the analysis allows us to easily visually compare the representation of WikiWorkshop at the Web Conference 2020. our SERP data to screenshots of SERPs. We find that the results identified in prior work regarding Google desktop search largely extend to other search engines and 1 Introduction to mobile devices. For instance, Wikipedia appeared in 81-84% of Previous work has highlighted critical interdependencies SERPs for our common queries, 67-72% for our trending queries, between Wikipedia and Google Search. For many query and 16-54% for our medical queries. Furthermore, using our spatial categories, Wikipedia links appear more than any other website analysis approach, we observe that for many of these queries, on Google’s search engine results pages (SERPs) [26]. This Wikipedia appears in the prominent area of SERPs visible without suggests that Wikipedia is playing an essential role in helping scrolling, through both Knowledge Panel elements and traditional Google Search achieve its core function of addressing information blue links. However, we also observe differences between desktop needs. It also raises the stakes of Wikipedia-related research and mobile SERPs, with Wikipedia appearing less prominently for findings, with Wikipedia’s collaboration processes, contribution mobile SERPs. 1 https://github.com/nickmvincent/se-scraper N. Vincent and B. Hecht Our results have implications for a variety of constituencies. 18]. Estimated incidence rates range from 34% to 99%, depending Past work suggested that Wikipedia was highly important to on the choice of queries (a point to which we will return below). Google users: good aspects (e.g. high-quality knowledge) and bad aspects (e.g. content biases [17]) of the Wikipedia community are amplified by Google. Our results suggest that this relationship 3 Methods extends beyond just desktop users of Google, and for some cases As mentioned above, modern SERPs include SERP (e.g. DuckDuckGo’s medical SERPs), Wikipedia is even more components that are much more complicated than “ten blue important. links”. These multifaceted SERPs pose challenges to ranking- Finally, our results also reinforce the importance of based approaches used in prior work, such Vincent et al.’s study volunteer-created data to intelligent technologies. By measuring which focused on using the Cascading Style Sheets (CSS) styles to how often and where Wikipedia links appear in SERPs, we can differentiate different result types (e.g. blue link vs. Knowledge better understand how the volunteer labor underlying Wikipedia Panel vs. News Carousel) and create a ranked list of these results fuels search engines, some of the most important and widely used [26]. The critical challenge with "ranking list” approaches is that intelligent technologies. they cannot account for (1) the fact that SERP elements come in a variety of sizes and (2) the fact that SERP elements appear in a separate right-hand column. 2 Related Work In contrast to prior work, we consider every link (i.e. each This paper builds on a variety of research that has studied HTML <a> element) in each SERP and look at the coordinate for search engines and SERPs, as well as research that has specifically each link, in pixels, relative to the top left corner of the SERP, an studied the role of Wikipedia articles in search results. approach specifically designed to handle modern SERPs. There are In McMahon et al.’s 2017 experimental study [13], a browser several other high-level benefits to this approach. It is easier to extension hid Wikipedia links from Google users and the authors analyze multiple search engines (the approach requires observed a large drop in click-through rate, possibly the most substantially less hard-coded rules based on CSS styles), and we important search success metric [7]. This study, and follow up can visually validate that are the representation of SERP data we work from Vincent et al. [26], which measured the prevalence of analyze reflects the appearance of modern, component-heavy user-generated content in SERPs, were motivated by a call from SERPs. the Wikimedia Foundation to study the re-use of Wikipedia Below, we detail our data collection, validation, and analysis content [24]. pipeline, and expand on the benefits of this spatial approach to These studies are part of a broader body of search auditing SERP analysis. literature, which has used auditing techniques (i.e. collecting the outputs of search engines) to study aspects of search such as 3.1 Search Engines and Queries personalization and the components of SERPs [8, 11, 19]. Notably, In this work, we focus on three U.S. based search engines: in a study that focused on political SERPs, Robertson et al. [19] Google, Bing and DuckDuckGo. Google and Bing are the two most identified a large number of Wikipedia articles in the SERP data popular search engines in the United States, while DuckDuckGo they collected, in line with the results from Vincent et al.’s user- is the most popular privacy-focused search engine [27]. Market generated content-focused study. Related work has focused share estimates from different analytics services vary. For January specifically on the complexities of SERP elements like the 2020, analytics company StatCounter estimates Google served Knowledge Panel [12, 20]. Lurie and Mustafaraj identified that 81% of desktop queries, Bing served 12%, and DuckDuckGo served Wikipedia played a critical role in populating the “Knowledge 1.5%, and for mobile devices Google served 95%, Bing served 1.5%, Panel” shown in Google SERPs: queries about news source with and DuckDuckGo served 1.2% [23]. Data from marketing research Wikipedia articles frequently had a Knowledge Panel in the company ComScore suggests Bing has a more substantial market corresponding SERP [12]. Rothschild et al. studied how Wikipedia share: Google served about 62% of desktop search queries, while links in Google Knowledge Panel components influenced user Bing served about 25% [3]. perception of online news [20]. Following past work [8, 26], we selected categories of queries The study that we most directly replicate and extend is that are particularly important to search users. We consider Vincent et al.’s 2019 study, which used a ranking-based analysis queries from three important query categories: common queries, to examine the role user-generated content played in search trending queries, and medical queries. Common queries are made results for the desktop version of Google search, and found that very frequently.