Auditing Wikipedia’s Hyperlinks Network on Polarizing Topics

Cristina Menghini Aris Anagnostopoulos Eli Upfal DIAG DIAG Dept. of Computer Science Sapienza University Sapienza University Brown University Rome, Italy Rome, Italy Providence, RI, USA [email protected] [email protected] [email protected]

ABSTRACT readers to deepen their understanding of a topic by conveniently ac- People eager to learn about a topic can access Wikipedia to form cessing other articles." Consequently, while reading an article, users a preliminary opinion. Despite the solid revision process behind are directly exposed to its content and indirectly exposed to the the encyclopedia’s articles, the users’ exploration process is still content of the pages it points to. influenced by the hyperlinks’ network. In this paper, we shed light Wikipedia’s pages are the result of collaborative efforts of a com- on this overlooked phenomenon by investigating how articles de- mitted community that, following policies and guidelines [4, 20], scribing complementary subjects of a topic interconnect, and thus generates and maintains up-to-date and high-quality content [28, may shape readers’ exposure to diverging content. To quantify 40]. Even though tools support the community for curating pages this, we introduce the exposure to diverse information, a metric that and adding links, it lacks a systematic way to contextualize the captures how users’ exposure to multiple subjects of a topic varies pages within the more general articles’ network. Indeed, it is im- click-after-click by leveraging navigation models. portant to stress that having access to high-quality pages does not For the experiments, we collected six topic-induced networks imply a comprehensive exposure to an argument, especially for a about polarizing topics and analyzed the extent to which their broader or polarizing topic. topologies induce readers to examine diverse content. More specif- Users differently use Wikipedia, according to their information ically, we take two sets of articles about opposing stances (e.g., needs. Singer et al. [45] show that users curious about a topic explore guns control and guns right) and measure the probability that users it by browsing the encyclopedia. In fact, they rely on hyperlinks to move within or across the sets, by simulating their behavior via find correlated or complementary content to the subject of interest. a Wikipedia-tailored model. Our findings show that the networks Therefore, it is crucial to evaluate the extent to which the current hinder users to symmetrically explore diverse content. Moreover, link structure encourages users to browse related topics to develop a on average, the probability that the networks nudge users to remain more comprehensive view and perspective of a subject. This theme in a knowledge bubble is up to an order of magnitude higher than becomes particularly important when users look for an overview that of exploring pages of contrasting subjects. Taken together, on polarizing topics spanning across multiple articles. those findings return a new and intriguing picture of Wikipedia’s Wikipedia’s Neutral Point of View (NPOV) encourages editors network structural influence on polarizing issues’ exploration. to work such that articles’ content fairly and proportionately repre- sents all the significant views that have been published by reliable KEYWORDS sources on the subject [51]. Although the NPOV document gathers many suggestions to properly curate the direct content of pages, it Wikipedia, Hyperlinks Network, Polarization, User Behavior does not refer to the impact links might have in determining users’ ACM Reference Format: exposure to indirect content. Cristina Menghini, Aris Anagnostopoulos, and Eli Upfal. 2021. Auditing Suppose we consider the topic abortion. It is a broad issue, which Wikipedia’s Hyperlinks Network on Polarizing Topics. In Proceeding of The distributes across multiple articles on Wikipedia. Moreover, due to Web Conference 2021, April 19–23, 2021, Ljubljana, Slovenia. ACM, New York, its polarizing nature, it is possible to recognize pages about events, NY, USA, 13 pages. https://doi.org/10.1145/nnnnnnn.nnnnnnn people, subjects or organizations that are associated either to pro- choice or pro-life standings. Users willing to learn about abortion arXiv:2007.08197v4 [cs.SI] 8 Mar 2021 1 INTRODUCTION might access the encyclopedia to collect information and then de- velop their idea. Consider a user that enters the network reading Knowledge on Wikipedia is distributed across articles inter-connected the article Abortion-rights movement that portrays and outlines via hyperlinks. According to Wikipedia’s Linking Manual [49], "In- campaigns supporting abortion. We assume that the article’s body ternal links can add to the cohesion and utility of Wikipedia, allowing does not endorse the page’s subject due to the NPOV principle. So, we expect that the user acquires objective knowledge about Permission to make digital or hard copies of all or part of this work for personal or organizations supporting abortion and, maybe, also realizes the classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation existence of anti-abortion movements. Now, imagine that the user on the first page. Copyrights for components of this work owned by others than ACM decides to continue her exploration of the topic, and to do it, she must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, follows the hyperlinks within the current page. If the linkage to to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]. pages regarding subjects close to pro-life view is weak, our user has WWW ’21, April 19–23, 2021, Ljubljana, Slovenia little possibilities of collecting diverse views that contribute to the © 2021 Association for Computing Machinery. users’ development of a comprehensive perspective on the topic. It ACM ISBN 978-x-xxxx-xxxx-x/YY/MM...$15.00 https://doi.org/10.1145/nnnnnnn.nnnnnnn WWW ’21, April 19–23, 2021, Ljubljana, Slovenia Cristina Menghini, Aris Anagnostopoulos, and Eli Upfal follows that the lack of sufficient linkage among pages expressing on their behavioral patterns [43, 45], features determining diverse stances of a topic can be against the NPOV’s goals. the success of wikilinks [16, 31], and readers’ clickstream To our best knowledge, there are no studies that investigate data [53]. the validity of the NPOV principles concerning users’ exposure to • We find that the structure of the network facilitates users the indirect content (i.e., the one suggested by hyperlinks). Hence, to explore knowledge bubbles of homogeneous view rather analyzing Wikipedia’s links is particularly important to understand than opposing stances. Moreover, we show that readers’ if broad topics, which conceptually span across multiple articles, interest is biased toward one side of the topic based on the are effectively, proportionately, and fairly presented to readers, not internal and external traffic on Wikipedia (see Sect. 4.1.1, 5 only in terms of direct content (i.e., article’s body). and 6). Previous works addressed the issue of users’ polarization on To our knowledge this is the first work that analyzes Wikipedia’s social networks and showed that it is hard for users to interact readers’ exposure to diverse information through the link network. with content created/shared by users of opposing views [1, 9, 10, Before moving on, we want to emphasize that this work does not 19, 24]. Ribeiro et al. [42] empirically showed that the YouTube claim how the hyperlinks network should be, rather we aim to recommender system contributes to radicalize users’ pathways. study if the current connections among articles encumber users in Given the nature and role of Wikipedia as a primary source of visiting complementary pages about a polarizing topic. Also, our knowledge acquisition, the lack of broad exposure to different views conclusions come from a network-based analysis. More advanced of a topic appears to be critical to guarantee fair and balanced access investigation combining network properties and articles’ content is to a well-rounded knowledge. left out for future works. The code to replicate the paper is stored This paper provides a first observational study on Wikipedia in an anonymous folder2. that aims to quantify how the hyperlinks’ network topology can profoundly affect user exposure to diverse stances on polarizing 2 RELATED WORKS topics. Having a comprehensive view of the connections among Wikipedia pages and how they shape reader exposure to informa- We divide this paper related work in four categories: Improving tion is a difficult task to grasp for humans. Therefore, it requires Wikipedia, Navigating Wikipedia, Wikipedia Categorization and introducing algorithmic methods to audit and quantify the mutual Polarization on Social Media. level of exposure among articles of diverse content, especially for Improving Wikipedia. The scientific community proposed semi- polarizing matters. That is fundamental for the improvement of the automated procedures to improve Wikipedia’s quality. These works, encyclopedia and its role in promoting a self-critical society. check the veracity of references [18, 41], suggest articles’ structure By studying the hyperlinks network, we first aim to discover [39], look for hoaxes [30] or, recommend links [38, 54]. Although to what extent the network’s topology pushes users to explore link recommendation tools enrich the editing process, they do not diverse content, rather than keep them within knowledge bubbles1. provide editors a measure to evaluate the relationship among ar- Secondly, we aim to gain insights that may help to design a system ticles containing diverse opinions. In this work, we define such supporting editors in (1) contextualizing pages within the more metrics, Sect. 4.2. general encyclopedia’s network and (2) adding links connecting Wikipedia Navigation. The literature still lacks a model that articles of opposing/complementary views. generalizes Wikipedia’s users’ behavior. Previous studies [25, 27, In summary, this paper tackles the following research questions: 31, 46] focused on modeling and predicting human navigation in- side Wikipedia relying on traces from navigation games, i.e., Wik- RQ1 How do readers consume articles about polarizing topics? ispeedia [43, 48] and WikiGame [13, 29, 46].3 While such games (Sect. 5) provide valuable insights about how users exploit links to go from RQ2 To what extent does the hyperlinks’ network expose read- one concept to another, Singer et al. [45] and [15, 17] showed that ers to diverse information? (Sect. 6) users display different behavioral patterns depending on their in- By answering them, we make the following contributions: formation needs and the links’ position within pages. Thus, we • We initiate a discussion that aims to shed light on the role exploit the insights provided by Singer et al. [45] to define a general that the hyperlink network plays in connecting articles be- model mimicking localized and more in-depth topic exploration. longing to different categories. We focus our work on ana- We further enrich the model characterizing users’ next-link choices lyzing this phenomenon on a set of polarizing topics, such according to findings in [15, 17], Sections 4.1.2 and 4.1.3. as, abortion, guns, evolution. Wikipedia Categorization. In this work, we need to collect ar- • We define two metrics, the exposure to diverse information ticles expressing the distinct facets of a polarizing topic. Wikimedia and the (mutual) exposure to diverse information to quan- provides a supervised classifier, i.e., ORES,4 that based on features tify the strength of connections among sets of articles (e.g., derived from the articles’ text, categorize an article into a manually- pages about abortion-rights and anti-abortion). These met- designed categories taxonomy5 [3]. Alternatively, one can use topic rics quantify to what extent the network topology assists models [5, 6]. Unfortunately, none of the above approaches provide readers to visit pages of contrasting subjects and whether it does it equally for all them (see Sections 4.1, 4.1.2 and 4.2). 2https://drive.google.com/drive/folders/1CJr_YiFE2YlyAtB9yKaGe8CLwVLWx9Ta? To this end, they embed readers possible behavior, relying usp=sharing 3These games ask readers to go from one article to another using wikilinks. 1We intend as knowledge bubbles the sets of pages presenting one side of a con- 4https://ores.wikimedia.org/ tentious subject (i.e., pages about pro-life or pro-choice movements). 5https://www.mediawiki.org/wiki/ORES/Articletopic#Taxonomy Auditing Wikipedia’s Hyperlinks Network on Polarizing Topics WWW ’21, April 19–23, 2021, Ljubljana, Slovenia

and are known as wikilinks.6 This set of links includes those in the infoboxes7. Among the vertices, we identify a set of pages T ⊂ 퐴, about the different polarizing sides of a given topic. We partition T into two sets 푃 and 푃¯ (i.e., 푃 ∩ 푃¯ = ∅ and 푃 ∪ 푃¯ = T ). Each of them gather pages related to the same side of the topic. Then, we define the set of nodes N that includes all vertices at one-hop distance from the Figure 1: From Wikipedia’s graph to a topic-induced net- vertices in T . The reason we consider nodes representing pages work. The image on the left shows the original Wikipedia’s outside T is twofold: (1) We want to include in the graph those graph. On the rights we have the final topic-induced net- nodes related to the topic that do not appear in T because describe work. The dashed circles in 푊 identify the set of nodes that subjects neutral to the topic8. (2) When we will consider readers we use to build the topic-induced network 퐺. The color red exploring the network, we want to account for the possibility that refers to the set of nodes 푃. We use the blue to indicate 푃¯ they reach pages about entities of opposing opinion passing through and green and yellow for N and 푠, respectively. To keep the articles not strictly related to the topic (see Sect. 4.1). image tidy we do not specify the edges direction. To reduce the complexity of our analysis, we cluster all the pages in 푆 = 퐴 \ (T ∪ N), in one super node 푠. Note that nodes in 푆 are only connected to vertices in N. For each node 푣 ∈ N, we can have multiple edges going to 푠. We compress them in a unique edge us the requested granularity. So, we exploit the collection proce- (푣, 푠). Respectively, 푠 can point multiple times to the same node dure employed by Shi et al. [44] who needed the same data to study 푣 ∈ N. So, we compress them to a unique edge (푠, 푣). In both cases, how polarization in teams impact articles’ content about polarizing the weights of (푣, 푠) and (푠, 푣) will be the sum of weights of the topics see Sect. 3. aggregated edges. Polarization on Social Media. There is a large spectrum of Finally we built a directed weighted network 퐺 = (푉, 퐸), that we works related to detect [1, 10, 12, 19, 36], model, quantify and mit- call topic-induced network, whose set of vertices 푉 is T ∪N ∪{푠} of igate [2, 9, 21–24, 32, 34, 35, 37] polarization on social media. We cardinality 푛 + 1, and the edges 퐸 are the links connecting the pages. focus on the work of Garimella et al. [24] that better relates to our The edge weights are transition probabilities, as follows. Let 푀 be metric of exposure to diverse information (ExDIN). They introduced an (푛 + 1) × (푛 + 1) right-stochastic transition matrix associated a graph polarization measure based on random walks, i.e., Random to 퐺, that is, a matrix such that each entry 푚푖,푗 is a probability, Walk Controversy score (RWC). On a social graph, it quantifies to Í푛+1 with 푚푖,푗 = 0 if (푖, 푗) ∉ 퐸, and such that 푗=1 푚푖,푗 = 1. The entry what extent opinionated users are more exposed to their own opin- 푚푖,푗 describes the probability that being on article 푖 a reader clicks ion than the opposite, thanks to a chain of retweets (represented by page 푗. In Section 4.1.2 we propose different characterizations of the random walks). While RWC is conceived for networks of users the transition matrix. and measures the overall polarization of a graph, ExDIN works Summarizing, to extract the topic-induced network of a given on information networks and quantifies how the network’s topol- topic, we first extracted data from a complete English Wikipedia ogy impacts the users’ exposure to diverse information when they database dump.9 From this dump, we build the graph 푊 . To collect navigate the graph. the corpus of articles expressing different opinions about the topic, Cultural bias on Wikipedia. Callahan and Herring [8] showed (i.e, T ), we rely on the collection strategy adopted by the authors the presence of cultural bias in the same articles of different lan- of [44] (see Sect. 2). In particular, the subcorpus belonging to 푃 guages. Other studies highlighted differences between women and consists of all articles categorized under a Wikipedia category de- men biographies [26, 47]. These content-based analyses call for scribing a viewpoint, and its subcategories. For instance, the corpus the need for a thorough investigation of the phenomenon. To this of abortion articles consists of two subcorpora, pro-life (푃 ) and end, we decide to investigate the presence of bias in the hyperlink pro-choice (푃¯) articles. The pro-life subcorpus consists of all articles network by quantifying the diversity of pages it suggests to users categorized under the seed category “Anti-abortion movement” and browsing the network of articles. its subcategories.For instance, the article “Fetal rights” is directly un- der the seed category, whereas the article “Crisis pregnancy center” 3 DATA COLLECTION is located under the subcategory “Anti-abortion organizations.” The To audit a polarizing topic on Wikipedia, we encode it by building pro-choice corpus is collected in a similar fashion starting from the a topic-induced network. This representation embeds both the category “Abortion-rights movement.” Note that, because we want network structure and readers’ interactions with the topic. 6We exclude links within the same page. Moreover, while building the graph, we resolve all the redirects [52]. Specifically, for any given node 푟 pointed by 푢, and 3.1 Topic Induced Networks redirecting to 푣, we replace the edges (푢, 푟) and (푟, 푣) with (푢, 푣). The final effect of this operation is that we exclude all the redirecting nodes from 퐴, while retaining In this section, we explain how to build a topic-induced network. their connections to the rest of the graph. We suggest the reader to follow the process looking at Figure 1. 7An infobox is a fixed-format table usually added to the top right-hand cornerof First, we consider the directed English Wikipedia’s graph, 푊 = articles to consistently present a summary of some unifying aspect that the articles share. (퐴, 퐿). The nodes of the graph are encyclopedia’s pages classified 8For instance, articles that present an overall introduction/description of the topic. as Articles [50]. The edges represent the links connecting pages 9Unless differently specified we refer to the dump of September 2020. WWW ’21, April 19–23, 2021, Ljubljana, Slovenia Cristina Menghini, Aris Anagnostopoulos, and Eli Upfal

¯ ¯ ¯ Topic |푉 \{푠}| |푃 | |푃 | |N | |퐸| |퐸푃→푃¯ | |퐸푃¯→푃 | |퐸푃→N | |퐸푃¯→N | |퐸N→푃 | |퐸N→푃¯ | 푄 (푃, 푃) %Unreach(푃) %Unreach(푃) Abort. 56861 481 291 56093 1.9M 205 97 21843 14492 21396 29889 0.29 (0.30,0.41) 2.1 4.81 Cannabis 32743 45 231 32470 1.1M 8 6 1089 15055 656 27823 0.27 (0.14, 0.03) 11.36 3.49 Guns 65743 167 187 65393 2.5M 98 115 18342 12304 56702 16608 0.26 (0.24, 0.30) 3.63 0.00 Evolution 84788 342 1334 83113 1.99M 391 135 18289 45472 15601 58720 0.20 (0.22, 0.27) 1.69 5.62 Racism 129963 1024 1022 127953 4.8M 746 560 64359 41566 74354 58195 0.32 (0.21, 0.31) 2.72 2.55 LGBT 150563 459 640 149479 4.6M 195 143 28100 22678 92975 81706 0.34 (0.30, 0.13) 2.44 5.35 Table 1: Networks’ statistics. The notation 푄 (푃, 푃¯) ∈ [0, 1] indicates the modularity among the partitions. Higher 푄 means that connections within partitions exceed those among them.

0.75

0.50

0.25

Links' position 0.00

Pro-life Activism Control Rights Racism Support Pro-choice Prohibiiton Evol. Bio. Anti-racism Creationism Discrimination Opposite opinion Same opinion

Figure 2: Links’ position distribution within pages. Given 푃 and 푃¯, the orange boxplots show the distribution of links within pages in 푃 (resp. 푃¯) that point to articles in 푃¯ (resp. 푃). The green boxes represent link’s placement among pages only belonging to 푃 (resp. 푃¯). The value of the y-axis is the relative position re-scaled with the 푡푎푛ℎ to similarly score links at the top of the page. Higher the value, higher the position in the page is.

푃 and 푃¯ to be disjoint, articles belonging to both “Anti-abortion Topic 푃 푃¯ Seed 푃 Seed 푃¯ 10 Anti-abortion Abortion-rights movement” and “Abortion-rights movement” are assigned to N . Abortion Pro-life Pro-choice movement movement Once we have the list of pages in T , we proceed building the topic- Cannabis Prohibition Activism Cannabis prohibition Cannabis activism Gun control Gun rights induced network as described in the first part of this section. The Guns Control Rights advocacy groups advocacy groups articles we collect gather pages about different entities, such as Evolutionary Evolutionary Evolution Creationism Creationism organizations, people, events. The inclusion of a heterogeneous set biology biology of pages for each viewpoint allows to capture the different way a Racism Racism Anti-racism Racism Anti-racism Discrimination against LGBT rights LGBT Discrimination Support user can learn/know about a topic. LGBT people movement Before moving on, we need to make two remarks: (1) Throughout Table 2: The table indicates what opinion of a topic the par- the paper, when we talk about articles "expressing an opinion" or titions 푃 and 푃¯ correspond to. "describe a viewpoint" of a topic, we do not mean that they endorse the position of any subject they describe. But they objectively talk of entities that are close to one side of the issue. (2) Since subcategories are often redundant or not entirely related to the parent category, vs. anti-racism. Information about the seed categories of each topic we check them manually. In this way, we avoid cases like having are in Table 2. The full category lists and sample titles are provided articles about anti-racism falling into the racism category. Moreover, in the code folder, Sect. 1. we do not consider categories whose names do not include topic- For the rest of the paper, we refer to the opinions about a topic specific keywords. using 푃 and 푃¯. In Table 2, for each topic, we match each set to the real opinion it represents. 3.2 General Statistics on Topics’ Networks Before presenting the general statistics of the retrieved networks, Following the procedure explained in the previous section, we we remark that when we assign the articles to partitions, we put collect the topic-induced network related to six different topics to the set N those assigned to both partitions. The size of the that we pick from the List of controversial issues on Wikipedia11 intersections among partitions (i.e., the number of common articles) and other resources that indicate some controversial issues in our are the following: abortion is 2, cannabis is 3, evolution is 2, guns is 1, society. These topics are: abortion, cannabis, guns, evolution, LGBT, lgbt is 5, racism is 7. Recalling that we do not remove these articles and racism. These are critical topics that often polarize as follows: (i.e., they belong to N), they can still act as bridges connecting 푃 pro-choice vs. pro.life, cannabis activism vs. cannabis prohibition, and 푃¯ in sessions longer than 1 click. Instead, when we consider gun control vs. gun rights, creationism vs. evolutionary biology, the direct connections among partitions (1 click), we discard them support to LGBT rights vs. opposition to LGBT rights, and racism since they do not explicitly categorized into one partition. In Table 1, we show some statistics on the six topic-induced ¯ 10We report the size of the intersections between partitions in the next section. networks. Immediately, we observe that the size of 푃 and 푃 differ 11https://en.wikipedia.org/wiki/Wikipedia:List_of_controversial_issues substantially for all the topics except for racism and guns. It means Auditing Wikipedia’s Hyperlinks Network on Polarizing Topics WWW ’21, April 19–23, 2021, Ljubljana, Slovenia that we have one of the two opinions represented by more articles. results in [15, 17]. If a link appears more than once, we average its In terms of content, this does not necessarily imply that neither position. one of the two views is incomplete nor insufficiently represented. In Figure 2, we show the position distributions. According to Indeed, a topic spans a few articles or may require more pages to the t-test, whose significant level is fixed to 훼 = 0.95, the average be complete. On the other hand, the unbalanced sizes can affect an position of links in pro-choice pointing to pro-choice is significantly opinion’s exposure within the entire Wikipedia’s network. Practi- different than the average position of links pointing to pro-life. Also cally, if a set of articles is large and well connected to the rest of the position of links from guns control to guns control is signifi- the network, the chances that users who randomly browses reach cantly higher than those to guns rights. For evolutionary biology, it are higher than those of going to a small partition. Moreover, if whose distribution of links to creationism are placed statistically readers exploit the random article functionality of Wikipedia, an significantly lower than those to evolutionary biology. The same opinion more represented gets more chances of being randomly happens for LGBT. sampled. For the sake of completeness of the analysis, even if not used The topics showing the higher unbalance are cannabis, where further in the paper, for each topic, we study the quality of the pages there are five times more pages about activism than about prohi- populating it. In particular, we use the ORES API to get the “article bition, and evolution where there are four times more pages about quality.” We observe that overall, for all the topics between 60 and evolutionary biology than about creationism. If we consider the edges 70%, the articles are classified as stubs or start. Then the 22-29% is across partitions, the number of cross-partition edges is higher for in B-class, the 0-5% are Featured Articles, and the remaining belong bigger sets. This is reasonable because more nodes can point to the to the C-class.12 opposite side. Despite that, for evolution the edges from creationism to evolutionary biology are ∼3 times more, and for LGBT the edges 4 METRICS from discrimination to rights are 36% more. Despite the low number In this section we define the models and metrics that we use toan- of edges across cannabis partitions, we decide not to discard the swer the research questions formulated in Sect. 1. First, we describe topic. how we characterize readers’ consumption, either by analyzing Above, we said that one of the two partitions might connect real users’ data or by simulating their behavior (see Sect. 4.1). Then, better to the rest of the encyclopedia. We observe that the sizes of 푃 we introduce the core metrics of the paper ExDIN and M-ExDIN, and 푃¯ are not linear in the number of edges that point out or to the see Sect. 4.2. nodes in the partitions. For instance, the number of articles about pro-choice (291) is half of the nodes related to pro-life movement 4.1 Content Consumption pro-life (481). Although the nodes in are twice as many as those in To understand readers’ consumption of polarizing topics, we need pro-choice pro-choice , the number of links pointing to pages about different modeling strategies that we describe in the following pro-life is 36% more than those pointing to articles. This happens, subsections. with different magnitude, also for guns and LGBT. We will see later that the fact that a side of a topic is better blended in the network 4.1.1 Metrics Based on Clickstream. We build two metrics upon the has implications on the readers’ exposure to one of the two sides information we extract from users’ clickstream data that are made of the topic (Sect. 6). publicly available by Wikimedia and preserve users’ privacy [14, We also investigate how many pages in 푃 and 푃¯ cannot be 54]13. reached by users, unless they enter Wikipedia directly on those From these data we infer 푐푖,푗 counting how many times a hyper- pages. The sets of articles with the highest number of unreachable link to 푖 ∈ 푉 is clicked from page 푗. The page 푗 may be either an nodes are in the category of cannabis prohibition (11.36%), followed internal Wikipedia page (푗 ∈ 퐴, recalling that푉 = T ∪N ∪푆 include by the 5.62% of evolutionary biology and LGBT rights (5.35%). all the Wikipedia pages), or external if corresponds to a page from Furthermore, we compute the modularity 푄 among 푃 and 푃¯. outside Wikipedia (e.g., a search engine). Thus, we define the vari- Higher 푄 means that connections within partitions exceed those able 훿푗 , which indicates whether 푗 is an external page or it belongs to among them. In Table 1, we report three values computed on dif- the topic-induced network: 훿푗 = 1 if 푗 is external and 0 otherwise. ferently weighted graphs with probabilities assigned to click the Given a page 푖, we indicate with J the set of external and internal Í link of each page as follows: (1) uniform, (2) proportional to the pages pointing to it; see Figure 3. We define 푐푖 = 푗 ∈J 푐 푗,푖 to be Í position of the link within the page, and (3) proportional to readers’ the total clicks to the page. 푗 ∈J 훿푖푐 푗,푖 is the total number of clicks clickstream (see Sect. 4.1.2). Overall, if we consider the position of from external websites, therefore the difference between 푐푖 and this links and readers’ clickstream, it seems that the partitions are more summation is the number of visits from internal (Wikipedia) pages. modular. Now we are ready to define the following metrics: Based on that, we study how links across and within partitions Reader Search Rate (RSR). Given a page 푖 ∈ 푉 , the empirical position in pages. First, we define the position of a link. Given a probability that a visit to page 푖 is from an external website is Í page, we have its list of links in order of appearance. We get the 푗 ∈J 훿푖푐 푗,푖 relative rank within the list for each link and re-scale it by the tanh. 푅푆푅푖 = . (1) 푐푖 In this way, we have values in [0, 1], and the links at the top of the list get a more similar score. The set of links includes those in the 12https://en.wikipedia.org/wiki/Template:Grading_scheme 13Description of the data is at https://meta.wikimedia.org/wiki/Research: infoboxes. We regard them as at the top of the article, according to Wikipedia_clickstream. The provided information is enough to extract the clickstream based metrics WWW ’21, April 19–23, 2021, Ljubljana, Slovenia Cristina Menghini, Aris Anagnostopoulos, and Eli Upfal

Click Through Rate (CTR). Given a page 푖 ∈ 푉 , the empirical probability that a reader clicks a link within the page is internal Í i internal 푗 ∈푁표푢푡 (푖) 푐푖,푗 퐶푇푅푖 = , (2) 푐푖 external where 푁표푢푡 (푖) is the the set of pages 푖 points to. (Multiple clicks from the same page are counted as originating from different visits Figure 3: Information from the clickstream dataset. For each to 푖 and, thus, counted multiple times in 푐푖 .) node we extract the number of views coming from inter- 4.1.2 Model Clicks Within Pages. When readers visit a page, they nal and external websites. Moreover we know how many ac- have the possibility of clicking any of the present links. However, cesses on a page turn into a click toward another article. according to the information needs they want to satisfy, each of the links may have a different probability of being clicked [45]. Now, we propose three models to describe the distribution probability different behaviors, accordingly to chosen parameter. We empha- of clicking a link “j” within an article “i.” First, let 푖 be an article size that the scope of this model is not to perfectly replicate users’ behavior on Wikipedia. Rather, we want to see how users simu- in 푉 and 푗 ∈ 푁표푢푡 (푖). We define 푝표푠(푗 |푖) as the rank of 푗 among lated from a reasonable and general model are exposed to diverse all links in 푖, and 푟 (푗 |푖) = |푁표푢푡 (푖)| − 푝표푠(푗 |푖), such that a higher value indicates a higher ranking position. Moreover, we introduce information. 2푥 − In other words, we want to define a stochastic process, with 푛 + 1 tanh 푥 = 푒 1 , which we use to transform ranking positions to 푒2푥 +1 states, corresponding to the 푛 + 1 pages in 푉 , that approximates the values between 0 and 1, such that links at the top of the page get probability of reaching any of the articles starting at random from similar scores. 푝 ∈ 푃 (or from 푃¯). The Clicks Within Pages models (CwP) are directly applicable on We model this, by considering the process {푋 ℓ ; ℓ = 0, 1, . . . 퐿}, on 퐺 by setting the transition matrix 푀 in one of the following modes. the set of nodes푉 induced by transition matrix 푀 with starting state 푢 ( ) = 1 × (1) 푀 (Uniform), whose entry 푚 푖, 푗 |푁 (푖) | mimics read- 0 0 = {( ) } ∈ R1 푛 표푢푡 푋 selected from the probability distribution 흅푃 휋푃 푖 ers who click each link in a page uniformly at random; over 푉 . We recall that the transition matrix 푀 can vary according to 푝 tanh푟 (푗 |푖) (2) 푀 (Position), whose entry 푚(푖, 푗) = Í ( | ) the CwP models (Sections 3.1 and 4.1.2). Based on the assumption 푗∈푁표푢푡 (푖) tanh푟 푗 푖 captures the scenario in which readers click with higher that users’ session length (the number of clicks) is finite, we evaluate ℓ probability links appearing first in the page. This model is the process on a finite number of states 퐿. We have that Pr(푋 = ) = ( ℓ ) ℓ based on previous work that shows how the link position is 푗 휋푃 푗 , where the (row) vector 흅푃 is given by the following a good predictor to determine the success of a link [16, 31]; variation of the Personalized Random Walk with Restart (RWR). 푐 푐푖,푗 (3) 푀 (Clicks), whose entry 푚(푖, 푗) = Í represents (Navigation Model). Let 푀 be the transition ma- 푗∈푁표푢푡 (푖) 푐푖,푗 Definition 1. 0 0 the empirical probability that users in 푖 will click the link trix embedding a click-within-pages model, 흅푃 the distribution of the 14 toward 푗. When 푐푖 푗 < 10 we substitute it with 10 , the starting state over 푃, and 훼 ∈ [0, 1] the restart parameter. We have minimum number of times the link must be clicked to be 1 0 흅 = 흅 · 푀0 (3) included in the dataset [53]. 푃 푃 and for ℓ ≥ 1 For the sake of completeness, we recall that 퐺 includes a super ℓ+1 ℓ 0 node, 푠. To fill its corresponding entries in the transition matrices, 흅푃 = (1 − 훼)흅푃 · 푀ℓ + 훼 (흅푃 · 푀ℓ ), (4) we need to aggregate over the edges we compressed to build the  −1 15 = (( ( )푇 )푇 ) = + ℓ−1 ( ) graph, see Sect. 3.1. where 푀ℓ norm 퐷 푀ℓ−1 and 퐷 푑푖푎푔 1 흅푃 . norm 푀 transforms matrix 푀 into a right-stochastic matrix by normalizing 4.1.3 Readers Navigation Model. The main goal of this paper is each row independently such that it sums to 1. to audit the mutual exposure to diverse information across 푃 and 푃¯. We can do it by simply looking at a snapshot of the graph and This process is a variation of the standard random-surfer (PageR- counting the links going from 푃 to 푃¯ and vice-versa. To do a step ank) model, with the difference that the transition matrix is updated further, we recall that the Wikipedia’s network is conceived to let in each step. It takes into account the probability that an article users move, fulfilling their own information needs. Thus, we want has already been visited in a previous iteration. Specifically, the ℓ to understand how different users’ navigation behavior can affect vector 흅푃 that we get at the end of each iteration, represents the readers’ exposure to diverse information. likelihood that each node is reached at step ℓ if it starts uniformly at To do that, it would be optimal to have access to users’ log ses- random from a node in 푃. We assume that readers, within the same sion. Because these data are not available to the public, we define a session, do not click more than once the same link. Thus, we desire parametric model that simulates users’ navigation by embedding that, at step ℓ + 1, the nodes that are clicked with high probability at step ℓ see their probability of being reached deflated, and those 14We aim to model users on the current version of Wikipedia. Thus, to include all the links, we assign a smoothing factor equal to 10 to links clicked less than 10. This with lower probability have more chances of being clicked. We implies a small probability of clicking these links. Setting the smoothing factor to 10 achieve this by dividing the rows of 푀 by the vector of probabilities is a deliberate choice. However, we experimentally verified that setting any number 흅ℓ +1, where 1 is a smoothing factor to avoid divisions by 0, and between 1 and 10 does not affect the results. 푃 15The computation of these quantities is straightforward so we omit it from the then normalize the matrix to get the updated stochastic matrix to body of the paper. use in the next iteration. Auditing Wikipedia’s Hyperlinks Network on Polarizing Topics WWW ’21, April 19–23, 2021, Ljubljana, Slovenia

10

5

Log(Pageviews) 0 (a) Star-like (훼 = 1) (b) Star-like Rand. Navigation(0 < (c) Random Naviga- 훼 < 1) tion (0 = 훼) Rights Pro-life Control Racism Support Activism Evol. Bio.

Figure 4: Navigation model for different 훼. The green nodes Pro-choice Prohibiiton Anti-racism represent the starting navigation pages. Creationism Discrimination

Figure 5: Pageviews distribution. For each topic, we have Overall, as we will later see in Section 4.2, this approach allow us a purple and yellow boxplot. They represent the average to investigate how the exposure to diverse information varies for (over all pages in the group 푃 or 푃¯) number of pageviews. users who behave differently in terms of navigation session length All the distribution distributions except for abortion are sta- (meant as the number of clicks) and next-link choices. tistically different at confidence level 훼 = 0.95. The topics Looking deeper at the model: in order are: abortion, cannabis, guns, evolution, racism and • When 훼 = 1, Figure 4(a) the model emulates the reader LGBT. whose navigation consists in just opening links from the starting page. We call this behavior star-like and basically Definition 2. (Exposure to diverse information (ExDIN)). Given consists in opening pages from the starting node. With this two sets of pages 푃, 푃¯ in 푉 , let 흅ℓ be the vector indicating for each kind of exploration, readers locally explore articles likely 푃 article the probability of being reached at step ℓ (ℓ ≥ 1) starting from semantically related to each other [49]. a random page in 푃. We say that the exposure of 푃 to 푃¯ is • For 0 < 훼 < 1, Figure 4(b), we simulate two cases: (1) readers ∑︁ ∑︁ open sequential articles and then jump back to the starting 푒ℓ = Pr(푋 ℓ = 푗) = 흅ℓ 푃→푃¯ 푃 (5) page, (2) readers keeps multiple path open. The more 훼 is 푗 ∈푃¯ 푗 ∈푃¯ close to 1 the more users show a star-like behavior. Instead, and describes the probability that a reader in 푃 reaches an arbitrary the closer 훼 is to 0 the more users navigate navigate in a node in 푃¯ at the ℓth click. more DFS-oriented fashion. Thus, readers move randomly according to the CwP model and from time to times jump We employ this metric in two ways: back to the starting page. (1) (Topological exposure to diverse information) If ℓ is 1 and • If 훼 = 0, Figure 4(c), the users sequentially clicks links, so the CwP model is 푀푢 (see Sect. 4.1.2), it only quantifies each click depends only on the CwP model. In this case, the topological property of the network to connect pages especially if related articles are not densely connected, the belonging to different sets. exploration can lead to articles less related to the starting (2) (Readers’ exposure to diverse information) For any parameter page and returning to the origin following hyperlinks may and model that we pick, the metric tells us how the readers, be difficult. characterized by the CwP and Navigation models, change Because Wikipedia does not have a button that allows readers to their exposure to diverse information over a session (i.e., go back to the previous page, we assume the jumping back action sequence of clicks). to consist in clicking the back button of the browser in use, until Moreover, we notice that Definition 2 can be extended to multiple reaching again the session starting page. The restart parameter sets. Consider the case where we want to understand how one set of indirectly embeds the back-button action, which for the absence of nodes 푃 is exposed to three sets of nodes, 푄, 푍, and 퐿. To calculate back-links on Wikipedia can not be tracked on the graph. the ExDIN, if we want to know the total exposure to the three sets, The behaviors replicated through the model recall those de- we define 푃¯ = 푄 ∪ 푍 ∪ 퐿. Otherwise, if we want to have the ExDIN scribed in [43, 45]. ℓ w.r.t. to each set, namely, 푒푃→푄, 푒푃→푍 , 푒푃→퐿, we take 흅푃 and sum up the probabilities of the nodes within each set. 4.2 Exposure Metrics Now that we have a metric to compute the exposure to diverse At this point, we have all the ingredients to define the exposure information, we want to compare the flows among the sets. Thus, to diverse information. The metrics aim to quantify how much the we introduce the mutual exposure to diverse information. network structure allows readers to reach one, or multiple sets of Definition 3. (Mutual exposure to diverse information (M-ExDIN)). articles. To do that, we rely on both the CwP and Navigation models. Let 푒ℓ and 푒ℓ be the exposure to diverse information of sets 푃 The application of the following metrics is not limited to polarizing 푃→푃¯ 푃¯→푃 ¯ topics. In fact, they can generalize to the analysis of any sets of and 푃. We say that the mutual exposure between the sets is nodes in a graph. For this reason, we adopt a more general notation { ℓ ℓ } min 푒 → ¯, 푒 ¯→ in their definition. 휖ℓ = 푃 푃 푃 푃 ∈ [0, 1]. (6) max{푒ℓ , 푒ℓ } 푃→푃¯ 푃¯→푃 WWW ’21, April 19–23, 2021, Ljubljana, Slovenia Cristina Menghini, Aris Anagnostopoulos, and Eli Upfal

the seasonality effect and weight links according to page changes, Anti-racism Racism in terms of hyperlinks.

LGBT rights LGBT discrimination Evolutionary biology 5.1 Pageviews Distribution Creazionism

Gun rights To start our analysis we count the average number of times a Gun control page has been visited over 34 months. In Figure 5 we plot the Activism Prohibition log-distributions of the pageviews for each topic and opinion. By running a t-test, we conclude that for all topics except for abortion Pro-choice Pro-life the difference of the means of opinions’ pageviews is statistically 0.0 0.5 1.0 1.5 2.0 2.5 significant for 푝 < 0.05. This finding demonstrate that users tend to % of pageview from opposite partition visit more pages expressing/supporting one of the two viewpoints. From a network’s perspective, to increase the exposure to opposing Figure 6: Percentage of pageviews coming from opposite opinions, it is desirable for pages that are frequently visited to be side. Topics in order from the bottom: abortion, cannabis, well connected to articles expressing opposing opinions. guns, evolution, LGBT, racism. In Figure 6, we break down the pageviews, showing how many of them come from pages of the opposing partition. Overall, the fraction of visits from the opposite side is low (below 0.5%). The If either 푒ℓ or 푒ℓ is 0, then 휖 = 0. 푃→푃¯ 푃¯→푃 category LGBT rights has the highest ratio of visits from LGBT dis- This measure quantifies to what extend the exposure to diverse crimination pages, about 2.5%. For topics such as guns and abortion, information is balanced across 푃 and 푃¯. the percentage of visits from opposite partition shows that there The closer 휖 is to 1, the more balanced the probabilities of moving are somewhat fewer visits to pages of a liberal inclination from from one set to the other are. In this case, the network topology does articles expressing a more conservative opinion. In fact, the 0.28% not favor connections from one set to the other. On the other hand, of visits to pro-choice come from pro-life, compared to 0.6% visits of if 휖 is close to 0, it the network structure tends to favor either the pro-life from pro-choice. navigation from 푃 → 푃¯, or from 푃¯ → 푃. On this perspective, if we observe a tendency in the network of facilitating the exploration 5.2 External or Internal Access to the Topic? from one of the sets to the other, we may say that the network topology is biased toward a direction. Thus, we can think of using We now investigate how readers access content about a topic. As M-ExDIN to measure the bias in the network w.r.t. two sets of introduced in Section 4.1.1, from the clickstream data we can com- nodes. pute the RSR, which indicates whether a page is accessed more Even though the mutual exposure to diverse information cap- by external sources or by navigating Wikipedia. In Figure 7, we tures the balance among ExDIN of 푃 and 푃¯ when they are of com- provide a visualization that depicts the flows of the cumulative parable size, it may fail if one is much smaller than the other. For visits from external and internal pages towards the two partitions. instance, suppose 푃 is 10 times larger than 푃¯, then, if pages of both Referring to Figure 7(c), the 44.8% of visualizations come from partitions have a similar out-degree distribution, one would expect internal pages. The click stream from internal pages is broken down 푒ℓ ≈ 10 · 푒ℓ , and, as a result, 휖 ≈ 0.1. The same happens if to see the proportion of flow towards guns control and gun rights. 푃→푃¯ 푃¯→푃 The internal views of guns control articles are 3.4 times more than they have similar in-degree distribution. For this reason, when we those of gun rights. We observe that also from external websites compute either ExDIN and M-ExDIN, we check whether the sizes most of the traffic is towards gun control (2.7 times more than gun of the communities are unbalanced and we proceed as follows. If rights). Overall, the 26% of the total visits to gun related content is |푃 | < |푃¯|, we define 푃¯′ obtained by sampling |푃 | articles from 푃¯. concentrated on gun rights. Thus, we use the new set for all computations. Because of the ran- The abundance of traffic towards one of the two opinions does domness of the phenomenon, we repeat the measurements multiple not characterize only the guns topic. Indeed, among all the topics, times. the 59–74% of visits is accumulated by one partition. Moreover, 5 RQ1: READERS’ TOPIC CONSUMPTION readers’ preferences appear consistent among external and internal accesses, that is, they both point more towards the same view of Before looking into how readers are exposed to diverse content, the topic. For both internal and external views, the distribution we investigate how they have consumed each of the six topics of accesses toward partitions is approximately the same (i.e., the that we concentrate on, over the last four years. In particular, we percentage of visits from external to 푃 (resp. 푃¯) is the same of collect monthly clickstream data from November 2017 to September from internal to 푃(resp.푃¯)). The only exception is evolution, whose 2020. We note that when we count the click views of a page, we 16 external visits to creationism is 45.3% lower than internal accesses. consider the average over the number of months the page existed . We note that partitions with higher views are not necessarily the Accordingly, when computing the occurrences for the transitions biggest in the topic-induced networks. matrix based on clickstream, we consider the average clicks of the In general, the largest amount of visits to topics’ articles comes link over the number of months it exists. In this way, we reduce from external pages. Particularly, only the 23.6% and 33.5% of traffic 16Based on the temporal graphs extracted by [11] to evolution and racism is generated by the internal Wikipedia’s Auditing Wikipedia’s Hyperlinks Network on Polarizing Topics WWW ’21, April 19–23, 2021, Ljubljana, Slovenia

Extterrnall Extterrnall Extterrnall Extterrnall Prro--choiice Evoll.. Biio.. Extterrnall Raciism Diiscrriimiinattiion Conttrroll Extterrnall Acttiiviism

IIntterrnall IIntterrnall IIntterrnall IIntterrnall Prro--lliiffe Crreattiioniism IIntterrnall Anttii--rraciism Supporrtt Riightts IIntterrnall Prrohiibiiiitton

(a) Abortion (b) Cannabis (c) Guns (d) Evolution (e) Racism (f) LGBT

Figure 7: Cumulative Pages’ Traffic. Each plot indicates: (1) On the left, the cumulative amount of accesses comingfrom external web pages or internal Wikipedia’s articles. (2) The flows of visits from external and internal pages to partitions. (3)Onthe right, the cumulative accesses to 푃 and 푃¯.

Topic %C(¯ 푃) %C(¯ 푃 → 푃) %C(¯ 푃 → 푃¯) %C(¯ 푃¯) %C(¯ 푃¯ → 푃¯) %C(¯ 푃¯ → 푃) Anti-racism Abortion 68.89 90.82 57.64 70.26 89.81 45.37 Racism Cannabis 70.81 95.01 37.50 65.78 96.52 16.67 LGBT rights Guns 52.34 78.69 35.68 59.63 75.35 39.28 LGBT discrimination Evolution 71.15 84.49 64.47 72.69 99.00 55.13 Evolutionary biology Creazionism Racism 56.36 88.41 34.32 71.87 90.63 65.44 Gun rights LGBT 61.66 89.42 52.5 72.42 92.52 59.17 Gun control Table 3: Average of links within pages clicked less than 10 Activism Prohibition times. %C(¯ 푃) is the average percentage of un-clicked hyper- Pro-choice links within pages in 푃. %C(¯ 푃 → 푃¯) is the average percentage Pro-life of un-clicked links, within 푃, pointing to 푃¯. 0 5 10 15 20 25 30 Avg(Click Through Rate) X 100

Figure 8: Average click-through rate. In this plot we report This suggests that articles having higher out-degree, offer more the average CTR of pages belonging to the same set 푃 (resp. options to users. Presumably because of this, users are more likely 푃¯). The score indicate the average probability that a link to continue the exploration from those articles. within a page 푝 in 푃 (resp. 푃¯) is clicked. Topics in order In addition, we count the number of links clicked fewer than 10 from the bottom: abortion, cannabis, guns, evolution, racism, times over the last three years; see Table 3. As an example, given LGBT. a page in creationism, on average 71.15% of its links have been clicked fewer than 10 times. If we distinguish between references to creationism and references to evolution, readers did not click navigation. The same quantity, for the remaining topics ranges the 84.49% of links pointing to creationism and the 64.47% of those ranges between 44 and 47%. pointing to evolutionary biology. We point out that readers’ consulting articles about abortion, cannabis and guns are inclined toward pages conveying liberal views on the topic. Instead, it is more complicated to draw inter- 6 RQ2: EXPOSURE ACROSS TOPIC pretations about the remaining topics. One explanation may be VIEWPOINTS that users look for information generally less covered in the public The main contribution of this paper is to examine to what extent mainstream debate. current Wikipedia’s topology supports users to explore diverse facets of polarizing issues. In particular, we study (1) how readers are 5.3 How Much Readers Navigate Links? locally exposed to diverse information, and (2) how their exposure Once readers visit a page, they can decide to click any of its links. to plural opinions may change throughout a navigation session. We want to understand how frequently they do so. For that, we compute the average pages’ click-through rate (Sect. 4.1.1). 6.1 Exposure to Diversity We plot this information in Figure 8. Overall, we see that the To evaluate the exposure to diversity induced by the network’s percentage of access turning into a visit to another page ranges topology, we compute the exposure to diverse information for ℓ = 1 between 10–28%. Dimitrov and Lemmerich [14] observed that the using the uniform CwP model. Recalling that ℓ indicates the users’ CTR average for the whole Wikipedia is 12%. So, most of the subset session length, if we set it equals to 1, we study the exposure to of pages we consider have a CTR higher than Wikipedia’s average. diversity over one-click sessions. The CTR of guns control is the highest (28%), the pages about racism Plots in the first row of Figure 9, show the value of ExDIN for follow with 26%. The articles that over the years have generated all the topics when CwP is 푀푢 . less internal traffic are those about evolutionary biology and LGBT For instance, let evolution be the topic we analyze. If readers rights. start uniformly at random from a page about creationism, the prob- Examining pages’ connections, we found that those with higher ability of visiting an article of the same partition is 5.76%. On the CTR have more links (the Pearson correlation coefficient of is 0.52). contrary, the chances of entering a page about evolutionary biology WWW ’21, April 19–23, 2021, Ljubljana, Slovenia Cristina Menghini, Aris Anagnostopoulos, and Eli Upfal

Pro-choice 1.33 7.48 Activism 0.39 0.38 Rights 1.61 4.78 Evol. Bio. 0.61 3.27 Anti-racism 1.3 8.6 Support 0.63 3.95

Pro-life 4.56 0.61 Prohibiiton 3.38 0.08 Control 4.44 1.16 Creationism 5.76 0.18 Racism 8.76 1.42 Discrimination 4.24 0.54 Uniform

Pro-choice 1.15 7.81 Activism 0.37 0.37 Rights 1.38 4.73 Evol. Bio. 0.54 3.29 Anti-racism 1.29 8.34 Support 0.56 3.56

Pro-life 4.28 0.56 Prohibiiton 3.32 0.08 Control 4.68 0.89 Creationism 5.53 0.16 Racism 8.92 1.59 Discrimination 3.94 0.42 Position

Pro-choice 1.29 13.57 Activism 0.81 0.65 Rights 2.08 13.05 Evol. Bio. 0.69 5.2 Anti-racism 1.16 15.03 Support 0.66 6.53

Clicks Pro-life 12.74 1.23 Prohibiiton 8.64 0.13 Control 13.76 1.37 Creationism 11.57 0.3 Racism 23.67 1.29 Discrimination 11.07 0.53 Rights Pro-life Control Racism Support Activism Evol. Bio. Pro-choice Prohibiiton Anti-racism Creationism Discrimination

Figure 9: ExDIN. Each patch of the matrix is the exposure to diverse information across partitions; for example, 푃 → 푃, 푃 → 푃¯. The 푦-axis indicates the source and the 푥-axis is the destination. To each row corresponds the exposure to diverse information computed for different CwP. Darker colors indicate higher probability of being in the correspondent square inoneclick. is 0.18% (32 times smaller). On the other hand, readers starting the placement of links within pages contributes in reducing the uniformly at random from an article about evolutionary biology exposure to diverse information. In other words, users who tend have 3.27% chances of reading pages conveying the same opinion. to click with higher probability the links located towards the be- This probability is 5 times larger than that of visiting creationism ginning of a page have less possibility to read about contrasting pages. opinions. It is worth to point out that the current network’s topology, not Finally, we analyze how the phenomenon changes when we use only nudges users in reading more about the same opinion, but the click CwP model (third row). In this case, we assume readers also hinders them to explore diverse content symmetrically. Indeed, make the next-click choice similarly to past users. Going back to users reading about evolutionary biology have higher chances of evolution, we immediately observe that the probability to start a reading one article about creationism (3 times more) than users from session in creationism and to continue reading about it after one creationism of reading about evolutionary biology. After repeating click grows from 5.76% of the uniform model to the 11.57%. the same analysis for all the topics, we realize that the aforemen- For all the topics we verify a significant increment of the proba- tioned observations hold for most of them. Moreover, we note that bility to visit pages of the same opinion. Simply interpreting this the probabilities to continue the session reading a page of the same result, we can say that real users click more the links strictly related opinion is greater for one of the two partitions of a given topic. to the page they are reading. From another perspective, combining Taken together, these measurements highlight that the structure this finding with the previous remark saying that the “topology of of the network facilitates users to explore knowledge bubbles of the network seems to drive users to explore knowledge bubbles of homogeneous view, and makes the measure of mutual exposure to homogeneous view,” we ask the reader the following question. Has diverse information smaller than 1. the behavior of past users been influenced by the network topology? The findings above report the intrinsic capability of the network Unfortunately, because of lack of information, we can not answer to spur users towards diverse content. If we want to combine it this question but we hope it will be addressed in future works. with readers’ next-click choice behavior, we use the the position Furthermore, we observe that for some topics like abortion, the and clicks CwP models, instead of the uniform. We show the results probability of reaching pro-choice articles from pro-life duplicates. in the second and third row of Figure 9. This is a sign that users may be willing to explore content proposing Referring back to evolution, we now consider the matrix corre- diverse view. sponding to the ExDIN computed using the position CwP model Before moving to the next section, we want to underline that (second row). We see that if users click with higher chances links stronger relations among pages of similar content is an intrinsic at the top of the page, w.r.t. the uniform model, the probabilities are property of Wikipedia. In fact, in Wikipedia’s Linking Manual [49], only slightly modified. These modest variations are coherent with editors are asked to link related content [7, 33]. Although this is a links’ placement within pages, Figure 2. fact, we believe that it would be valuable to provide to editors met- For a few topics, such as guns, the links’ position plays a more rics and tools making them aware of the effect that a new/current significant role, worsening the user exposure to diverse information. links have on users’ exposure to diverse information. This is not Indeed, in pages about guns control, links belonging to the guns meant to alter the core and essential intrinsic property of Wikipedia, rights partition seem to be mentioned later in the page. The con- rather to avoid this property to become harmful when it prevents sequence is that the probability of reaching an article supporting users from accessing diverse content. guns rights, starting uniformly at random from an article in guns control, has a 30% drop w.r.t. the probability observed using the uniform CwP model. Therefore, we conclude that for some topics, Auditing Wikipedia’s Hyperlinks Network on Polarizing Topics WWW ’21, April 19–23, 2021, Ljubljana, Slovenia

Abortion Cannabis Guns Evolution Racism Lgbt 2 1.0 1.0 2 1.5

P 1.0 0.5 1.0 P 1 1 0.5

e 0.5 0.5 0.0 0 1 3 5 7 9 11 13 15 1 3 5 7 9 11 13 15 1 3 5 7 9 11 13 15 1 3 5 7 9 11 13 15 1 3 5 7 9 11 13 15 1 3 5 7 9 11 13 15

1.5 0.6 1.5 0.50 2 1.5 P 1.0 0.4 1.0

P 0.25 1.0

e 1 0.5 0.2 0.5 0.00 0.5 0.0 1 3 5 7 9 11 13 15 1 3 5 7 9 11 13 15 1 3 5 7 9 11 13 15 1 3 5 7 9 11 13 15 1 3 5 7 9 11 13 15 1 3 5 7 9 11 13 15

100 100 100 100 80 75 80 75 80 80 50 60 60 50 60 40 60 40 25 25 1 3 5 7 9 11 13 15 1 3 5 7 9 11 13 15 1 3 5 7 9 11 13 15 1 3 5 7 9 11 13 15 1 3 5 7 9 11 13 15 1 3 5 7 9 11 13 15 Number of Clicks

uniform alpha=0 uniform alpha=0.2 uniform alpha=1 position alpha=0 position alpha=0.2 position alpha=1 clicks alpha=0 clicks alpha=0.2 clicks alpha=1

Figure 10: Dynamic (Mutual)ExDIN for 1 ≤ ℓ ≤ 15. The first and second rows show the probabilities of moving across partitions. The third row indicate the mutual exposure to diverse information. Each color correspond to a different level of 훼, the restart parameter. The markers’ shape indicates the CwP model in use. Higher values of the M-ExDIN mirror more symmetric expo- sures between opinions. We repeat the computations of the metrics 100 times and report the standard deviations, to account for the randomness 4.2.

6.2 Dynamic Exposure to Diversity to the exposure of their starting navigation page. So, the more links In this section, we suppose users navigate the network for sessions to the opposite partition they click in the first steps, the more their longer than 1, and see how their (mutual) exposure to diversity may exposure to diversity decreases, and vice-versa. change. According to the combinations of models employed to mea- (3) The curves of users who randomly navigate the network (sky sure the ExDIN and M-ExDIN, we provide different insights about blue, 훼 = 0) show two trends. For both cases, after the first few the effect of the current network’s topology on users’ exposure to clicks, the ExDIN is lower than at the beginning of the session. diverse content over a navigation session. Then, it inverts the trend. In one case, it reaches or exceeds the In Figure 10, for sessions of length 15, we plot the ExDIN from starting exposure. On the other hand, it grows getting steady below 푃 to 푃¯ (resp. from 푃¯ to 푃), and the respective M-ExDIN. We can the initial exposure. The more the destination partition is connected notice that each of the topics shows its own trends. For this reason, to the rest of the graph, the more users randomly navigating the ¯ we decide to highlight and provide an explanation for the most network are able to reach it. Sometimes, ExDIN from 푃 to 푃 of recurrent patterns. Moreover, for better understanding, we suggest LGBT, the curves start to decrease after many steps. This happens to cross-check the following explanations with the analysis done when the pages within the destination partition have been reached above in the paper. We start describing how, given the current net- with high probability. work’s topology, the ExDIN changes over the course of a navigation (4) Users characterized by a star-like random navigation (blue, session (first and second row). 훼 = 0.2) are exposed to diverse content similarly to users exploring (1) The curves corresponding to the same value of 훼 (same color) randomly the network, but the ExDIN magnitude is greater because show very similar trends. Depending on the respective CwP model of the possibility of jumping back to the starting point at each step. (marker’s shape), they are shifted up or down. This implies that, Given this observation, we analyze the ExDIN of guns. With the when users share the same navigation behavior, the way they make current network’s topology, starting from the guns control partition, the next-click choice plays a crucial role on determining the mag- the users with higher exposure to guns rights pages (2% probability) nitude of their exposure to diverse content. In general, the CwP are those characterized by a star-like behavior. As soon as the users model (markers’ shapes) corresponding to higher exposure is 푀푐 , navigate in a random fashion, this probability drops down. This is followed by 푀푢 and 푀푝 . because the guns rights partition has a number of in-going edges (2) If users navigate mirroring a star-like behavior (green, 훼 = 1) that prevent random users who walk away to get back to it. We their exposure to the opposite opinion is steady. It can slightly observe the opposite for users who start their sessions in guns rights. decrease or increase when the probability of clicking links to the Indeed, for users randomly navigating the graph, the exposure to opposite side becomes higher or lower, respectively, in the first the guns control partition is higher or comparable to that of star-like iterations. This happens because these kind of users are only subject behaving users. The probability of reaching guns control after many WWW ’21, April 19–23, 2021, Ljubljana, Slovenia Cristina Menghini, Aris Anagnostopoulos, and Eli Upfal steps becomes high because of the higher incoming edges of the content. Moreover, on average, the probability that the partition. network nudges users to remain in a knowledge bubble From complementary analysis, we observe that for sessions is up to an order of magnitude higher than that of ex- longer than 2 page visits, the topology of the graph is not such ploring pages of contrasting opinions.. In Section 6.1, to keep random users within the knowledge bubble they accessed. the analysis suggests that users reading about an opinion Indeed, for all the topics, the probability to visit articles of the same have higher chances of continuing to explore articles of sim- opinion of the starting page becomes close to 0 (refer to Fig. 9 to ilar views than of the opposite. Furthermore, for each of the cross-check the probabilities at the first click). For users with star- topics that we explored, the users of one of the two views like behavior, the probabilities show slightly descending curves. had substantially more tendency, albeit small, to visit pages This demonstrates that they tend to pick pages of the same opinion of the opposing view than the ones of the other one. at the first iterations. Due to space constraints, figures picturing • For sessions of length > 1, the network’s topology is these phenomena could not be displayed. typically biased toward one opinion. In Section 6.1, we Now we compare the ExDIN curves of each topic to understand if observe that the mutual exposure to diverse information is users reading about different opinions have equal chances of visiting never achieved by users navigating completely at random. each other. We recall that to do it, we use the Mutual exposure to The better one of the two opinions is connected to the rest of diverse information; see Sect. 4.2. For all the topics, the mutual the network, the more the graph nudges users toward that exposure to diverse information is lower than 100%, meaning that opinion. for none of them does the network topology provide an equal • For sessions of length > 1, the probability of reading exposure across opinions. If we consider topics like abortion and about the same opinion decreases for users browsing guns, the longer the users’ session is, the more the network topology according to the random navigation model. In Section 6.2, prevents readers to symmetrically explore different opinions. results suggest that after a few clicks the exposure to infor- In general, if we detect low mutuality, the topology of the net- mation of the same inclination diminishes. On the other work favors the exploration of one side of the topic more than the hand, if users explore the network with a star-like behavior other. Moreover, we want to stress that all the comments regarding their level of exposure to the same opinion is similar to those users navigating according to the uniform CwP model, express the who only do one click. intrinsic topological exposure of the network. On Wikipedia, two scenarios determining this phenomenon may be (1) the knowledge In our study we analyze sets of articles assigned to opinions ac- on the encyclopedia is complete but articles are underlinked [49], cording to editors’ crafted categories [44]. Although this approach that is, there is the content and keywords to become anchors thus represents a solid starting point for analysis, it can cause article mis- we need a strategy to densify the network ensuring mutual exposure classification. As future work, we plan to investigate a more reliable to diversity; (2) the knowledge on the encyclopedia is incomplete, classification strategy to improve the accuracy of our analysis, an that is, there are no words to attach links, thus the addition of com- analysis which should include also the content of the articles along plementary content may be necessary. An in depth investigation of the line information. Secondly, the performance of a longitudinal this conditions may be an interesting future work. study with the goal of understanding the dynamics that brought to the current state of the encyclopedia’s network, would provide 7 CONCLUSIONS further understanding of the users’ behavior. Finally, in the light of our findings, we deem crucial to design tools to help editors to Our work provides a first analysis to understand how the current contextualize articles within the network, such that they are aware Wikipedia’s network topology assists readers to explore opposite of the effect of links insertion on users knowledge exploration. stances of polarizing topics spanning over sets of articles. We for- The prevalence of bias and polarization is well established in malize the problem by introducing two metrics: the Exposure to multiple areas of our life, and filter bubbles aggravate this phenom- diverse information and the Mutual exposure to diverse information enon. Understanding better how they manifest in Wikipedia (and (see Sect. 4.2). The former quantifies the ease to jump across articles other media) is a crucial first step for finding ways to attenuate it expressing opposing viewpoints. The latter evaluates whether the and our hope is that this work is a step towards this goal. relationship across diverse views is symmetric, that is, whether Acknowledges. Partially supported by the ERC Advanced Grant the flow and the opportunity to go from one side to the otheris 788893 AMDROMA "Algorithmic and Mechanism Design Research comparable for the two directions. in Online Markets" and MIUR PRIN project ALGADIMAR "Algo- We investigate the phenomenon on six polarizing topics (Sect. 6). rithms, Games, and Digital Markets". In addition, we also study the overall users’ topics consumption. Our main findings suggest the following: REFERENCES • The traffic on polarizing issues is biased toward one [1] Lada A Adamic and Natalie Glance. 2005. The political blogosphere and the 2004 view of the topic. In Section 5, we show that accesses com- US election: divided they blog. In Proceedings of the WWW-2005 Workshop on the Weblogging Ecosystem. ing from both external and internal pages suggest that read- [2] Leman Akoglu. 2014. Quantifying political polarity based on bipartite opinion ers are inclined to seek content about one facet of the topic. networks. In Eighth International AAAI Conference on Weblogs and Social Media. Most seem to have bias toward liberal content (see Fig. 7). [3] Sumit Asthana and Aaron Halfaker. 2018. With few eyes, all hoaxes are deep. Proceedings of the ACM on Human-Computer Interaction 2, CSCW (2018), 1–18. • For sessions of length = 1, the current network’s topol- [4] Ivan Beschastnikh, Travis Kriplean, and David W McDonald. 2008. Wikipedian ogy, hinders users to symmetrically explore diverse Self-Governance in Action: Motivating the Policy Lens.. In ICWSM. Auditing Wikipedia’s Hyperlinks Network on Polarizing Topics WWW ’21, April 19–23, 2021, Ljubljana, Slovenia

[5] David Blei, Lawrence Carin, and David Dunson. 2010. Probabilistic topic models. and Social Media. 143–152. IEEE signal processing magazine 27, 6 (2010), 55–65. [30] Srijan Kumar, Robert West, and Jure Leskovec. 2016. Disinformation on the Web: [6] David M Blei, Andrew Y Ng, and Michael I Jordan. 2003. Latent dirichlet allocation. Impact, Characteristics, and Detection of Wikipedia Hoaxes. In Proceedings of Journal of machine Learning research 3, Jan (2003), 993–1022. the 25th International Conference on World Wide Web. [7] Ulrik Brandes, Patrick Kenis, Jürgen Lerner, and Denise Van Raaij. 2009. Net- [31] Daniel Lamprecht, Kristina Lerman, Denis Helic, and Markus Strohmaier. 2017. work analysis of collaboration structure in Wikipedia. In Proceedings of the 18th How the structure of wikipedia articles influences user navigation. New Review international conference on World wide web. of Hypermedia and Multimedia 23, 1 (2017), 29–50. [8] Ewa S Callahan and Susan C Herring. 2011. Cultural bias in Wikipedia content [32] Q. Vera Liao and Wai-Tat Fu. 2014. Expert voices in echo chambers: effects of on famous persons. Journal of the American society for information science and source expertise indicators on exposure to diverse opinions. In Proceedings of the technology (2011). SIGCHI Conference on Human Factors in Computing Systems. 2745–2754. [9] Uthsav Chitra and Christopher Musco. 2020. Analyzing the Impact of Filter [33] Dmitry Lizorkin, Olena Medelyan, and Maria Grineva. 2009. Analysis of commu- Bubbles on Social Network Polarization. In Proceedings of the 13th International nity structure in Wikipedia. In International conference on World wide web. Conference on Web Search and Data Mining. ACM. [34] Antonis Matakos, Evimaria Terzi, and Panayiotis Tsaparas. 2017. Measuring and [10] Michael D Conover, Jacob Ratkiewicz, Matthew Francisco, Bruno Gonçalves, moderating opinion polarization in social networks. Data Mining and Knowledge Filippo Menczer, and Alessandro Flammini. 2011. Political polarization on twitter. Discovery 31 (2017), 1480–1505. In Fifth international AAAI conference on weblogs and social media. [35] Antonis Matakos, Sijing Tu, and Aristides Gionis. 2020. Tell me something my [11] Cristian Consonni, David Laniado, and Alberto Montresor. 2019. WikiLinkGraphs: friends do not know: diversity maximization in social networks. Knowledge and A complete, longitudinal and multi-language dataset of the Wikipedia link net- Information Systems 9 (2020), 3697–3726. works. In Proceedings of the International AAAI Conference on Web and Social [36] Alfredo Jose Morales, Javier Borondo, Juan Carlos Losada, and Rosa M. Benito. Media, Vol. 13. 598–607. 2015. Measuring political polarization: Twitter shows the two sides of Venezuela. [12] Alessandro Cossard, Gianmarco De Francisci Morales, Kyriaki Kalimeri, Yelena Chaos: An Interdisciplinary Journal of Nonlinear Science 25, 3 (2015), 033114. Mejova, Daniela Paolotti, and Michele Starnini. 2020. Falling into the Echo Cham- [37] Cameron Musco, Christopher Musco, and Charalampos E. Tsourakakis. 2018. ber: The Italian Vaccination Debate on Twitter. In Proceedings of the International Minimizing Polarization and Disagreement in Social Networks. In Proceedings of AAAI Conference on Web and Social Media. the 2018 World Wide Web Conference on World Wide Web - WWW ’18. [13] Alexander Dallmann, Thomas Niebler, Florian Lemmerich, and Andreas Hotho. [38] Ashwin Paranjape, Robert West, Leila Zia, and Jure Leskovec. 2016. Improving 2016. Extracting Semantics from Random Walks on Wikipedia: Comparing website hyperlink structure using server logs. In Proceedings of the Ninth ACM Learning and Counting Methods.. In Wiki@ ICWSM. International Conference on Web Search and Data Mining. 615–624. [14] Dimitar Dimitrov and Florian Lemmerich. 2019. Democracy and difference: Dif- [39] Tiziano Piccardi, Michele Catasta, Leila Zia, and Robert West. 2018. Structuring ferent topic, different traffic: How search and navigation interplay on Wikipedia. Wikipedia articles with section recommendations. In The 41st International ACM The Journal of Web Science 6 (2019), 67–94. SIGIR Conference on Research & Development in Information Retrieval. [15] Dimitar Dimitrov, Philipp Singer, Florian Lemmerich, and Markus Strohmaier. [40] Alessandro Piscopo and Elena Simperl. 2019. What we talk about when we talk 2016. Visual positions of links and clicks on wikipedia. In Proceedings of the 25th about Wikidata quality: a literature survey. In Proceedings of the 15th International International Conference Companion on World Wide Web. 27–28. Symposium on Open Collaboration. 1–11. [16] Dimitar Dimitrov, Philipp Singer, Florian Lemmerich, and Markus Strohmaier. [41] Miriam Redi, Besnik Fetahu, Jonathan Morgan, and Dario Taraborelli. 2019. 2017. What makes a link successful on wikipedia?. In Proceedings of the 26th Citation Needed: A Taxonomy and Algorithmic Assessment of Wikipedia’s Veri- International Conference on World Wide Web. 917–926. fiability. In The World Wide Web Conference. [17] Dimitar Dimitrov, Philipp Singer, Florian Lemmerich, and Markus Strohmaier. [42] Manoel Horta Ribeiro, Raphael Ottoni, Robert West, Virgílio AF Almeida, and 2017. What Makes a Link Successful on Wikipedia?. In Proceedings of the 26th Wagner Meira Jr. 2020. Auditing radicalization pathways on youtube. In Pro- International Conference on World Wide Web. ceedings of the 2020 Conference on Fairness, Accountability, and Transparency. [18] Besnik Fetahu, Katja Markert, Wolfgang Nejdl, and Avishek Anand. 2016. Finding 131–141. news citations for wikipedia. In Proceedings of the 25th ACM International on [43] Aju Thalappillil Scaria, Rose Marie Philip, Robert West, and Jure Leskovec. 2014. Conference on Information and Knowledge Management. 337–346. The last click: Why users give up information network navigation. In Proceedings [19] Seth Flaxman, Sharad Goel, and Justin M Rao. 2016. Filter bubbles, echo chambers, of the 7th ACM international conference on Web search and data mining. 213–222. and online news consumption. Public opinion quarterly 80, S1 (2016), 298–320. [44] Feng Shi, Misha Teplitskiy, Eamon Duede, and James A Evans. 2019. The wisdom [20] Andrea Forte, Vanesa Larco, and Amy Bruckman. 2009. Decentralization in of polarized crowds. Nature human behaviour (2019). Wikipedia governance. Journal of Management Information Systems 26, 1 (2009), [45] Philipp Singer, Florian Lemmerich, Robert West, Leila Zia, Ellery Wulczyn, 49–72. Markus Strohmaier, and Jure Leskovec. 2017. Why we read wikipedia. In Pro- [21] Kiran Garimella, Gianmarco De Francisci Morales, Aristides Gionis, and Michael ceedings of the 26th International Conference on World Wide Web. 1591–1600. Mathioudakis. 2017. Reducing Controversy by Connecting Opposing Views. In [46] Philipp Singer, Thomas Niebler, Markus Strohmaier, and Andreas Hotho. 2013. Proceedings of the Tenth ACM International Conference on Web Search and Data Computing semantic relatedness from human navigational paths: A case study Mining (WSDM ’17). on Wikipedia. In International Journal on Semantic Web and Information Systems [22] Kiran Garimella, Gianmarco De Francisci Morales, Aristides Gionis, and Michael 9. 41–70. Mathioudakis. 2018. Political discourse on social media: Echo chambers, gate- [47] Claudia Wagner, Eduardo Graells-Garrido, David Garcia, and Filippo Menczer. keepers, and the price of bipartisanship. In Proceedings of the 2018 World Wide 2016. Women through the glass ceiling: gender asymmetries in Wikipedia. EPJ Web Conference. 913–922. Data Science (2016). [23] Kiran Garimella, Aristides Gionis, Nikos Parotsidis, and Nikolaj Tatti. 2017. Bal- [48] Robert West and Jure Leskovec. 2012. Human wayfinding in information net- ancing information exposure in social networks. In Advances in Neural Informa- works. In Proceedings of the 21st international conference on World Wide Web. tion Processing Systems. 4663–4671. [49] Wikipedia. [n.d.]. Manual of Style/Linking. In https://en.wikipedia.org/wiki/ [24] Kiran Garimella, Gianmarco De Francisci Morales, Aristides Gionis, and Michael Wikipedia:Manual_of_Style/Linking. Mathioudakis. 2018. Quantifying controversy on social media. ACM Transactions [50] Wikipedia. [n.d.]. Namespace. In https://en.wikipedia.org/wiki/Wikipedia: on Social Computing (2018). Namespace. [25] Patrick Gildersleve and Taha Yasseri. 2018. Inspiration, captivation, and misdi- [51] Wikipedia. [n.d.]. Neutral Point of View. In https://en.wikipedia.org/wiki/ rection: Emergent properties in networks of online navigation. In International Wikipedia:Neutral_point_of_view. Workshop on Complex Networks. Springer, 271–282. [52] Wikipedia. [n.d.]. Redirect. In https://en.wikipedia.org/wiki/Wikipedia:Redirect. [26] Eduardo Graells-Garrido, Mounia Lalmas, and Filippo Menczer. 2015. First [53] Ellery Wulczyn and Dario Taraborelli. 2017. Wikipedia Clickstream. https: Women, Second Sex: Gender Bias in Wikipedia. In Proceedings of the 26th ACM //doi.org/10.6084/m9.figshare.1305770.v22 (2017). Conference on Hypertext & Social Media. [54] Ellery Wulczyn, Robert West, Leila Zia, and Jure Leskovec. 2016. Growing [27] Denis Helic, Markus Strohmaier, Michael Granitzer, and Reinhold Scherer. 2013. wikipedia across languages via recommendation. In Proceedings of the 25th Inter- Models of human navigation in information networks based on decentralized national Conference on World Wide Web. 975–985. search. In Proceedings of the 24th ACM conference on hypertext and social media. 89–98. [28] Brian Keegan, Darren Gergle, and Noshir Contractor. 2011. Hot off the wiki: dynamics, practices, and structures in Wikipedia’s coverage of the Tohoku¯ catas- trophes. In Proceedings of the 7th international symposium on Wikis and open collaboration. 105–113. [29] Tobias Koopmann, Alexander Dallmann, Lena Hettinger, Thomas Niebler, and Andreas Hotho. 2019. On the right track! Analysing and predicting navigation success in Wikipedia. In Proceedings of the 30th ACM Conference on Hypertext