Who Wants to Read This?: a Method for Measuring Topical Representativeness in User Generated Content Systems Amanda Menking1, David W

Home , Conservapedia

Who wants to read this?: A method for measuring topical representativeness in user generated content systems Amanda Menking1, David W. McDonald2, Mark Zachry2 1The Information School, 2Human Centered Design & Engineering University of Washington Seattle, WA, USA {amenking, dwmc, zachry}@uw.edu

ABSTRACT (GOMI), The Potterverse, and Creepypasta have very This methods paper details an approach for identifying the specific content offerings—ranging from blog reviews to representativeness of content in a user generated content Harry Potter trivia and fan fiction to paranormal stories— (UGC) system while also accounting for endogeneity bias. created and consumed by users who have expressed interest We leverage metadata from an independent content in these topics and chosen to become members of these provider to generate sets of commercially viable terms very specific UGC systems. That these UGC systems presumed to be of interest to specific audiences linking represent the interests and biases of their users would meet those terms to UGC. We describe our method and heuristics our expectations of those systems. at a level of detail allowing others to follow or modify it to study both content representativeness and content gaps in However, some UGC systems, like Wikipedia, do not UGC systems. We illustrate the method by investigating require membership for users to view, edit, or contribute how well the English language Wikipedia addresses the content. In fact, Wikipedia expresses a desire to be a content interests of four sample audiences: readers of men’s general source of information potentially anyone and and women’s periodicals, and readers of political everyone finds interesting. Wikipedia also promises to be periodicals geared toward either liberal or conservative “the encyclopedia anyone can edit.” Yet researchers and ideologies. We also share preliminary findings from each community members alike have recognized that many of case study to demonstrate our method. Wikipedia’s social factors affect the makeup of their contributor base [e.g., 39,42], and some of Wikipedia’s Author Keywords policies make it more difficult for certain kinds of content Methods; user generated content; representativeness; to be included [51]. Moreover, studies have identified how content gaps; Wikipedia topical skews in content increase conflict in UGC systems ACM Classification Keywords [29] and influence search results [28]. Our goal is to H.5.3. Information interfaces and presentation (e.g., HCI): develop a systematic and reproducible method for assessing Group and Organization Interfaces -- Theory and Models. the representativeness of content in a UGC system. General Terms: Design, Measurement BACKGROUND AND MOTIVATION INTRODUCTION Our method is focused on understanding representativeness Assessing topical representativeness in user generated in proportion to specific, identifiable populations. We use content (UGC) systems is a difficult and ambiguous task. representativeness to describe whether topical content is On the one hand, content often represents the interests of present in a UGC system in proportion to likely consumers contributors. Take, for example, a UGC system like of that topic. We do not use representativeness to imply Ravelry, “a knit and crochet community” dedicated to aspects or the degrees to which a specific audience or supporting users who are interested in fiber arts. While it is population is the focus of content in a UGC system [e.g., free to sign up for, browse, and contribute to Ravelry, one 49]. That is, we draw a distinction between must register to access the subject-specific content the site representativeness and representation, or forms of provides. Likewise, UCG systems like Get Off My Internets representation in a UGC system. Also, we do not use representativeness to describe how some UGC may or may Permission to make digital or hard copies of all or part of this work for not mirror societal biases and inequities. Moreover, we do personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies not use representativeness to examine the degree to which bear this notice and the full citation on the first page. Copyrights for topical content is present across a UGC system. (For components of this work owned by others than the author(s) must be example, our method does not consider the frequency of a honored. Abstracting with credit is permitted. To copy otherwise, or topic and does not explicitly address the density of topical republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from coverage, although it could be extended to address these [email protected]. CSCW '17, February 25-March 01, 2017, Portland, aspects.) OR, USA Copyright is held by the owner/author(s). Publication rights licensed to ACM. ACM 978-1-4503-4335-0/17/03…$15.00 DOI: Because our method considers content in relation to http://dx.doi.org/10.1145/2998181.2998254 identifiable populations of users, our method is useful for studying large-scale collaborative systems and not systems thereby inform our understanding of the representation of created or used by individuals. This approach makes the the object of the content. method more applicable to CSCW, social computing and Some recent work by Sengul-Jones identifies the challenge other disciplines that are interested in the relationship of focusing on identifying gaps. In her report for a between content, the groups who create that content, and Wikimedia Foundation Individual Engagement Grant the possible consumers of that content. entitled “Full Circle Gap Protocol,” she points out that Furthermore, our method complements methods used to identifying the “unknown unknowns” becomes a “bestiary evaluate total coverage of topical content. Coverage is a of gaps” [46]. She finds gaps derive from and are tied to a measure of the degree to which some total possible content range of factors such as infrastructural access, skill and is present in the UGC system. Many studies of Wikipedia literacy divisions, time and interest gaps, emotion work, content are focused on understanding coverage or the lack and knowledge-legitimacy exclusions [47]. Many of these of coverage (gaps in coverage). While the method we items are specific characteristics and biases present in the describe is intended to help understand representativeness range of skills and knowledge of individual contributors. In of topical content within a given UGC system, a side effect short, gaps and omissions are difficult to find, and many of of our method is the ability to identify both coverage and the reasons why they exist may be inherent in the UGC potential gaps or omissions. system, a function of social and political forces, or an aspect of the users’ lived experiences. Measuring Representation and Finding Gaps Through different research methods, prior work has These studies help us understand coverage or gaps in considered both gaps in topical content on Wikipedia—a Wikipedia, but many of the techniques they employ fail to prominent and commonly studied UGC system—and the provide a general method for assessing other UGC systems. different ways in which Wikipedia’s content represents Further, some of these approaches are specific to a different populations. For example, Omnipedia, a system particular topic or aspect of Wikipedia and may not apply to that allows users to interact with 25 different language other UGC systems that may represent topical content versions of Wikipedia, visualizes content similarities and differently from the way it is represented in Wikipedia. differences between language editions [4]. This approach However, the challenge of assessing representativeness illustrates potential gaps in content based on cultural or prompts other problems that need to be addressed first. linguistic differences. However, mutually missing content that is a function of the participants’ common biases is not Identifying Topical Content: Endogenous versus Exogenous Source detected. Our attempt to assess content representativeness faced a Other prior work has analyzed the distribution of content number of challenges. One problem is the scope of potential into a set of content categories. Work by Kittur, Chi & Suh topics. In some prior work, the methods to identify subsets (2009) built upon existing work by [18,25] to identify are very narrow and do not represent the broad interests of topical distribution of articles in Wikipedia based on user potential content consumers. Representativeness would fail generated annotations of categories. Their approach in that case because the population of potential consumers revealed the category of popular culture accounted for a is so narrow as to be difficult to compare article coverage to large proportion of articles while other content categories potential readership. Studies like those of biographies [e.g., represented very small proportions of articles. While this 16,52,53] or controversial issues [e.g., 26] are topically approach helps us understand the distribution of content, it focused and provide an understanding of those topics, but does not tell us what content is missing, or whether the cannot tell us much about representativeness. In other prior distribution of articles in the content categories effectively work, the methods generate broad subsets of content, but reveals anything about coverage of topics or the the potential population of content consumers is diffuse in representativeness of the content. That is, this approach such a way as to make judging representativeness of the does not answer the question: Who may want to read this? content untenable. For example, very broad and diffuse readerships of content related to ‘work’, ‘relationships’ and Another approach to understanding who and what is ‘leisure’ [6] would make it hard to know whether these represented is to study the composition of the content. In topics are reasonably represented within Wikipedia or any the case of Wikipedia, researchers have studied articles to other UGC system. understand what they address. For example, recent studies have considered how phrases missing from Wiktionary Another underlying problem is that—within any UGC might be identified and eventually incorporated [57], how system—differential participation may yield content skews. articles about women might be expanded by pulling missing For example, in the context of Wikipedia, a well- data from obituaries [36], and how German and English documented skew in participation [23] may result in a skew language Wikipedia articles about drugs might be improved in topical content. Therefore, any sampling method that by comparing them to pharmacology textbooks [31]. These relies on content from Wikipedia may be unknowably approaches focus on the composition of the content and biased from the beginning. All of these concerns point to a broader challenge for Step 1: Generating Lists of Topics Associated with techniques that purport to assay any aspect of a UGC or Specific Populations social media system: the challenge of endogeneity. Instead of settling for existing approaches that have some Endogeneity, the state of being endogenous, or coming clear drawbacks (e.g., endogeneity bias, narrow topical from within, is an underappreciated problem for many prior focus), we developed a method that works independently studies of UGC systems and social media. We will not from the unique characteristics of the UGC system under address the problems of these studies that have clear biases study to generate a large set of topics associated with the based on endogenous sampling. We merely raise the issue interests of a given population. here to acknowledge that the challenge is pervasive and that To generate our initial topic lists, we first identify content our method is one of the few methods designed to free an providers (e.g., magazines, journals, monographs, etc.) analysis from an endogeneity bias. commercially targeted to a population of interest. The In our review of the literature, we found two notable selection of this population must be executed with exceptions that also made an effort to address the challenge sensitivity to two of the considerations discussed earlier. of endogenous biases by relying on exogenous sources. First, the population must not be too narrow, identifiable Reagle & Rhue’s (2011) study compares women’s only with an un-differentiatable content area. Additionally, biographies in Wikipedia to women’s biographies in the the population must not be too broad or diffuse, making it online Encyclopædia Britannica. This study shows the impossible to identify meaningful groups within it that degree to which Wikipedia and Encyclopædia Britannica would likely have distinguishable topical interests. For the covered similar content. The other study is by Klein & purposes of this method we call that target population a Konieczny’s (2015). They compare “a Wikipedia-derived target demographic or, simply, demographic. gender inequality indicator (WIGI)” to four commonly used With the population selected, we then identify specific gender inequality indices. These exceptions can help expose content sources targeted to this population, using widely bias in coverage that may be a function of the UGC available reports from the publishing industry about participants’ choices, but cannot tell us the degree to which publications marketed to different demographics and the existing content is representative of the potential circulation data. From this report data, we select the most readership. widely circulated sources based on published circulation Our method attempts to address these challenges (i.e., too data. Additionally, we select sources based on the criteria narrow of a scope, overly diverse potential readership, and that they were indexed uniformly. In the cases below, we endogeneity bias) by relying on an external content used presence in the most prominent U.S.-based indexer of provider. In the next section, we describe how we identified popular magazines, EBSCO. The selection criteria applied potential consumer populations and how we derived content when using this database can be adjusted as necessary to fit metadata that would facilitate a match to content in different research questions (e.g., time periods, number of Wikipedia. content sources to consider). METHOD The method we developed requires four steps to measure how well topical content aligns with potential readership populations (i.e., users) of a UGC system, like Wikipedia. The first step is to use an exogenous source to generate a set of unique topics of presumed interest to a readership population. The second step is to differentiate among identifiable groups within this population to isolate topics that are representative of the interests of these groups and not the entire population. The third step is to assess how those topics are covered in the system through the application of heuristic-guided search to resolve results that are not an exact match. Finally, the fourth step is to use an external, authoritative source to identify the size of the Figure 1. A screenshot of a database record for a content groups within the overall population and compare those source (e.g., Rolling Stone) with subjects and other metadata. numbers to the proportion of how their interests are We derived our topics from the subject category (“subj”). represented in the UGC system. Abstracting services, like EBSCO index a wide variety of content. Often these indexing records contain a wealth of The high-level details about how the method is executed are potential fields that do not apply to every indexed and discussed in the following subsections. Additional details of abstracted form of content. This screenshot shows only a few of the method and its application are then illustrated in two the relevant fields for the selected sources. Similar fields (not cases later in this paper. shown) might be useful when applying this method, but when choosing a different media as the initial topical source. Queries of the EBSCO database for the targeted publication For our demonstrations below, we developed a set of title in a given year yielded a data file including all indexed heuristics to match topics to Wikipedia article titles. items (articles) for the year. Each item included several We relied on the English language Wikipedia’s native fields of article-associated metadata, including, titles, search feature as the basis for generating a possible article subtitles, author(s), abstracts, and a list of subject terms match. This is a likely mechanism an average user would (topical keywords). (See Figure 1.) employ. Another approach would be to use a Google In our cases, we chose to use periodicals with the highest search, using the features of Google to constrain the search circulation numbers in the English language. For each to the domain of the specific UGC. We do note that the periodical, we remove all fields except the subject terms. Google search approach might yield differential results We then combine the subject terms for each periodical title because of the way that Google uses data (e.g., geo- for a demographic into a single list. For each of these lists, location, account preferences) that it has about an all duplicate items are removed. individual to tailor and shape its search results. Our process involved entering each of our topics into the Wikipedia These subject terms—or keywords—are generated by the search field and evaluating the results. Based on our abstracting service following traditional knowledge heuristics, we counted and tracked the search result for each organization (i.e., library science) practices. The keywords topic. The topic lists include a primary topical keyword (in are not generated by the editorial practices of the all caps) with some additional terms that clarify the topic. periodicals, or by us. (Detailing the methods for creating Some topics include additional parenthetical descriptors, subject terms and the practices of indexing publications comma series, or descriptors after dashes. When entering materials are beyond the scope of this paper and outside the topics into Wikipedia’s search engine, we used only top- bounds of the method that we define here.) level topic keywords and no extended descriptors. Step 2: Differentiating Groups and Associated Topics within the Overall Population Proper names of persons are a special case for our search Through Step 1, the method has produced a set of items that and match method. The majority of abstracting services are indicative of what topics are covered in a set of content handle proper names of people in a “LASTNAME, sources for each target demographic. To facilitate an Firstname” strategy. This is not the editorial standard for assessment of representativeness, though, the topic list Wikipedia. The current standard in Wikipedia is “Firstname associated with each demographic must be sorted into sub- Lastname.” Except, in Wikipedia, a significant number of lists. In the cases reported below, we achieved this by well-known people will have a “Lastname, Firstname” initially selecting periodical titles targeted to two redirect. In this special case, proper names of people, we identifiable groups maintaining separate lists of content modified how we entered the term metadata to align with terms for each demographic. These lists of terms could be the standards of the UGC system. That is, we swapped the used to assess general topical coverage for the interests of “Lastname, Firstname” data from the abstracting service to each target demographic. We can move beyond the align with the “Firstname Lastname” assumption of question of coverage to assess representativeness by Wikipedia. The results of that search were then used. generating a list of topical subject terms for each The heuristics we used for counting matches between topics demographic that is unique to that demographic. These and specific Wikipedia articles are as follows: unique lists are the disjoint sets of terms across all target demographic groups. In the demonstrations below, we 1. If the search yielded a direct match, we recorded the compare two lists and remove all items that appear in both name of the article page and noted it. lists. We point out that the intersection of terms across the 2. If the search resulted in a redirect, we recorded the name target demographics could also yield some interesting of the article page and noted it was the result of a insights, but that is not the way we implemented our redirect. assessment of representativeness in our demonstration cases 3. If the search resulted in a list of possible matches via a below. search results page, we carefully considered each item, with the goal of selecting the lowest ordinal result that Step 3: Detecting Topical Coverage in the UGC System seemed to be the closest match given additional (Wikipedia) information provided by any parenthetical descriptors The next step is to apply the differentiated list of target from the term list, and we noted the ordinal of that result terms to assess whether specific content exists in a given item. The research team reviewed these cases to UGC system. In the demonstration cases below we applied determine how they should be counted. the terms to the English language Wikipedia. The generated 4. If the search resulted in a disambiguation page, we topic lists provide a baseline for measuring coverage of scanned the suggestions to identify the closest match, topics presumed to be of interest to readers from each relying on any additional information provided by any demographic in the UGC system. The next challenge is to parenthetical descriptors from the term list. match topical terms to UGC to assess whether the topic from the external sources are currently present in the UGC. 5. If the search resulted in an article that was a list rather connections between the two, and—for the purpose of this than an actual encyclopedic article, we noted it and study—we treat them as separate, identifiable populations. counted it as a non-match. We chose these two cases to demonstrate the method 6. If the search did not result in a match—either through a because, for both cases, (a) there are well established direct hit, a redirect, a search result selection, or a periodicals with targeted, commercially viable readerships; disambiguation page—we marked the term as not (b) there is existing work about the domain and UGC; and having a corresponding article in Wikipedia. Many (c) there is some debate regarding the coverage of the searches yielded a “does not exist” message generated domains within Wikipedia. In the case of politics, the by Wikipedia. English language Wikipedia has been portrayed as having a Adjudicating Imprecise Matches liberal bias [e.g., 35,48], and Conservapedia—created to These heuristics were supplemented with team review counter Wikipedia—claims to be “free of liberal untruths” discussions to adjudicate imprecise matches. Review of [14]. In the case of gender, Wikipedia has a known “gender these imprecise matches was necessary because naming gap” in participation [e.g., 23] and the inclusion of articles conventions in Wikipedia are not consistent, and typos perceived to be of more interest to women is often occur in the databases of abstracting services. For example, contested [e.g., 7]. if a search did not result in a match—either through a direct hit, a redirect, a search result selection, or a disambiguation Also, for both cases, recent polls and census data for U.S. page—because the metadata lacked sufficient information, residents provide a baseline against which to measure such as, birth dates for individuals, we discussed the item potential representativeness as a function of a general and possible matches. In our review decisions, we erred on population. The 2015 Gallup poll reports there are now the side of inclusion. However, we also made exceptions if more people who identify as conservatives (38% of we agreed common sense challenged the results of our respondents) than as moderates (34%) or liberals (24%) heuristics. For example, when a search for “Apologizing” [45]. And the most recent U.S. Census reports there are led to a disambiguation page listing “An expression of slightly more respondents who identify as women (50.8%) remorse” with a hyperlink to the article “Remorse” as the than as men (49.2%) [2]. Therefore, if an “open” UGC first—and most applicable result—we reviewed the article, system like Wikipedia is representative of the interests of discussing it as a team, and decided “Remorse” was not a these populations, we would expect to see distributions of match for the keyword “Apologizing.” Additionally, we did topical content in relation to the possible interests of each not include matches that resulted in article sub-sections audience. rather than complete standalone articles. Some may argue periodicals such as popular magazines and Step 4: Comparing Representation of Topics in the UGC mass media in general are not good sources to use to elicit to the Population what is and is not a topic of interest to “men” or “women,” The final step in our method is to use an external, or “conservatives” or “liberals.” The common argument authoritative source to identify the proportion of the target against these sources is that the commercial needs of demographic within the overall population and then popular content providers serve only to reify binaries and compare those numbers to the proportion of how their stereotypes. That a binary may or may not be reified in the interests are covered in the UGC system. The details about content of the individual sources is not the focus of this how we executed this step in the two case studies are analysis. Nor was our goal to examine how gender and presented below. political parties are socially constructed. Rather, our goal was to find sources external to Wikipedia that exhibit both DEMONSTRATING THE METHOD established readership and commercial viability. Further, In the following sections, we explain why we chose our two the established commercial viability of these sources makes cases to test our method, frame each case with related work, claims to the interests and desirability of the topics, if not present how we worked through the method in each case, the specific presentation of the content. This means that the and share preliminary findings as demonstrative data of our potential readership of a UGC might want to learn more method. about these topics, even if it were from a different Choosing the Cases perspective than provided in the given external source. In this study, we focus on assessing the relative Furthermore, because of Wikipedia’s policies regarding representativeness of content in the English language notability [55,56], searching for keywords associated with Wikipedia by applying the method described above to two content published in popular periodicals with high cases: (1) topics perceived to be of interest to readers of distribution is likely to reflect the potential inclusion of the “women’s” and “men’s” periodicals; and (2) topics associated content in Wikipedia simply because an external, perceived to be of interest to readers of “conservative” and verifiable source is available. “liberal” periodicals. We understand there is often overlap Demonstration Case 1: Political Ideologies between these audiences, but we do not hypothesize any Recent events have drawn public attention to the ways in which UGC systems—and social media in particular— reflect and drive political discourse, sometimes using U.S. and accounting for whether they had been indexed unexpected and difficult to detect mechanisms. For uniformly by EBSCO. In our first case, we chose example, Facebook has been accused of manipulating its periodicals targeted to “conservative” and “liberal” readers. “trending topics” to reflect anti-conservative bias [e.g., 34], (See Table 1.) There is one notable aspect of our method activists claim Twitter has censored #WhichHillary, an anti- that arises here. The publication Cosmopolitan is a very Hillary Clinton hashtag [36], and TwitterAudit suggests that high circulation magazine that identifies its target 75% of Donald Trump’s followers are bots rather than readership as both “women” and “liberal.” In building our people [58]. example cases, we decided not to include it here and instead used Cosmopolitan for our second demonstration below. In the case of Wikipedia, five years after it was established We made this choice because the primary audience is in 2001, Andrew Schlafly created Conservapedia in women. We recognize political ideology and gender are not response to what he perceived to be the liberal biases separable, but as a demonstration this separation was a perpetuated by the growing UGC system [1]. RationalWiki, clearer way to illustrate the method. We replaced a wiki that does not claim to be an encyclopedia or value Cosmopolitan with the periodical that had the next highest neutral point of view, was created in 2007 as a counter to circulation in the U.S. Conservapedia after some editors were banned from the latter. Though Conservapedia and RationalWiki have failed Periodicals Marketed to Periodicals Marketed to to grow at the same rate and in the same ways as Liberals Conservatives Wikipedia, the contributors to these UGC systems remain Mother Jones American Spectator active and continue to maintain that Wikipedia does not represent their interests. The Nation National Review The New Republic Reason Scholars have studied political biases and content coverage in traditional media for decades. More recently researchers Rolling Stone Saturday Evening Post have begun to think about how UGC reflects political ideologies and how both citizens and politicians leverage Table 1. Periodicals with the highest subscribed circulation in the U.S. by targeted readership (i.e., category). UGC systems [e.g., 21,22]. Greenstein & Zhu (2012) sought to assess the political bias of Wikipedia content Following the method described above, we removed all relative to the ideology of the two principal U.S. political fields except the subject terms for one year’s worth (2014) parties. They found articles are created with “Democrat of metadata for each periodical. We then combined the leanings” but tend to become more neutral with time and subject terms for each of the four periodical titles in a additional revisions, and that the total bias in Wikipedia category (e.g., “conservative”) into a single list. This changes as content is added—though the slant of individual resulted in 4,576 terms from “conservative” periodicals and articles may not change significantly. Similarly, Kalla & 7,403 terms from “liberal” periodicals. For each of these Aronow (2015) looked at articles about U.S. senators to lists, all duplicate items were removed, leaving 3,313 determine whether negative or positive sentiments persist, unique terms from “conservative” periodicals and 6,140 to find a bias toward positivity over time. Brown (2011) unique terms from “liberal” periodicals. This gave us one also considered existing coverage of articles related to list of subject terms (or keywords) for 2014 from the four politics, looking at both U.S. politicians’ biographies and “liberal” periodicals listed above and a separate list for articles about U.S. election results. He found coverage was subject terms for 2014 from the four “conservative” good for recent and popular topics, but that there were periodicals listed above. omissions for older and more obscure topics. We generated two sets of disjoint topic terms by removing

all terms that were common to the set of terms (keywords) These studies about Wikipedia and politics have paid between the “liberal” list and the “conservative” list. careful attention to and used various techniques to analyze the content that exists within Wikipedia, what we defined as We then used a random number generator to select 400 the representation, but they have not addressed the issue of terms from each category list and searched for the resulting representativeness, especially as it relates to broad areas of 800 terms using Wikipedia’s native search feature. We interest. Our goal in this case was to use the method to test recorded what we found using the heuristics described whether the English language Wikipedia reflects general above for each search term and met to adjudicate imprecise representativeness of topics that are presumed to be of matches. At times, there were interesting instances that interest to an established readership of “conservative” and generated no matches until we slightly changed the entry of “liberal” periodicals. the metadata. For example, “DUOLINGO Inc.” did not result in any matches, but “Duolingo” resulted in a direct Applying the Method As noted above, we chose sets of periodicals targeted to hit. These results were put into a category of “Direct hit specific audiences based on the highest circulation in the edited term” to differentiate them from a clear direct match. Demonstrative Data was conceptually vague without access to the original We found that 73.8% of the randomly selected 400 material, etc.) favored the inclusion of content represented keywords from conservative-oriented periodicals were by keywords taken from the “conservative” list. We covered, and 81.5% of the randomly selected 400 keywords excluded twice as many keywords from the list taken from from liberal-oriented periodicals were covered. This “liberal” magazines (38 were excluded) than from represents a 7.7% difference in topical coverage. Overall, “conservative” magazines (19 were excluded). The majority 18.5% of the randomly selected topics from periodicals of excluded terms were either too vague (e.g., targeted to “liberal” readers and 26.3% of the randomly “PROGRESS”), or compound keywords (e.g., “DRUGS & selected topics from periodicals targeted to “conservative” crime”). readers did not have corresponding articles. (See Figure 2.) Summary Our preliminary findings suggest the English language Wikipedia is more likely to include topics presumed to be of interest to readers of “liberal” periodicals. In the case of political ideology, then, there is a clear need to investigate biases and gaps, and our method provides a systematic approach of identifying articles that might be of interest to “liberal” and “conservative” readers. Demonstration Case 2: The “Gender Gap” Recent UGC studies have begun to unpack the complex implications of participation and content differences among the genders. For example, women who contribute reviews to the Internet Movie Database (IMDb) enjoy less prestige and smaller audiences, even when they adjust their communication styles to mimic those of men who contribute to the same site [20]. Similarly, Facebook content perceived as “male” receives more feedback, even when posted by users who identify as women [34], and Google image searches for many occupations result in both gender stereotypes and underrepresentation of women [28]. To investigate the potential gender valence of content, prior studies have analyzed conversation topics of passing strangers [6,40], online comments on New York Times articles [41], tweets about climate change [26], and images curated via Pinterest [10,38] to determine the kinds of topics that may be of interest to men or women. These studies generally rely on an inductive approach: first Figure 2. Summary of results for topics from sources identifying the gender (or, in some cases, biological sex) of marketed to "liberal" and "conservative" readers. The last 5 the contributor and then classifying the kinds of content he gray shaded rows were omitted from the analysis. or she contributed. Although this inductive approach The “direct match” heuristic was the most activated indicates the types of content contributed, it cannot heuristic in our study with a total of 144 terms from the list effectively speak to the representativeness of the total taken from “conservative” periodicals and 179 terms from amount of content in the system. the list taken from “liberal” periodicals having direct hits— or corresponding articles—when entered into Wikipedia’s Another approach [11] relies upon topics selected by the search feature. Alternately, another way to consider researchers to test how participants respond to gender representativeness is to note that only 67 keywords from the stereotypes. While this method represents one approach, it list from “liberal” periodicals did not have corresponding also presents methodological challenges as it generates articles, whereas 100 keywords from the list from results that can be difficult to reproduce, and—as a “conservative” periodicals did not have corresponding method—it is likely to introduce biases from the articles. idiosyncrasies of the researchers. In Figure 2, the bottom five shaded rows represent the In the case of Wikipedia, research [3,12,15,23,32,33] has number of terms excluded from the analysis. The number of identified differential participation in contributors who topics excluded from analysis (e.g., because they matched identify as men or as women, and the phenomenon has something that was not a legitimate article, because the come to be known as the “gender gap.” In 2015 the topic could not be effectively disambiguated, or the topic Wikimedia Foundation launched the Inspire Campaign with the goal of funding ideas that address the “gender gap.” periodicals listed above and a separate list for subject terms One idea creator wrote, “I see a lot of speculation here for 2014 from the four “men’s” periodicals listed below. about whether the average potential female editor of Wikipedia is interested in fashion [...] or in female scientists Periodicals Marketed to Periodicals Marketed to [...]. It is hard to tailor our recruitment without having more Women Men data” [50]. A different but similar proposal noted, “There is Cosmopolitan Esquire a lack of clarity over what subjects women are interested in, what articles they edit, whether more women on Wikipedia Ladies Home Journal GQ would mean more coverage of certain areas, etc. We should Redbook Family Handyman research this rather than guessing” [44]. Although neither idea generated a fundable proposal, both touched on a Woman’s Day Men’s Health challenging question. They also troubled the implicit assumption that a diversity of contributors in an “open” Table 2. Publications with the highest subscribed circulation in the U.S. by targeted readership (i.e., category). UGC system will ensure representative content. We generated two sets of disjoint topic terms by removing Direct attempts to measure Wikipedia’s content-based all terms that were common to the set of terms (keywords) “gender gap” have been focused on particular domains and between the “men’s” list and the “women’s” lists. do not generalize well across broad encyclopedic content. As mentioned above, Lam et al. [14] studied the quality of We then used a random number generator to select 400 articles corresponding to movies rated highly by users who terms from each category list and manually searched for the identify as either men or women, and other studies have resulting 800 terms using Wikipedia’s native search feature. focused on the number and breadth of women's biographies We recorded what we found using the heuristics described [e.g., 16,52,53]. While these studies provide insight and above for each search term and met to adjudicate imprecise support the hypothesis of a kind of “gender gap” in content, matches. they do not speak to a broad set of topics. Demonstrative Data A “gender gap” in content is difficult to examine. To The results of our method for randomly selected terms from determine whether there is a “gender gap” in content, one “men’s” and “women’s” periodicals are striking at a high must decide how content is—if it is—“gendered.” Because level. We found that 67.6% of “women's” topics and 84.1% the “gendering” of content is subjective and socially of “men’s” topics were covered. This represents a 16.5% constructed, identifying a baseline set of content to test for difference in the topical coverage of Wikipedia as it is gender affinity must pay careful attention to the conflation represented from periodicals targeted to a specific of biological sex and gender. However, from a pragmatic “gendered” readership. Overall, 15.9% of “men’s” topics perspective of operationalization and of taking a “first cut” had no corresponding article whereas 32.4% of “women's” at this complex issue, a study needs to exploit perceptions topics had none. (See Figure 3.) of gender as a binary rather than as a continuum of a set of Considering the data a little more closely reveals some symbolic norms [19], or a performance [9]. Thus our goal interesting details. First, our stance was to come up with in this case, was to use our method to assay whether heuristics that would select an article from Wikipedia if the Wikipedia reflects general representativeness of topics that article existed in some clear form. That is, we biased toward are presumed to be of interest to an established readership inclusivity. This is an interesting decision because the of “women’s” and “men’s” periodicals. “direct match” heuristic yielded nearly the same number of Applying the Method articles for “men’s” and “women’s” presumed interests For our second case, we chose periodicals that had the when taken in sum across both samples (159 articles of largest distribution numbers in the U.S., were uniformly “women’s” topics, 160 articles of “men’s” topics). This indexed by EBSCO, and were targeted to either men or illustrates there are a number of ways the issue of content women readers. (See Table 2.) representativeness can be operationalized—some of which will show a difference and some of which might not. We then followed the same process described above for aggregating one year’s worth (2014) of metadata for each An alternative operationalization of representativeness periodical. We combined these lists of subject terms (or might come from an exclusionary perspective that considers keywords) into a set for each category (e.g., “women’s”). only when articles do not exist. That perspective tells a Initially, this gave us 4,567 terms from “women’s” slightly different story than the inclusionary perspective periodicals and 4,713 terms from “men’s” periodicals. We with only 58 “men’s” interest topics resulting in no possible then removed duplicates, leaving 4,039 unique terms from match whereas 120 “women’s” interest topics generated no “women’s” periodicals and 3,904 unique terms from possible match. From this perspective, there are twice as “men’s” periodicals. This gave us one list of subject terms many missing articles of possible interest to readers of (or keywords) for 2014 from the four “women’s” “women’s” periodicals than there are missing articles of possible interest to readers of “men’s” periodicals. participation gap nor does it propose a solution to those problems in Wikipedia or other UGC systems. However, it does support the findings of existing work regarding gaps in specific kinds of content, and it suggests the English language Wikipedia does bias toward an inclusion of content presumed to be of interest to readers of “men’s” periodicals. DISCUSSION Prior work that considers the content of UGC systems has often focused on understanding problems with content coverage, or a lack of coverage (a gap). Our contribution is a method that builds on the idea of coverage to consider topical coverage in the context of an identifiable potential readership or content consumer. We define that potential measure as representativeness. That is, we define representativeness as the proportion of content present in a UGC relative to the proportion of a potential consumer, audience or readership in a clearly identifiable population. Our approach is designed to address methodological challenges present in some of the prior work. In particular, our method is designed to circumvent the problem of biases resulting from endogeneity. The approach relies on content sources external to a UGC that are deemed relevant as a function of sustained readership and circulation. While

these external content sources may have their own editorial Figure 3. Summary of results for topics from sources biases, the topical focuses of the sources provide a clear marketed to women and men. The last 5 gray shaded rows scope of relevant content. Further we mitigate potential were omitted from the analysis. editorial biases by relying on more than one exogenous The “redirect” heuristic was the second most activated source when collecting the relevant metadata. heuristic in this study. This presents yet another perspective As a side effect, our approach can be used to assay topical on coverage. A redirect page is created when there is a content coverage. Applying our approach without relying belief that a term or phrase is directly synonymous with on readership populations or the relative readership another, such that a user looking for the one topic most demographic distributions can be interpreted as a measure often means the other. There were 54 article pages of general topical coverage in a UGC. In this way our identified from “women’s” interest topics and 85 article method is a more generalizable and repeatable method for pages identified from “men’s” interest topics based on the judging topical coverage. redirect heuristic. This is an approximately 2:3 ratio and either represents that some content has not been carefully Relying on exogenous sources provides another benefit. linked to a more likely topic of “women’s” interests, or that Understanding the coverage of a UGC relative to some the content is simply missing. known external sources provides a way to inspect the potential content biases in both the external sources and the Lastly, the number of topics excluded from analysis (e.g., UGC. For example, in our demonstration cases, the because they matched something that was not a legitimate editorial distinctions between the way proper names are article, because the topic could not be effectively handled in Wikipedia compared to standards established for disambiguated, or the topic was conceptually vague without the abstracting services would have allowed us to inspect access to the original material, etc.) was surprisingly stable some of the distinctions between well-known people and at 30-35 topics per randomly sampled 400 topics. less well-known people. Specifically, in the case of well- Summary known individuals it is likely that a Wikipedia editor has Our preliminary findings reveal some gaps in the created a redirect page so that the “Lastname, Firstname” representativeness of content on Wikipedia corresponding search works as well as the Wikipedia editorial standard of to that presumed to be of interest to readers of periodicals “Firstname Lastname.” targeted to women and men. We found some evidence that topics proven to be of commercial viability and notability Our method opens a number of possibly interesting research and perceived to be of interest to readers of “women’s” trajectories. In the way we executed the method, in both of periodicals are not represented in Wikipedia. This work, of our cases, we choose two points along a demographic course, does not resolve the implications of the continuum and implemented the method to create distinctions between the populations. In the case of gender their frequency in the targeted UGC system. For example, if we selected “men” and “women” and for political ideology the abstracting service was abstracting an article that talked we selected “conservative” and “liberal.” This particular about interesting new recipes that used apples, one of the choice is a convenience that makes the method a bit more terms generated might simply be “Apple (recipes).” Taking manageable. However, we could have picked multiple, that and counting the frequency with which the term identifiable readership populations along those dimensions. “apple” and “recipes” shows up in the UGC is probably not The method would only have to be changed in one simple what one wants as a measure of topical density – because way. Instead of using the metadata and creating two disjoint one has not measured a topic – one has measure a term sets of terms by removing the shared terms in the two frequency – and these are not the same. metadata sets, we would create n disjoint sets of terms, one One can also imagine modifying and applying our method disjoint set for each identifiable population and its to another UGC system like Quora, which has received associated metadata sources. The disjoint sets could then be media attention for having a both a known skew in used following the rest of the described procedures. participation and a misogynistic culture [5]. Instead of Another potential research trajectory is to consider the using a search feature, one could leverage Quora’s lists of shared interests of a given population. Our specific focus topical interests to investigate representativeness compared was to find representativeness that was a function of the, to exogenous sources, and then consider how sociotechnical supposedly, special or exclusive interests of each given features like up-voting and down-voting interact with topics population. Another approach would be to understand the presumed to be of more interest to women or men. degree to which supposed, shared interests of groups within Finally, our method begins to expose technical and the population were present in a given UGC. The shared infrastructural aspects of a UGC system that may not be interests potentially identify the strengths of UGC and peer obvious to the average consumer of UGC system. One production communities. That is, the assumption is that example has been briefly covered above, the nature of UGC communities contribute content that is a function of Wikipedia redirect pages. That a redirect handles differing the interests and expertise of the contributors. By focusing rules of “Lastname, Firstname” or “Firstname Lastname” on the terms and content metadata at the intersection might not be terribly problematic. Some recent work has (instead of the disjoint content) we could understand more focused on the impacts of failing to account for redirects about the shared interests of the contributors and how that [24] in many of the quantitative studies of Wikipedia. reflects the shared interests of the populations that might However, our case raises another concern. In our method consume or read the content. demonstration related to gender, the ratio of redirected Yet another possible research trajectory would be to use our content for “men’s” versus “women’s” topical terms yields method to identify topics that have been almost exclusively a question about semantic control over a term and how that targeted to one population (or readership) so that readers’ control can be made to bias for or against particular content. actual rather than presumed interests in these topics can be Consider this claim a little further by recalling that all of the interrogated through systematic sampling and repeated metadata we collected came from edited sources that had an observations. For example, one might elect to ask readers— article on the topic the term describes. This implies that all either directly or indirectly—questions about their political of the terms we were using had some reasonable claim to beliefs, and then measure the degree to which they are being a legitimate topic of an encyclopedic article. That interested in topics presumed to be of interest to them when there were more “men’s” interest related redirects makes presented with the opportunities to read about these topics those topics more prominent and cedes control of the outside of the context of the original content sources. redirected term to one interest group to the exclusion of Another extension of the method could consider another. The alternative is that a term could generate a frequencies of subject terms and the associated article disambiguation page, which we also considered. The main readership in the UGC. That is, one could use additional point is that the method we describe can help raise metadata from the indexing database to generate the set of consideration of these infrastructural aspects and make their subject terms weighted by frequency of occurrence during a implications more visible and, thus, more inspectable. given year. In a UGC system like Wikipedia, page view LIMITATIONS data could be used to understand how frequently the While our general method avoids endogeneity bias and topically associated page is requested. This data could then relies upon content proven to be commercially viable, we be used to understand whether editorially selected content recognize an editorial team’s choices to feature certain from exogenous sources corresponds to the way users kinds of content is as subjective as those made by a group request given content from a UGC system. of contributors, or by an algorithm. We also recognize that There is an interesting issue here with the problem of all taxonomies—even those built upon principles of judging frequencies as somehow equivalent to coverage knowledge organization and library science—evidence density. It would probably be a mistake to take terms (or some biases, but this is a larger issue beyond the scope of keywords) from an abstracting service and simply count our study. Admittedly our approach relies upon conceptualizations of Although challenging, developing—and sharing—a political ideologies as simplistic, bounded, and mutually systematic method for measuring representativeness and, exclusive, and of gender as tied to the binary of consequently, identifying potential gaps in content is an man/woman, a view often perpetuated by mass media and important task for researchers in CSCW and the broader popular culture. However, as mentioned above, the aim of HCI community. Despite the promises of increased access the method is not to challenge definitions or binaries but to and the normalization of social computing, we continue rather to develop a systematic and reproducible approach to to see existing inequalities reproduced and even reified examine how content perceived as being of more interest to online. If we want to design systems that encourage people specifically targeted audiences is represented on influential to engage in more pro-social or equitable ways and that UGC sites like Wikipedia. Further, we have mentioned serve as critiques of existing social norms, then we must above, a modification of the technique that would address first observe and understand existing behaviors and norms. interests at the intersection of one or more groups. Considering how UGC systems represent the presumed interests of different audiences and evidence gaps in content We also recognize some limitations of our specific can help us examine potential relationships between implementation of the general method. We selected diversity of participants and representativeness of content, magazines with the highest distribution in English in the implicit barriers to participation, and how popular, “open” United States, and we sampled only one year (2014) worth UGC systems may reinforce—rather than challenge— of terms from the content keywords. Sampling over several exclusionary policies and practices. years and including only topics that persist would most likely produce a more stable corpus for testing. ACKNOWLEDGMENTS Additionally, we tested our method using only the English We would like to thank Brian Keegan for early feedback, language Wikipedia. Finally, our method relies upon human and Siôn Romaine for his indispensible help with EBSCO. judgment rather than automation and is, therefore, We would also like to acknowledge support from National challenging—but not impossible—to scale. Science Foundation (NSF) grant, IIS-1162114. CONCLUSION REFERENCES As a prominent and powerful UGC system, Wikipedia is a 1. Nate Anderson. (2007, March). Conservapedia hopes source of information propagated throughout numerous to “fix” Wikipedia’s “liberal bias”. Ars Technica. other sites and systems. Take, for example, Helix, a plug-in Retrieved from designed to help writers with research as they write by http://arstechnica.com/uncategorized/2007/03/conserva suggesting information from various online sources pedia-hopes-to-fix-wikipedias-liberal-bias/ including Wikipedia [13]. Or consider that the ACM Digital 2. Annual estimates of the resident population for selected Library now uses IBM Watson to pull “Concepts in this age groups by sex for the United States, states, article” from Wikipedia’s content. Consequently, the counties, and Puerto Rico commonwealth and representativeness of Wikipedia’s content has become municipios: April 1, 2010 to July 1, 2014 Population increasingly important. Estimates. (n.d.). American FactFinder. Retrieved from The significant contribution of this paper, then, is the http://factfinder.census.gov/faces/tableservices/jsf/page description of an exogenous methodical approach to begin s/productview.xhtml?src=CF to unpack the representativeness of a UGC corpus by 3. Judd Antin, Raymond Yee, Coye Cheshire, and Oded relying on an independently generated set of terms Nov. (2011). Gender differences in Wikipedia editing. reflecting topical content aimed at specific demographics In Proceedings of the 7th International Symposium on with proven commercial viability. This method generalizes Wikis and Open Collaboration (WikiSym '11). ACM, to characterize representativeness of the content in a UGC New York, NY, USA, 11-14. system for other populations with which a researcher could 4. Patti Bao, Brent Hecht, Samuel Carton, Mahmood associate identifiable, cataloged content providers. For Quaderi, Michael Horn, and Darren Gergle. (2012). example, one can see how this method could be used to Omnipedia: bridging the Wikipedia language gap. In identify representativeness of topical content aimed at Proceedings of the SIGCHI Conference on Human children’s interests, or those with interests in fitness and Factors in Computing Systems. ACM, New York, NY, wellness, sports, religion, cooking, and others as these USA, 1075-1084. topical areas have multiple external content providers that are abstracted by content metadata service providers. More 5. Christie Barakat. (2014, May). Quora’s misogyny difficult to test, of course, is the kind of content socially problem. Social Times. Retrieved from marginalized in UGC systems like Wikipedia or by content http://www.adweek.com/socialtimes/quora-misogyny- providers in general, or content of interest to groups that are problem/149508?red=st so small (relative to the population) that commercial 6. Katherine Bischoping. (1993). Gender differences in providers do not exist. conversation topics, 1922–1990. Sex Roles 28(1), 1-18. 7. Torie Bosch.. (2012, July). Kate Middleton’s wedding 18. Alexander Halavais and Derek Lackaff. (2008). An gown and Wikipedia’s gender gap. Slate. Retrieved analysis of topical coverage of Wikipedia. JCMC 13, from 429-440. http://www.slate.com/blogs/future_tense/2012/07/13/k 19. Susan G. Harding. (1986). The science question in ate_middleton_s_wedding_gown_and_wikipedia_s_ge feminism. Cornell University Press. nder_gap_.html 20. Libby Hemphill and Jahna Otterbacher. (2012). 8. Adam R. Brown. (2011). Wikipedia as a data source Learning the lingo?: Gender, prestige and linguistic for political scientists: Accuracy and completeness of adaptation in review communities. In Proceedings of coverage. PS: Political Science & Politics 44(02), 339- the ACM 2012 Conference on Computer Supported 343. Cooperative Work (CSCW '12). ACM, New York, NY, 9. Judith Butler. (1990). Gender trouble: Feminism and USA, 305-314. the subversion of identity. New York: Routledge. 21. Libby Hemphill, Jahna Otterbacher, and Matthew 10. Shou Chang, Vikas Kumar, Eric Gilbert, and Loren Shapiro. (2013). What's Congress doing on Twitter?. In Terveen, L. (2014). Specialization, homophily, and Proceedings of the ACM 2013 Conference on gender in a social curation site: Findings from Computer Supported Cooperative Work (CSCW '13). Pinterest. In Proceedings of the 17th ACM Conference ACM, New York, NY, USA, 877-886. on Computer Supported Cooperative Work & Social 22. Libby Hemphill and Andrew J. Roback. (2014). Tweet Computing (CSCW '14). ACM, New York, NY, USA, acts: How constituents lobby Congress via Twitter. In 674-686. Proceedings of the ACM 2014 Conference on 11. Sapna Cheryan, Jessica Schwartz Cameron, Zach Computer Supported Cooperative Work (CSCW '14). Katagiri, and Benoît Monin. (2015). Manning up: ACM, New York, NY, USA, 1200-1210. Threatened men compensate by disavowing feminine 23. Benjamin Mako Hill and Aaron Shaw. (2013). The preferences and embracing masculine attributes. Social Wikipedia gender gap revisited: Characterizing survey Psychology. response bias with propensity score estimation. PloS 12. Benjamin Collier and Julia Bear. (2012.) Conflict, one 8(6), e65782. Chicago. criticism, or confidence: An empirical examination of 24. Benjamin Mako Hill and Aaron Shaw. (2014). the gender gap in Wikipedia contributions. In Consider the redirect: A missing dimension of Proceedings of the ACM 2012 Conference on Wikipedia research. In Proceedings of The Computer Supported Cooperative Work (CSCW '12). International Symposium on Open Collaboration. ACM, New York, NY, USA, 383-392. ACM, New York, NY, USA, 28. 13. Kate Conger. (2016, May). Helix conducts research as 25. Todd Holloway,, Miran Bozicevic, and Katy Börner. you write. TechCrunch. Retrieved from (2005). Analyzing and visualizing the semantic http://techcrunch.com/2016/05/08/helix-conducts- coverage of Wikipedia and its authors. ArXiv research-as-you-write/ Computer Science e-prints, cs/0512085. 14. Conservapedia: About. (n.d.). Retrieved from 26. Kim Holmberg and Iina Hellsten.. (2014). Analyzing http://www.conservapedia.com/Conservapedia:About the climate change debate on Twitter: Content and 15. Andrea Forte, Judd Antin, Shaowen Bardzell, Leigh differences between genders. In Proceedings of the Honeywell, John Riedl, and Sarah Stierch. (2012). 2014 ACM Conference on Web Science (WebSci '14). Some of all human knowledge: Gender and ACM, New York, NY, USA, 287-288. participation in peer production. In Proceedings of the 27. Joshua L. Kalla and Peter M. Aronow. (2015). ACM 2012 Conference on Computer Supported Editorial bias in crowd-sourced political information. Cooperative Work Companion (CSCW '12). ACM, PloS one 10(9), e0136327. New York, NY, USA, 33-36. 28. Matthew Kay, Cynthia Matuszek, and Sean A. 16. Eduardo Graells-Garrido, Mounia Lalmas, and Filippo Munson. (2015). Unequal representation and gender Menczer. (2015). First women, second sex: Gender stereotypes in image search results for occupations. In bias in Wikipedia. In Proceedings of the 26th ACM Proceedings of the 33rd Annual ACM Conference on Conference on Hypertext & Social Media (HT '15). Human Factors in Computing Systems (CHI '15). ACM, New York, NY, USA, 165-174. ACM, New York, NY, USA, 3819-3828. 17. Shane Greenstein and Feng Zhu. (2012). Is Wikipedia 29. Aniket Kittur, Ed H. Chi, and Bongwon Suh. (2009). biased?. The American Economic Review 102(3), 343- What's in Wikipedia?: Mapping topics and conflict 348. using socially annotated category structure. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '09). ACM, New 39. Amanda Menking and Ingrid Erickson. (2015). The York, NY, USA, 1509-1512. heart work of Wikipedia: Gendered, emotional labor in 30. Max Klein. (2015). Wikipedia in the world of global the world's largest online encyclopedia. In Proceedings gender inequality indices: What the biography gender of the 33rd Annual ACM Conference on Human gap is measuring. In Proceedings of the 11th Factors in Computing Systems (CHI '15). ACM, New International Symposium on Open Collaboration. York, NY, USA, 207-210. ACM, New York, NY, USA, 16. 40. Henry T. Moore. (1922). Further data concerning sex 31. Jona Kräenbring, Tika Monzon Penza, Joanna differences. Journal of Abnormal Psychology 17, 210- Gutmann, Susanne Muehlich, Oliver Zolk, Leszek 214. Wojnowski, Renke Maas, Stefan Engelhardt, and 41. Emma Pierson. (2015). Outnumbered but well-spoken: Antonio Sarikas. (2014). Accuracy and completeness Female commenters in the New York Times. In of drug information in Wikipedia: A comparison with Proceedings of the 18th ACM Conference on Computer standard textbooks of pharmacology. PloS one 9(9), Supported Cooperative Work & Social Computing e106930. (CSCW '15). ACM, New York, NY, USA, 1201-1213. 32. Shyong (Tony) K. Lam, Anuradha Uduwage, Zhenhua 42. Ioannis Protonotarios, Vasiliki Sarimpei, and Jahna Dong, Shilad Sen, David R. Musicant, Loren Terveen, Otterbacher. (2016). Similar gaps, different origins?: and John Riedl. (2011). WP:Clubhouse?: An Women readers and editors at Greek Wikipedia. In exploration of Wikipedia's gender imbalance. In Tenth International AAAI Conference on Web and Proceedings of the 7th International Symposium on Social Media. Wikis and Open Collaboration (WikiSym '11). ACM, 43. Joseph Reagle and Lauren Rhue. (2011). Gender bias New York, NY, USA, 1-10. in Wikipedia and Britannica. International Journal of 33. David Laniado, Andreas Kaltenbrunner, Carlos Communication 5, 21. Castillo, and Mayo Fuster Morell. (2012). Emotions 44. Research gender affinity for different subjects on and dialogue in a peer-production community: The Wikipedia. (n.d.). Retrieved from case of Wikipedia. In Proceedings of the Eighth https://meta.wikimedia.org/wiki/Grants:IdeaLab/Resea Annual International Symposium on Wikis and Open rch_gender_affinity_for_different_subjects_on_Wikipe Collaboration (WikiSym '12). ACM, New York, NY, dia USA. 45. Lydia Saad. (2015, January). U.S. Liberals at record 34. Issie Lapowsky. (2016, May). Of course Facebook is 24%, but still trail Conservatives. Gallup. Retrieved bias: That’s how tech works today. Wired. Retrieved from http://www.gallup.com/poll/180452/liberals- http://www.wired.com/2016/05/course-facebook- record-trail-conservatives.aspx biased-thats-tech-works-today/ 46. Monika Sengul-Jones (2016). Final report. Retrieved 35. Mike Ma. (2016, January). Wikipedia edit war over from Google-featured activist’s praise for Bin Laden. https://meta.wikimedia.org/wiki/Grants:IEG/Full_Circl Breitbart. Retrieved from e_Gap_Protocol:_Addressing_the_‘Unknown_Unknow http://www.breitbart.com/tech/2016/05/20/wikipedia- ns’/Final editors-scrub-references-activists-bin-laden-praise- following-breitbart-article/ 47. Monika Sengul-Jones. (2016). The “bestiary of gaps” on Wikipedia. Gap finding project. Retrieved from 36. J. Nathan Matias, Sophie Diehl, and Ethan Zuckerman. https://gapfindingproject.wordpress.com/2015/10/12/th (2015). Passing on: Reader-sourcing gender diversity e-bestiary-of-gaps-on-wikipedia/ in Wikipedia. In Proceedings of the 33rd Annual ACM Conference Extended Abstracts on Human Factors in 48. Matthew Sheffield. (2008, August). Conservatives miss Computing Systems. ACM, New York, NY, USA, Wikipedia’s threat. The Washington Times. Retrieved 1073-1078. from http://www.washingtontimes.com/news/2008/aug/21/c 37. Cynthia McKelvey. (2016, February). Political activists onservatives-miss-wikipedias-threat/?page=all claim Twitter censored #WhichHillary hashtag. The Daily Dot. Retrieved from 49. Eran Shor, Arnout van de Rijt, Alex Miltsov, Vivek http://www.dailydot.com/politics/which-hillary- Kulkarni, and Steven Skiena. (2015). A Paper ceiling: hashtag-twitter-censorship/ Explaining the persistent underrepresentation of women in printed news. American Sociological Review 38. Sudip Mittal, Neha Gupta, Prateek Dewan, and 80(5), 960-984. Ponnurangam Kumaraguru. (2014). Pinned it!: A large scale study of the Pinterest network. In Proceedings of 50. Survey potential female editors to determine most the 1st IKDD Conference on Data Sciences (CoDS popular topics of interest. (n.d.). Retrieved from '14). ACM, New York, NY, USA. https://meta.wikimedia.org/wiki/Grants:IdeaLab/Surve y_potential_female_editors_to_determine_most_popul Proceedings of the SIGCHI Conference on Human ar_topics_of_interest Factors in Computing Systems (CHI '13). ACM, New 51. Maja Van der Velden. (2013). Decentering design: York, NY, USA, 31-34. Wikipedia and indigenous knowledge. International 55. Wikipedia: Notability. (n.d.). Retrieved from Journal of Human-Computer Interaction 29:4. https://en.wikipedia.org/wiki/Wikipedia:Notability 52. Claudia Wagner, David Garcia, Mohsen Jadidi, and 56. Wikipedia: Notability (people). (n.d.). Retrieved from Markus Strohmaier. (2015). It's a man's Wikipedia?: https://en.wikipedia.org/wiki/Wikipedia:Notability_(pe Assessing gender inequality in an online encyclopedia. ople) In Proceedings of the International AAAI Conference 57. Jake Ryan Williams, Eric M. Clark, James P. Bagrow, on Web and Social Media (ICWSM2015). Christopher M. Danforth, and Peter Sheridan Dodds. 53. Claudia Wagner, Eduardo Graells-Garrido, David (2015). Identifying missing dictionary entries with Garcia, and Filippo Menczer. (2016). Women through frequency-conserving context models. Physical Review the glass-ceiling: Gender asymmetries in Wikipedia. E 92(4), 042808. arXiv preprint arXiv:1601.04890. 58. Samuel Wooley and Phil Howard. (2016, May). Bots 54. Yi-Chia Wang, Moira Burke, and Robert E. Kraut. unite to automate the presidential election. Wired. (2013). Gender, topic, and audience response: An Retrieved from analysis of user generated content on Facebook. In http://www.wired.com/2016/05/twitterbots-2/