Revisiting 'The Rise and Decline' in a Population of Peer Production Projects [769 Wikis]

CHI 2018 Paper CHI 2018, April 21–26, 2018, Montréal, QC, Canada Revisiting “The Rise and Decline” in a Population of Peer Production Projects Nathan TeBlunthuis Aaron Shaw Benjamin Mako Hill University of Washington Northwestern University University of Washington Seattle, Washington, USA Evanston, IL, USA Seattle, Washington, USA [email protected] [email protected] [email protected] ABSTRACT Do patterns of growth and stabilization found in large peer 2.0 ● ●● ● ● production systems such as Wikipedia occur in other commu- ● ● ● ● ● ● ● ● ● ●● nities? This study assesses the generalizability of Halfaker et ● ● ●●●● ● ● ●● ● ● al.’s influential 2013 paper on “The Rise and Decline of an ● ●● ● ● ●● ● ● ● ●● ● ●● ●● ● Open Collaboration System.” We replicate its tests of several ● ● ● 1.5 ● ● theories related to newcomer retention and norm entrenchment ● ● using a dataset of hundreds of active peer production wikis ●● ● ● from Wikia. We reproduce the subset of the findings from ● ● ● ●● Halfaker and colleagues that we are able to test, comparing ●●● both the estimated signs and magnitudes of our models. Our units) editors (Std dev Active 1.0 results support the external validity of Halfaker et al.’s claims Year 0 Year 1 Year 2 Year 3 Year 4 Year 5 that quality control systems may limit the growth of peer pro- Wiki age duction communities by deterring new contributors and that norms tend to become entrenched over time. Figure 1. Mean of the number of editors with at least 5 edits per month in standard deviation units for wikis in our sample. The dashed lines represent the results of a LOESS regression. The error bars represent ACM Classification Keywords bootstrap 95% confidence intervals. This replicates Figure 2 in RAD. H.5.3. Information Interfaces and Presentat ion (e.g. HCI): Group and Organization Interfaces – Computer-supported co- operative work This paper replicates analysis from Halfaker et al.’s “The Rise and Decline of an Open Collaboration System” [11] (which we abbreviate “RAD”) in a sample of 740 active wikis hosted Author Keywords on Wikia.1 RAD makes one of the most influential and highly governance; peer production; online communities; quality cited claims about peer production dynamics, attributing En- control; retention; replication; Wikipedia; wikis glish Wikipedia’s decline in contributors since 2007 to entrenchment (RAD uses the term “calcification”) within the INTRODUCTION community as norms and policies become difficult to change, “Peer production” describes a way of organizing collaborative especially for newer users. Our results reproduce most of information production in online commons [2]. Over the RAD’s findings. Like RAD, we find that the average com- last decade, peer production has become a central object of munity in our dataset experiences a “rise and decline,” that HCI research. However, the vast majority of peer production newcomers are less likely to survive over time, that rejected research has studied a small number of the largest communities newcomers are less likely to survive, that editors with longer [3, 6]. An enormous portion of empirical studies of peer tenure have more influence over norms, and that norms be- production in HCI are of the English-language version of come entrenched as wikis age. In addition to providing an Wikipedia. Unfortunately, HCI’s historical focus on novelty external validation of RAD’s findings, we rule out alternative has meant that tests of the applicability of findings shown in explanations of RAD’s results that emphasize unique attributes one setting to other contexts rarely happens [27]. As a result, of Wikipedia or the timing of the editor decline in that com- we know little about the degree to which theory and design munity. claims from studies of Wikipedia apply more broadly. BACKGROUND Entrenchment in Wikipedia and Peer Production Permission to make digital or hard copies of part or all of this work for personal or Active peer production communities often experience a period classroom use is granted without fee provided that copies are not made or distributed of rapid growth followed by stabilization [20, 23]. Following for profit or commercial advantage and that copies bear this notice and the full cita- tion on the first page. Copyrights for third-party components of this work must be 1Wikia is a wiki hosting platform where anyone can start a wiki. honored. For all other uses, contact the owner/author(s). Copyright is held by the au- In 2016, Wikia partially rebranded as “Fandom” to emphasize sup- thor/owner(s). port for fan communities. See: https://www.wikia.com/ (https: CHI 2018, April 21–26, 2018, Montréal, QC, Canada. //perma.cc/TL79-VB57 ACM ISBN 978-1-4503-5620-6/18/04. ). https://doi.org/10.1145/3173574.3173929 Paper 355 Page 1 CHI 2018 Paper CHI 2018, April 21–26, 2018, Montréal, QC, Canada this pattern, the number of contributors to English Wikipedia of crowdsourced fund-raising [1], social media [5], and online grew exponentially until March 2007, when it began to decline collaborative mapping [21]. [26]. Although early accounts of peer production argued that projects such as Wikipedia and the Linux kernel mobilized This paper assesses the replicability and external validity of massive collaboration without the sorts of formal hierarchies RAD to provide an empirical foundation for such generaliza- or bureaucracies used in formal organizations [2, 17], organi- tion. In doing so, we also evaluate several alternative explana- zational research has argued that the formalization of rules, tions that RAD could not rule out. In particular, the simulta- norms, and routines accompany this trajectory in many types neous decline and entrenchment RAD observes in Wikipedia of organizations [13, 18, 24]. Drawing from this work, early could be driven by external factors related to time, such as the explanations for Wikipedia’s decline included bureaucratic rise of other online communities (e.g., Facebook) that might compete for newcomers. By studying a population of commu- overhead and increasing resistance to contributions from less nities whose trajectories start at different points in time, we active editors [26, 7]. During this same period, algorithmic can model wiki age accounting for calendar time in ways RAD tools such as “bots” became important parts of Wikipedia’s could not. By studying many communities, we can also better quality control systems [10], and the proportion of edits that were rejected increased [12]. Building on this prior work, Hal- understand the scope of RAD’s generalizability by measuring faker et al.’s “The Rise and Decline of an Open Collaboration variation between wikis. System” [11] found evidence in support of the theory that three METHODS elements of Wikipedia’s quality control system—newcomer We attempt to follow RAD’s measures and methods to the rejection, algorithmic tools, and norm entrenchment—could fullest extent possible. In some places, we are forced to explain the transition from growth to decline. However, de- make changes to accommodate differences between English spite the impact and influence of RAD’s explanation, it has Wikipedia and Wikia and the fact that our analysis includes not been replicated beyond Wikipedia until now. multiple wikis. To describe our methodology, we briefly sum- marize RAD’s techniques and note several ways that we di- Replication in Social Computing Research verge. Additional detail on operationalization is provided in Although HCI research prizes novelty and provocation, it also RAD. We also provide access to the complete R source code seeks to build scientifically rigorous, replicable, and generaliz- that we used to complete our analysis in the supplementary able knowledge [27]. Replicability refers to how well results material that accompanies this paper. In the description that hold up when other researchers follow reported procedures. follows, variable names are italicized. Hornæk et al. define replication studies as attempts “to confirm, The RAD authors present three interdependent analyses. The expand, or generalize an earlier study’s findings” [15]. Gen- first tests whether the rejection of edits made by newcomers eralizability (external validity) refers to the degree to which causes decreased newcomer retention which in turn leads to a results hold up across different populations [4]. Replication decline in the number of active editors. To support this claim, studies thus assess whether details of context or methodologi- the RAD authors use data from English Wikipedia to plot three cal choice explain results. trends: the number of active contributors, the rate of newcomer Although comparative analysis of peer production communi- survival, and the rate of newcomer rejection. The first plot ties has emerged as an important means to understand the life shows the rise and decline in active contributors (i.e., individu- cycles and dynamics of social computing systems [20, 22, 25], als who make at least 5 edits in a given month). The second there have been few efforts to establish whether findings from plot shows that the proportion of good-faith newcomers who Wikipedia and other large communities replicate or generalize “survive” falls over time. The third plot shows that the pro- [3, 14]. In one important exception, Kittur and Kraut [16] ex- portion of good-faith newcomers “rejected” in their first edit amine the prevalence of social mechanisms related to conflict session rises over time. RAD considers a newcomer to have and coordination in Wikipedia among nearly 7,000 wikis from survived if the newcomer edits during the period between 60 Wikia. Their work found both similarities and differences days and 180 days after their first edit session (i.e., sequence between these communities and Wikipedia. of consecutive edits less than one hour apart) and to have been rejected if a change the newcomer makes to an article in their first edit session is undone. Using data from all newcomers Replicating RAD drawn from a set of Wikia wikis, we replicate these plots in Do the relationships described in RAD generalize to other our Study 1.

Load more