A Quantitative Portrait of Wikipedia's High-Tempo Collaborations During

TBD

A Quantitative Portrait of Wikipedia’s High-Tempo Collaborations during the 2020 Coronavirus Pandemic

BRIAN C. KEEGAN, Department of Information Science, University of Colorado Boulder, United States CHENHAO TAN, Department of Computer Science, University of Colorado Boulder, United States The 2020 coronavirus pandemic was a historic social disruption with significant consequences felt around the globe. Wikipedia is a freely-available, peer-produced encyclopedia with a remarkable ability to create and revise content following current events. Using 973,940 revisions from 134,337 editors to 4,238 articles, this study examines the dynamics of the English Wikipedia’s response to the coronavirus pandemic through the first five months of 2020 as a “quantitative portrait” describing the emergent collaborative behavior atthree levels of analysis: article revision, editor contributions, and network dynamics. Across multiple data sources, quantitative methods, and levels of analysis, we find four consistent themes characterizing Wikipedia’s unique large-scale, high-tempo, and temporary online collaborations: external events as drivers of activity, spillovers of activity, complex patterns of editor engagement, and the shadows of the future. In light of increasing concerns about online social platforms’ abilities to govern the conduct and content of their users, we identify implications from Wikipedia’s coronavirus collaborations for improving the resilience of socio-technical systems during a crisis. Additional Key Words and Phrases: COVID-19; online collaboration; crisis informatics; public health ACM Reference Format: Brian C. Keegan and Chenhao Tan. 2020. A Quantitative Portrait of Wikipedia’s High-Tempo Collaborations during the 2020 Coronavirus Pandemic. Proc. ACM Hum.-Comput. Interact. 3, CSCW, Article TBD (Novem- ber 2020), 36 pages. https://doi.org/0000001.0000001

INTRODUCTION The coronavirus disease 2019 (COVID-19) pandemic was a historic social disruption with significant health, economic, political, and cultural consequences. Wikipedia is the “free encyclopedia that anyone can edit” with more than 6 million articles in English. Because Wikipedia’s editors have a well-established and remarkable ability to create and revise encyclopedic content about current events [80, 81], how did Wikipedia cover the pandemic? We offer a “quantitative portrait” characterizing Wikipedia’s collaborations around topics related to the coronavirus pandemic focusing on article-level activity, user-level engagement, and network-level dynamics. The primary contribution of this paper is to provide theoretically-grounded descriptive insights arXiv:2006.08899v1 [cs.SI] 16 Jun 2020 into how volunteer editors working with twenty year-old interfaces and affordances are able to produce timely content about a historic global event. A deadly pandemic causing unprecedented social disruption around the globe seemingly presents a different scale of challenges to manage

Authors’ addresses: Brian C. Keegan, Department of Information Science, University of Colorado Boulder, INFO 129, 1045 18th St., Boulder, Colorado, 80309, United States, [email protected]; Chenhao Tan, Department of Computer Science, University of Colorado Boulder, ECES 118A, 2055 Regent Drive, Boulder, Colorado, 80305, United States, brian. [email protected].

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice andthe full citation on the first page. Copyrights for components of this work owned by others than the author(s) must behonored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]. © 2020 Copyright held by the owner/author(s). Publication rights licensed to ACM. 2573-0142/2020/11-ARTTBD $15.00 https://doi.org/0000001.0000001

Proc. ACM Hum.-Comput. Interact., Vol. 3, No. CSCW, Article TBD. Publication date: November 2020. TBD:2 Brian C. Keegan & Chenhao Tan the coupling of diverse humans and distributed content artifacts in terms of the scope of content demanding editorial attention, the volatility and duration of information to process, and meeting shifting demands for information. This study examines the dynamics of the English Wikipedia’s response to the coronavirus pandemic between December 2019 through May 2020. Using publicly-available digital trace data like revision history logs and archived article content for 973,940 revisions by 134,337 editors to 4,238 articles related to the pandemic and its effects, we use mixed quantitative methods and inductive interpretations of these descriptive results to provide overlapping evidence around the structure and dynamics of an on-going and large-scale collaboration. We structure our analysis by focusing on analysis at the level of articles, editors, and networks. First, we analyze the changes in daily revision activity, similarity of article content, and pageview activity. Second, we examine the patterns of editors’ engagement in these collaborations, the timing of their contributions, and the effects of their participation on their previous contributions. Finally, we use a social network perspective to understand how the structure of the collaboration network linking editors and articles together changes over time as well as how the structure of the hyperlink network of articles linking to each other has evolved. Drawing on perspectives from computer-supported cooperative work, organizational behavior, and crisis and public health informatics, we synthesize these descriptive quantitative findings through an inductive process to find four consistent themes around (1) external events as drivers of activity, (2) spillovers of activity from core to peripheral articles, (3) complex patterns of user engagement, and (4) the shadows of the future. These findings—while primarily descriptive and inductive in nature—characterize Wikipedia’s unique large-scale, high-tempo, and temporary online collaborations with potential implications for designing more resilient socio-technical infrastructure. In the face of increasing concerns about online social platforms’ ability to moderate incendiary content, intervene against disinformation campaigns, and limit mechanisms that drive polarization, Wikipedia’s responses to current events serve as an important counterfactual about the pro-social, scalable, and resilient possibilities of online communication during a crisis.

BACKGROUND Wikipedia is a free online encyclopedia using a peer production model of volunteers authoring content as well as managing the community [73, 120]. The production and consumption of information on Wikipedia is a matter of theoretical, empirical, and popular interest given its prominence as one of the most visited websites while also being a volunteer-led online social platform. Wikipedia’s editors, their managerial systems, and their reliance on technological tools makes it a canonical example of a socio-technical system [104]. Since its earliest days, Wikipedia’s online community of editors have been able to use this platform to rapidly create and update encyclopedic content in the aftermath of accidents, disasters, and other crises. The communities of Wikipedia volunteers who rapidly and temporarily self-organize to author and edit content about breaking news and current events are examples of high-tempo collaborations. This section reviews prior work on coordination, crisis informatics and digital epidemiology, and Wikipedia’s high-tempo collaborations to motivate our analysis of Wikipedia’s coverage of the 2020 coronavirus pandemic.

High-tempo, high-reliablity, and temporary coordination Coordination is the management of dependencies between activities to accomplish tasks [95, 106]. Coordination is a particularly important construct for understanding the processes of sharing, transfer, accumulation, transformation, and co-creation in knowledge collaboration [40]. Classic approaches to coordination emphasize task uncertainty, task interdependence, and work unit size as crucial determinants of impersonal (rule and schedules), personal (horizontal or vertical

Proc. ACM Hum.-Comput. Interact., Vol. 3, No. CSCW, Article TBD. Publication date: November 2020. A Quantitative Portrait of Wikipedia’s High-Tempo Collaborations during the 2020 Coronavirus Pandemic TBD:3 communication), or group (meetings) coordination modes [143]. In contrast to these explicit coordination mechanisms, teams can also employ implicit coordination mechanisms that rely on anticipation [149], heedfulness [146], shared mental models [24], and other informal strategies [122]. But many collaborative contexts—particularly online—lack formal leadership, defined roles, and predictable work units for explicit coordination and the social information like identity, familiarity, and expectations required for implicit coordination. The teams typically studied by organizational researchers rarely face the unpredictability, urgent demands, and immediate consequences as groups in high-tempo contexts like disaster response [94], emergency medicine [43], carrier flight decks [146], SWAT teams [13], and journalism [15]. Tempo- rary organizations are where people work together on complex, non-routine, consequential, and high-risk tasks and disband after a deadline or resources are expended [6, 7, 23]. High-reliability organizations operate in technologically complex settings like nuclear reactors, aircraft cockpits, and trading desk where errors can quickly cascade into into catastrophic consequences so there is an overwhelming focus on safety [16, 32, 147]. These frameworks all characterize distinct types of organizations with varying degrees of overlap: temporary organizations do not need to be high-tempo or high-reliability (a board game night), high-tempo organizations do not need to be temporary (a carrier flight deck), high-reliability organizations do not need to be high-tempo (a nuclear reactor control room), and temporary organizations can still be high-reliability (an emergency room). Wikipedia’s breaking news collaborations can be characterized as high-tempo because of the rapid pace of editing activity from dozens or hundreds of users in response to new information about an event, high-reliability because contributions are immediately live to an audience of thousands of information-seeking users, and temporary because eventually information about an event saturates and editors’ interests shift to other topics. However, many organizations sharing these characteristics employ similar coordination strategies: improvisation [11], regeneration [12], jamming [36], representation [84], delegation [85], heedful interrelating [146], swift trust [98], collective responsibility [142], emergent roles [5], reciprocal exchange [41], sensitivity to operations [145], mindfulness [134], privileging expertise [42], rendering work visible [84], vigilant interactions [72], interdependency awareness [14], folding [126], and knowledge shaping [151]. It is beyond the scope of this study to summarize or synthesize the specifics of all these mechanisms, but to instead emphasize that many kinds of high-tempo, high-reliability, and/or temporary organizations have developed mechanisms for coordinating under stress that could be implemented, adapted, or re-discovered by Wikipedia editors to support their breaking news collaborations. Some Wikipedia-specific mechanisms are summarized in a later section of this background.

Crisis informatics and digital epidemiology The emergence of self-organizing altruistic communities engaged in recovery work in the aftermath of crises is one of the most robust findings in disaster sociology35 [ , 118, 133]. With the proliferation of wireless, mobile, and social technologies, people use digital communication to respond to disaster warnings, responses, and recovery [109]. The field of crisis informatics has primarily focused on the use of social media to understand the dynamics of emergent communities during disasters [111, 128], but other information technologies like SMS messages [110], chatrooms [105], and geographic information systems [131] are also active sites of technologically-mediated post- disaster engagement. The “improvisation of order out of chaos” alongside collective feelings of equanimity and egalitarianism [132], while temporary, align well with many of Wikipedia’s core values like neutrality and openness [73, 120]. However, Wikipedia is generally not a primary data source for crisis informatics researchers or emergency managers because of the lack of geo-location and lags in reporting pageview data.

Proc. ACM Hum.-Comput. Interact., Vol. 3, No. CSCW, Article TBD. Publication date: November 2020. TBD:4 Brian C. Keegan & Chenhao Tan

Fig. 1. An example of the “In the News” (ITN) template on Wikipedia’s homepage featuring articles related to current events. During the pandemic, the ITN template had a dedicated “special header” devoted to COVID-19 related topics (at top) like the pandemic, disease, virus, testing, local responses, timeline, and deaths.

Digital epidemiology and public health informatics use digital trace data like search engine query logs and social media posts to augment and accelerate classic methods for surveilling for diseases [3, 56, 123, 124, 127]. Wikipedia is a prominent online resource for health information with adequate quality information for consumers [66, 125, 130]. Wikipedia’s WikiProject Medicine has distinguished itself through concerted efforts within the community and outreach to medical professions and researchers to expanding and improving the quality of health information [65, 75, 119]. Because Wikipedia consistently ranks in the first ten results of many health queries [90] and it publicly shares access log data in the form of pageviews, a growing body of literature explores Wikipedia’s potential as an epidemiological data source. Specifically, pageviews to Wikipedia articles about diseases reliably burst during an outbreak as people seek information about the disease in response to media coverage [1, 54, 125, 137]. While some studies emphasize the feasibility of using Wikipedia data to forecast influenza10 [ , 55, 68, 96, 137, 153], dengue [67], zika [140], and other diseases [39, 135], other research concludes behavioral heterogeneity on social platforms like Wikipedia cannot reliably beat canonical epidemiological forecasting methods [3, 91, 108, 116]. We emphasize the goal of our study is not to evaluate the effectiveness of Wikipedia data in supporting emergency management or forecasting coronavirus incidence. Instead, we expect that coronavirus-related activity on Wikipedia will mirror previous disease outbreaks in terms of spikes of information seeking and contributions from editors of medical topics. Preliminary work has also examined shifts in collective attention on Wikipedia as a result of quarantining efforts [121] and citation practices on COVID-19 related articles [29]. Because the effects of the pandemic are greater than any outbreak since Wikipedia’s founding in 2001 [61, 117], we expect the scale of information production and seeking will demand new forms of collaboration and coordination that have not been seen before.

High-tempo collaborations on Wikipedia Despite early apprehensions about its radical “anyone can edit” model polluting the web with misinformation [33, 97], Wikipedia’s responsiveness to current events is a major factor in its identity, rules, and history [81]. The creation and revision of Wikipedia content in the aftermath of major events is not done automatically or through formal delegation, but is accomplished by high-tempo collaborations in which large numbers of volunteers—most of whom have never worked together before—self-organize into intense and temporary collaborations using twenty-year-old

Proc. ACM Hum.-Comput. Interact., Vol. 3, No. CSCW, Article TBD. Publication date: November 2020. A Quantitative Portrait of Wikipedia’s High-Tempo Collaborations during the 2020 Coronavirus Pandemic TBD:5 interfaces and affordances. These collaborations are an important boundary case that challenges our assumptions and theories about whether online communities can succeed [22, 86] in the face of unstable membership, asynchronous interaction, diverse motivations, unclear leadership, and volatile information. But the quality of the work done in these collaborations is reliably excellent: Wikipedia articles about current events have been cited as exemplars of timeliness, breath, and accu- racy around events like U.S. presidential elections [19, 28], school shootings [27], earthquakes [103], and other events [31, 47]. The risks and opportunities associated with Wikipedia’s coverage of current events have likewise been noted by countries banning access [63, 129], politicians polishing their articles [37, 57], and corporations spinning controversies [38, 59]. Like other forms of high-reliability organizing, overlapping practices and resources enable Wiki- pedia editors to engage in these unusual collaborations. Editors delegate labor-intensive tasks using bots and templates [48, 52, 78], regenerate structures from previous crises [80], alter their interaction practices [79], adopt emergent social roles based on their previous experience [77], develop routines to handle unpredictable but regular events [83], migrate between articles to ensure their consistency [141], respond to media coverage and agenda setting [82], and adding culturally-salient media [115]. Wikipedia is a site of collective memory formation for major historical events like the Vietnam War [93], September 11 attacks [46], Arab Spring [44, 45], and deaths of celebrities [83]. Because Wikipedia preserves the content of every editor’s revision to its articles, it is possible to mine these histories to observe the evolution of framings and interpretations around individuals, institutions, and events [50, 76, 112, 113]. Collective memory repertoires like re-appraising historical events during a contemporary event can be significant drivers of attention and information demand [141]. The evolution of Wikipedia’s content around the coronavirus will be of immense interest to future scholars to explore how our understanding of the virus and its implications changed or how Wikipedians’ discussions reflected larger debates [62].

APPROACH In this section, we discuss the details of our data collection and sampling procedures.

Sampling Articles. The English Wikipedia has standalone articles for the event (“COVID-19 pandemic”), the disease (“Coronavirus disease 2019”), and the virus (“Severe acute respiratory syndrome coronovirus 2”). We focus on the event article—“COVID-19 pandemic”— as the seed article for subsequent sampling steps because it was the largest, longest, most active, and more viewed of the three primary articles. We retrieved related articles through an iterative snowball sampling approach to identify a set of related articles. This iterative sampling methodology consisted of several steps. First, we extracted all the member and child pages of the primary category (i.e., “Category:COVID-19”), retrieving 3,069 pages in English. Examples of these category articles include “2019–20 coronavirus outbreak by country and territory”, “Severe acute respiratory syndrome coronavirus 2”, and “Tem- plate:2019–20 coronavirus outbreak data”. Second, we identified prior revisions of the pandemic event article at a weekly frequency going back to Sunday, 5 January 2020. For each weekly revision of the seed article, we retrieved the hyperlinks from that week’s version of the article to neighbor articles. We combined these weekly neighbor articles together into a super-set of neighboring articles that were ever linked to from the seed article over these revisions. The first two steps give us two distinct sets of similar articles: category-related articles and neighbor-related articles. The articles related by category (the first step), by definition, have extremely high topical similarity but also include administrative pages like templates. The articles related by hyperlink neighbors (the second step) are related to the event in some way, ranging from obvious relationships (e.g., “Wuhan”) to less obvious relationships (e.g., “Diabetes”). Combining both of

Proc. ACM Hum.-Comput. Interact., Vol. 3, No. CSCW, Article TBD. Publication date: November 2020. TBD:6 Brian C. Keegan & Chenhao Tan these sampling approaches enables us to balance precision (category-related articles) and recall (neighbor-related articles) for a set of articles likely impacted by the coronavirus pandemic. We use the union of both of these samples as the set of articles in the rest of the analysis: 717,776 revisions to 3,064 category-related articles and 256,174 revisions to 1,174 neighbor-related articles for a total of 973,940 revisions to 4,238 articles from 134,337 editors. We note this is not the only sampling approach possible to retrieve COVID-19 related articles: the Wikimedia Foundation publishes summary statistics of revisions and pageviews over a set of related pages derived from Wikidata relationships.1 Table2 shows example pages in each type of Wikipedia articles that we consider in this work. Editors. The third step in our sampling approach is to identify a relevant set of users contributing to these articles. Because user-level revision activity is highly right-skewed and distributed across thousands of articles with their own local editing microclimates, we instead prioritize retrieving the contribution histories for all the registered editors who contributed to the event, disease, and virus articles from December 2018 through May 2020.

Information retrieval For each article in the union of category-related and neighbor-related articles and the contributors, we retrieved data from December 2018 through May 2020. This sampling window captured a baseline of revision activity over an approximately 12-month period before the outbreak and pandemic as a baseline. We used a custom and openly-licensed Python library2 to retrieve data from the official Wikipedia API endpoints about revision histories, user contributions, revision content, and pageviews. Page revision histories. The revision histories of each of the 4,238 articles were retrieved and include the revision ID, the name or IP address of the user who committed the revision, the timestamp of the revision, the size of the page at the time, and a SHA-1 hash of the page content to aid in revert tracking. Secondary features like the size of the changes introduced by the revision (“diff”), the time elapsed since the previous revision (“lag”), and the time elapsed since the page was created (“age”) were also competed for each revision. Editor contribution histories. The contribution histories of a sample of users who contributed to articles was also retrieved. While this is only 3% of the users across all the articles, it still captures 178,550 revisions, or 20% of all observed revisions in our sample. These users’ contribution histories across all articles and other pages (in all namespaces) were retrieved from December 2018 through May 2020, generating 3.9 million unique revisions. Like the revision history data, these data include the revision ID, the timestamp, namespace of the page, and size of the revision. Secondary features like the size of the changes and time elapsed since the previous revision were also computed. Revision content. The content of individual historical revisions were retrieved from the API. The full HTML of these revisions were stored and custom scripts parsed them into structured data like lists of hyperlinks, external links, references, and plain text for analysis over time. These revisions were only sampled on a weekly frequency, retrieving the last revision of each article every Sunday. Pageviews. The number of times a page was served through a website or mobile device was retrieved from the official API. These pageview data are available at a daily frequency andare not disaggregated by geography, language, etc. in order to preserve privacy. We retrieve the “all-access user” data combining web and mobile views but excluding views from agents likely

1https://covid-data.wmflabs.org/ 2Link to library removed for anonymization.

Label Event(s) Date(s) A China reports first case 2 January 2020 B China begins quarantines, U.S. reports first cases 22–25 January 2020 C Diamond Princess quarantined in Japan 4 February 2020 D U.S. reports community transmission and first death 26-29 February 2020 E Italy quarantines; schools and pro-sports canceled; “Black Thursday” market crash 6–13 March 2020 F Statewide stay-at-home orders begin 16–23 March 2020 Table 1. U.S.-centered timeline of events

Article title Type Created Revisions Editors Pageviews Size COVID-19 pandemic Seed 2020-01-05 19,858 2,749 63,833,280 332,803 Coronavirus disease 2019 Seed 2020-02-05 4,255 881 14,486,983 230,522 SARS coronovirus 2 Seed 2020-01-09 2,919 646 7,374,993 79,997 Template:COVID-19 pandemic data Category 2020-01-28 24,083 1,001 269,827 171,627 2020 Democratic Party primaries Category — 8,628 1,417 9,494,957 274,742 COVID-19 pandemic in the U.S. Category 2020-02-17 7,730 1,330 12,921,675 382,169 2019–20 Hong Kong protests Neighbor 2019-06-10 7,383 1,156 432,714 443,948 Donald Trump Neighbor — 3,941 456 9,989,039 400,926 Boris Johnson Neighbor — 2,074 650 7,051,692 253,617 Table 2. Basic information for the seed, category-related, and neighbor-related English Wikipedia articles for activity between December 2018 and May 2020. We show three examples of category-related and neighbor- related articles. Some article titles have been abbreviated for space. Page creation dates for articles that were created before December 2018 are not reported.

to be spiders/bots. Pageview data is assigned to the article even if the article is a redirect [69], so the redirect graph for each article in our sample was constructed, the pageviews for each of those primary and redirected articles retrieved, and the pageview numbers for the focal and redirect articles were aggregated together.

ARTICLE-LEVEL ACTIVITY Using the articles as a primary unit of analysis, we examine how the activity related to revisions, pageviews, and content similarity changed for the three seed articles (the pandemic event, the disease, and the virus) as well as the larger sample of category-related and neighbor-related articles.

Revisions How did the number of revisions to articles change over the pandemic? Figure2 plots the number of daily revisions for the pandemic, disease, and virus articles. The activity on these three seed articles is uneven over time with notable “bursts” of activity involving hundreds of revisions in a single day occurring around major events. With so many events unfolding simultaneously around the globe coupled with lags in reporting and article editing, it is difficult to causally identify which specific external events drove the observed bursts of activity. The pandemic event article predates the other two seed articles and has spikes of activity over time windows that correspond with major external events (see Table1 with events annotated in Figure2). The article about the virus was created before the article about the disease, but revision activity on the disease article overtook the virus article. By mid-April, revision activity on all three of these seed articles had fallen below

Proc. ACM Hum.-Comput. Interact., Vol. 3, No. CSCW, Article TBD. Publication date: November 2020. TBD:8 Brian C. Keegan & Chenhao Tan

Fig. 2. Daily revision counts for the pandemic (blue), disease (orange), and virus (green) articles and mean Fig. 3. Summed daily revision counts for articles daily counts for category-related (red) and neighbor- related by category (red) and neighbor (purple) com- related articles (purple). Events from Table1 are pared to pandemic event article (blue). annotated by letter.

50 revisions per day, well below their peak activity: 861 revisions to the event article on January 25, 181 revisions to the disease article on March 12, and 123 revisions to the virus article on January 25, indicating the decline of updates and interest on these pages. The largest spikes of revision activity on the event and virus articles happen late January around the announcements of Chinese quarantines and the first U.S. cases, well before more significant global disruptions in March. A critical but unobserved confounder in these data is that page protections barring “non-autoconfirmed” users3 from making revisions are placed on the pandemic event article by administrators first intermittently and then indefinitely. Page protection isan important content moderation tool on Wikipedia for preventing disruptions and vandalism, but page protections also prevent new and low-activity accounts from directly participating in the collaborations [70]. The indefinite page protections introduced after January would have the intended effect of capping the number of accounts qualified to contribute and thus reducingthe number of revisions made to articles during the severe escalation in March. Figure3 plots the number of daily revisions for the pandemic event article and the sum of the daily revisions for the category and neighbor-related articles. Because there are hundreds or thousands of articles within the category and neighbor-related article sets, even low levels of activity distributed across these pages aggregates into more total activity than the primary articles. Category-related pages (n = 3, 072) had over a thousand daily revisions by early January while neighbor-related pages (n = 1, 166) only briefly passed a thousand daily revisions in mid-March. The late-January burst of revision activity on the pandemic event article was matched by bursts of revisions on the category-related articles and a few days later by an uptick in revisions on the neighbor-related articles. From late February through March, activity on the category-related articles more than quadrupled from fewer than 2,000 revisions per day to 8,099 revisions on March 13. The burst of revisions to category-related articles in late February through March rather than the primary burst of revisions in January is a much better match for the likely demand for information than any of the seed articles’ protection-censored revision activity suggests. Again, neighbor- related article revision activity surged a few days later with 1,083 revisions on March 15 and peaking at 1,131 revisions on March 21. The lag in revision activity on neighbor-related articles compared to category-related articles could be the result of several mechanisms such as editors

3“Auto-confirmed” accounts are typically created more than 4 days ago and have more than 10 revisions.

Fig. 5. Average time between revisions (in hours) for category-related (red), neighbor-related (purple), Fig. 6. Median size of revisions for category-related and pandemic event (blue) articles. (red), neighbor-related (purple), and pandemic event (blue) articles.

prioritizing updates to the most topical content, maxed-out editorial capacity requiring a few days for changes to trickle down to other articles, or consensus to delay updating important neighboring articles like “United States” until more reliable sources exist. Finally, while revision activity on the pandemic event article declines after April below pre-peak activity, revision activity on both the category-related and neighbor-related articles remain elevated compared to their pre-peak levels. Figure4 provides another perspective on the revision activity dynamics by plotting how revision activity changed by hour of the day for each week in 2020. Previous research has shown strong diurnal patterns exist in online social behavior [58, 150]. Despite the global nature of both the event and an- glophone Wikipedians, there is nevertheless a strong and consistent diurnal signal in the revision activity of editors: more revisions in the early evening than the middle of the morning. This diurnal pattern persists over time despite profound differences in the size and intensity of the collaboration over time. While ground truth labels on editors’ geographic locations are not publicly available, these activity patterns are indicative of a geographic bias in the editors residing within the United States as they edit Fig. 4. Heat map of the number of unique users outside of traditional work hours and sleep hours. making revisions by week (x-axis) and hour of the This within-day temporal patterning of revision ac- day (y-axis). tivity has implications for collaboration and coordination if the most active editors are concentrating their attention and activity within narrow windows of time within the day. This points to informal and pluritemporal forms of anticipation, coordination, and delegation exist to manage conflict and to process new information within these collaborations [26, 105].

Proc. ACM Hum.-Comput. Interact., Vol. 3, No. CSCW, Article TBD. Publication date: November 2020. TBD:10 Brian C. Keegan & Chenhao Tan

Revision persistence and size At least two consequences flow from the significant increases in daily revision activity: thetime between revisions must decrease and the size of the revisions should decrease. Figure5 plots the average time between revisions (in hours) on the pandemic event article and the category-related and neighbor-related articles. The intense editing activity on the pandemic event article means that any given revision to an article only persists for at most an hour before it is overwritten by another editor’s revision. When edit activity peaked in late January, the average revision persistence was only a few minutes. The pace of activity on the pandemic event is much higher than on the average category-related or neighbor-related article (hours versus days of persistence), but both the category-related and neighbor-related articles also see a marked decrease in the persistence in late February and through March before increasing again in April. Figure6 plots the median size of a revision (in bytes) on the pandemic event article and the two sets of related articles. The MediaWiki software on which Wikipedia runs does not support synchronous editing so edit conflicts can arise when one editor commits a change to the article before another editor finishes theirs. To avoid losing their contributions due to edit conflicts, editors need to change their contribution strategies to reflect a higher-tempo environment. One strategy is to make smaller and more incremental edits, which would manifest as a decrease in the median size of a revision. As expected, median revision size decreases for the pandemic event article from dozens of bytes to 0 bytes during the late-January activity burst. The article briefly rebounds, but returns to 0 median bytes for the rest of February reflecting contributions like incremental copy-editing rather than authoring large new sections. When the pace of events picks up in late February and March, the median edit size becomes sizable again as new content is added and there is less competition to make edits while the page is protected. The median revision size increases significantly in early April, suggesting more reliable information is being produced (hence bigger updates) while edits are slowing down. The behavior of the category-related and neighbor-related articles are a study of contrasts. Category-related articles see no substantial changes in median revision size over the whole time window. Even as the edit persistence time drops, category-related articles evidently do not become so high-tempo as to require new modes of coordination and editors because editors are consistently able to make larger changes, enabling these articles to grow. Neighbor-related articles’ median revision size changes substantially over the time window, from small incremental 1-byte revisions before January to greater than 10-byte changes from late January through March. Their longer edit persistence is one factor enabling larger edits to be made. These findings demonstrate the importance of characterizing the revision activity content related to a focal set of articles to better understand how Wikipedia contributors shift their editorial attention over a large information space before, during, and after major events.

Article creation and genealogies The events surrounding the COVID-19 pandemic require not only updating existing articles but creating new articles. These new articles include both high profile articles like those about the pandemic event, disease, and virus but also includes more mundane new articles like “Imperial College COVID-19 Response Team” and “Graduate Together”. Figure7 plots the number of articles created each day within the window in our sample. The article creation activity does not track the category-related article revision activity patterns seen in Figure3 perfectly ( r = 0.698): there is no first peak in late January, a second major peak in mid-March rather than rising inlate February, and declining activity through April and May. The prominent peak happens March 12–13 when hundreds of new geographic child articles (“COVID-19 pandemic in Alabama”, “COVID-19

Fig. 7. Number of users making their first edit (pink) Fig. 8. Median daily size (in bytes) for articles re- and articles created (brown) daily. lated by category (red), neighbor (purple), and individual articles.

Fig. 9. Visualization of the genealogical graph of the three seed articles and their top three children. pandemic in Burundi”, etc.) are created. These geographic articles were created by a small handful of editors, initially redirecting to larger geographic entities like “2020 coronavirus pandemic in the United States” or “2020 coronavirus pandemic in Africa” before the redirect was removed and the article populated with standalone content days or weeks later. Figure7 also captures the number of editors who made their first revision to any of the articles in the combined sample. The timing of editors’ first engagements with these collaborations tracks overall category-related article revision activity very well (r = 0.940). However, the top ten most active editors within the sample began their revision activity to articles in this sample before 2020; they were already making revisions to articles that were later linked to by the pandemic event article or migrated into the category-related articles. The topics of these most active editors’ first revisions to articles in the sample are not obvious predictors of editors’ intense activity during the pandemic response: “2019–20 EHF Champions League”, “Sepsis”, “2020 Caribbean Club Championship”, “Singapore Changi Airport”, and “Supergirl (TV series)”. Previous analyses of Wikipedia’s high-tempo online collaborations have found similar patterns of editors migrating their expertise from prosaic but related articles to current events articles [78, 79, 83, 141]. What motivates these editors to migrate to current events collaborations remains an open question. Another way of exploring the processes of article creation is to employ a genealogical framework. New articles are not created in isolation by novices, but are the product of dedicated and/or disaffected users who move their contributions from a previous article to a new article: article creations are also editor migrations. We employ a method for building genealogical graphs within online communities by connecting the communities (in this case, individual Wikipedia articles) together if the earliest editors of new articles were also editors previously contributing to other articles [136]. Note these are distinct from collaboration networks, because a directed relationship only exists if the “parent” article and its activity predates the “child” article. These “family trees” of online communities could play a role in the success by recruiting committed members or

Proc. ACM Hum.-Comput. Interact., Vol. 3, No. CSCW, Article TBD. Publication date: November 2020. TBD:12 Brian C. Keegan & Chenhao Tan

Fig. 10. Changes in the similarity of the text (A), images (B), hyperlinks (C), and external links (D) of the pandemic (blue), disease (orange), and virus (green) articles compared to the most recent week. ensuring access to important resources [30]. Figure9 visualizes the backbone of the strongest genealogical relationships around the three seed articles. Some of these genealogical relationships are unsurprising (the disease article is a child of the virus article) but others reveal hidden flows of editors moving from general to more specific or unexpectedly related topics.

Article size and content similarity Figure8 plots the changes in the size of articles (in bytes) over the time window. The median size for the neighbor-related articles are large and relatively stable: these high-profile articles (see Table2) tend to be old, long, and active with only minor changes needed to be updated for the COVID- 19 pandemic. The median size for the category-related articles are much smaller but decrease significantly from 22,652±2,708 bytes on average to 14,908±2,450 bytes (F = 385.9,p < 0.001) after the mid-March article creation burst (Figure7). This is an artifact of there being hundreds of smaller “stub” articles, like the geographic child articles, bringing the median article size down. The median size of these category-related articles remains depressed throughout April and May, reflecting low levels of revision activity on these articles (Figure2). The category-related and neighbor-related article sizes also provide context for understanding the scale and intensity of the collaborations on the seed articles. The pandemic event article was created on January 5 at 1,098 bytes and grew extremely rapidly in size, passing the category-related articles’ median size on January 15 and neighbor-related articles’ median size on January 22. The disease article was created much later on February 5, but had a very large initial commit (14,063 bytes), and passed the category-related articles’ median size on February 10 and neighbor-related articles’ median size on March 4. The virus article was created on January 9, started small (776 bytes), and grew more slowly, passing the category-related articles’ median size on January 25, although its size has not yet passed the median neighbor-related article size. The rapid growth in the length of the pandemic event and disease articles is remarkable: they overtake the detailed neighbor-related articles within weeks. The pandemic event article has an apparent ceiling on its size: the article’s size was 323,885±49,807 bytes in February and 334,932±23,200 bytes after March, a non-significant difference (F = 2.34,p = 0.129). The event article’s content obviously did not stagnate (Figures2) but rather saw content being replaced (Figure6) as sections were moved to child articles and new citations, media, and templates were added. The disease article has shown slower but consistent growth since its creation only the beginning of a ceiling while the virus article plateaued in February but has also started to grow in length since mid-March. The sizes of the seed articles do not capture the dynamics of how their content has changed over time. Figure 10 plots changes over time in the similarity of four content features on the three seed articles compared to the most recent version of each article in our data.

Textual similarity (Figure 10A) is measured as the cosine similarity between a previous and the most recent revisions’ unigrams in the body of the article (excluding text in image captions and templates) after pre-processing for common English stopwords and punctuation. The pandemic event article shows a pattern of consistently increasing similarity, reflecting on-going but incremental changes in the text rather than sudden shifts. The disease and virus articles have distinctive “steps” in their textual similarity over time, reflecting major additions or edits of content. Text similarity of all articles is relatively high (>0.9) before April, implying that the structure of these articles stabilized quickly and the revision activity is only making incremental changes to the text. Image similarity (Figure 10B) is measured as the cosine similarity between a previous and the most recent revisions’ counts of large images (>100 pixel width to exclude icons). There is a much lower level of similarity in early versions of the articles indicating that few of the images currently present in the article were included then. The consistent change in image similarity week-over-week captures on-going turnover rather than stabilization in the articles’ images. Hyperlink similarity (Figure 10C) is measured as the cosine similarity between a previous and the most recent revisions counts of hyperlinks. Like images, there is an initially low level of similarity and consistent week-over-week growth in similarity reflecting on-going turnover rather than stabilization in the articles’ hyperlinks. External link similarity (Figure 10D) is measured as the cosine similarity of the normalized counts of top-level domains present in external links between a previous revision and the latest revision. Examples include links to the bbc.com, who.int, cdc.gov, and nytimes.com. Like text, early versions of the article have relatively high similarity to the current version (>0.6) and changes over the subsequent weeks are incremental. The content dynamics here show a plateau in similarity scores for February into March, reflecting an stabilizing consensus. The content on these seed articles show different dynamics and provide additional context explaining the revision activity (Figure2), revision size (Figure6), and article size (Figure8). The text and length of these seed articles may not have changed significantly since March, but there is still a significant amount of on-going turnover in the choice of images, hyperlinks, and external links. The changes within a type of content are similar across the three articles, implying the dynamics of expanding and sustaining the content of new articles about current events may follow consistent pathways. The rapid stability in the textual similarity of the article suggests that early editors’ efforts to frame the article may be extremely durable over a time-span of weeks while high-tempo collaborating demands other editors make more incremental changes that manifest in the consistent and significant changes in the similarity of images and hyperlinks. Page protection again is acritical unobserved confounder [70] limiting the kinds of editors who can contribute content. Previous researchers have characterized this collaborative inertia as territoriality [138, 139], but opportunities for larger-scale editing and reappraisal may open on these articles, as it does for other Wikipedia articles that are subject to high-tempo collaborations [83, 141], after a few months.

Pageviews The previous article-level analyses have focused on the dynamics around the production of content, but Wikipedia’s pageview statistics provide a (coarse) measure of the consumption of this content over time. As we noted in the description of our approach, the Wikimedia Foundation does not disaggregate pageview data by geography, device, etc. for privacy reasons. Figure 11 plots the daily pageviews for the three seed articles and the median pageviews for articles in the set of category-related and neighbor-related articles. The event and virus pages show a characteristic “double spike” of attention centered on the late-January quarantines in China and the mid-March

Proc. ACM Hum.-Comput. Interact., Vol. 3, No. CSCW, Article TBD. Publication date: November 2020. TBD:14 Brian C. Keegan & Chenhao Tan

Fig. 11. Daily pageviews for the pandemic (blue), disease (orange), and virus (green) articles with me- Fig. 12. Daily pageviews for articles related by cate- dian values for category-related (red) and neighbor- gory (red), neighbor (purple), and pandemic (blue). related (purple) articles. quarantines in Europe and the United States followed by a consistent decline in attention. Because the disease article was not created until February 5, there was no first peak on this article. But pageviews to non-existent pages are still logged and reported and there actually is a small but perceptible burst of a few hundred pageviews in late January for the yet-to-be-created disease article that redirected at the time to the pandemic event article. If extremely actively-viewed articles like “Donald Trump” or “United States” receive approximately 50,000 pageviews daily, the relative level of attention to these three pandemic-related seed articles during this time window is profound. The last time the pandemic event article had fewer than 50,000 pageviews was January 19 and the disease article on February 22. The maximum pageviews on the pandemic event article passed 1,000,000 for several days in March but still remains in excess of 100,000 daily. The pageviews for the neighbor-related articles (p¯vpre = 2501 ± 573,p¯vpost = 3511 ± 636, F = 113,p < 0.001) and category-related articles (p¯vpre = 1090±303,p¯vpost = 2670±1024, F = 197,p < 0.001) increase modestly but significantly after March 1. Median pageview activity on category- related articles is always less than on neighbor-related articles, again reflecting the greater relative prestige of articles like “New York City” than “COVID-19 pandemic in New York City”. But the correlated increases of attention to category-related (r = 0.965) and neighbor-related (r = 0.869) articles is an example of an “attention spillover” as people navigate to or from the seed articles from the category-related and neighbor-related articles driving up their pageview numbers as well [34, 87, 88, 148, 152]. Similar spillovers in editorial attention were seen in Figures2 and 12 when revision activity on related articles increased significantly as editors updated these articles with new information. While attention spillovers—particularly to the neighbor-related articles—could also be driven by other exogenous events unrelated to the pandemic (2019–2020 Persian Gulf crisis, 2020 Democratic Party presidential primaries, etc.), observing a significant increase across articles representing a diverse set of topics (public health, transportation, politicians, etc.) gives us greater confidence that the correlated bursts of attention on related articles are pandemic-related information seeking. Figure 12 plots the daily pageviews for the pandemic event seed article as a baseline and the sum of pageviews for all articles in the category-related and neighbor-related sets. While the median pageviews of category-related articles never overtook the neighbor-related articles (Figure2), there is an unmistakable surge of demand for information about category-related topics in March that overtakes all the neighbor-related articles. The pandemic event article makes up only a small fraction of this surge and while the remaining attention may have been split across more than

Fig. 14. Fraction of users active who also made a Fig. 13. Cumulative number of unique contributors revision to any in-sample article (category and neigh- on pandemic event (blue), disease (orange), and virus bors) at any time before (grey), in the last week (green) articles. (olive), and yesterday (cyan).

3,000 category-related articles, this captures the enormous demand for information as the severity of the pandemic and its social effects accelerated. Measuring event-driven attention spillovers, the neighbor-related articles see a much clearer increase for the January events than the category- related articles as well as the characteristic second peak. The absence of this first peak for the category-related articles can likely be attributed to the lack of child articles about impacts, locations, and response created in February and beyond (Figure7).

EDITOR-LEVEL ACTIVITY Using the editors who contribute to articles as the primary unit of analysis, we examine how the size of these seed articles collaborations grew over time, the patterns of user re-engagement, edit session length, and changes from editors’ previous contribution behavior.

Re-engagement The number of editors active daily on an article closely tracks the total number of daily revisions (Figure2). Figure 13 plots the cumulative number of unique editors who contributed to each of the three seed articles by day. Over the December 2018 through May 2020 sampling window, articles in our sample had a median of 25 and an average of 57.8 unique editors. The seed articles accumulated significantly more editors in a few months than these other articles had in years: the pandemic event article 2,749 unique editors, the disease article 881 editors, and the virus article 646 editors. Changes in the number of editors on these articles correspond with major turning points in the pandemic. This rapid growth in the number of contributors to these articles could be a source of tension as editors bring different values and expertise, but it is also the case that most editors’ engagement on these articles is extremely limited, typically making only one revision to one article (see Figure 21A and C and discussion in next section). How many of the editors active on an article one day were previously active? Figure 14 plots the fraction of editors who made a revision to any category-related or neighbor-related article between December 2019 and May 2020. On any given day, at least 20% of the editors also made a contribution yesterday and at least 60% of the editors made a contribution in the previous year. The re-engagement of editors in making contributions is not static and there is a significant increase after March 1 in editors’ re-engagement (re) from yesterday (re¯ pre = 28.4±4.7%,re¯ post = 42.4±4.8%, F = 346.4,p < 0.001), in the last week (re¯ pre = 43.3 ± 4.2%,re¯ post = 55.3 ± 5.0%, F = 259.1,p < 0.001),

Proc. ACM Hum.-Comput. Interact., Vol. 3, No. CSCW, Article TBD. Publication date: November 2020. TBD:16 Brian C. Keegan & Chenhao Tan

Fig. 15. Percentage of users active on the event, disease, or virus article who were active yesterday (left) or any time before (right) on the event, disease, or virus articles.

and in the past year (re¯ pre = 64.0 ± 2.9%,re¯ post = 68.7 ± 5.1%, F = 54.9,p < 0.001). Even as the size and intensity of the collaboration surges, a plurality of the contributing editors have previously been active on these pages. These high levels of re-engagement indicate a strong motivation among editors to monitor and participate in the collaboration rather than treating them as “drive by” contributions. Even as overall revision (Figure2) and pageview (Figure 11) activity has declined through April, these re-engagement statistics remain at their significantly elevated levels. The re-engagement statistics reported in Figure 14 are across the whole sample of articles and does not capture whether editors are “staying in their lane” contributing on their usual articles or how they are moving between articles. Figure 14 visualizes the re-engagement dynamics of editors moving within and between the three seed articles as measured day-over-day and from anytime since article creation. For example, 30–40% of editors on the pandemic event article on any given day after mid-January were also active on the previous day (blue line, Figure 14A). In mid-February, close to 60% of the editors active on the disease article were active on the event article the day, falling to 40% in mid-March, and rising again to more than 60% in early May (red line, Figure 14A). Fig. 16. Changes in the distribution of The February 5 creation of the disease article is an occasion users’ daily median revision latencies. for significant collaborative migrations of editors from other articles to contribute to the new article. The intensity of cross-article re-engagement declines in early March with the surge of activity related to external events bringing in newcomers. However, these newcomers leave or are converted into committed editors and the re-engagement numbers recover rising through April and into May. These cross-article re-engagement dynamics are a critical mechanism for coordination and standardizing content across articles while also illustrating a remarkably strong commitment among editors for stewarding their contributions and participating in these collaborations for an event lasting months.

Edit sessions Previous work exploring edit sessions on Wikipedia identified a characteristic double-peak in contribution patterns: one reflecting behavior occurring in relatively rapid sequences with only minutes between successive revisions and another reflecting daily patterns with approximately a

Fig. 17. The number of daily revisions to all articles from different editing cohorts (left) and the percentage of users active on all articles from different editing cohorts (right). Editing cohorts are weekly in 2020 andall users first active before 2020 are labelled 2019. day between revisions [51]. Given the significant, sudden, and uneven shifts in revision activity, attention, content, and re-engagement around pandemic articles, it is reasonable to hypothesize that stable patterns of edit sessions could not hold. Similar to the idea of edit lags within an article’s revision history in Figure5, we can also analyze the time between revisions in a user’s contribution history within this sample of category-related and neighbor-related articles. Figure 16 plots a two-dimensional histogram of users’ median revision latencies changing over time. The y-axis are logarithmic bins of time between observed revisions, ranging from seconds at the bottom to months at the top. The x-axis are weekly bins of activity since January 1. The values in each cell are the number of users in that week who had a median revision latency in that bucket’s range. We find a similar characteristic double-peak in revision latencies that is consistent overtime during even this high-tempo collaboration. There is one population of users who typically make multiple revisions only minutes apart and second population of users who typically make revisions once every few days. The number of editors in each of these populations increases in week 11 corresponding to the early March surge of activity, but the double-peak remains consistent over time.

Cohort analysis Simpson’s paradox arises when heterogeneous data is aggregated producing different associations and trends than exist in any of the underlying subgroups and is an endemic problem in the social and computational sciences [17, 92]. One strategy for surfacing the existence of this paradox are “time-aware analyses” by bucketing users into cohorts to explore whether differences exist in the behavior of these subgroups [2, 9]. We group editors into cohorts based on their first contribution to articles in our sample: a “2019” cohort for all users who made contributions before 1 January 2020 (n = 60, 258) and weekly cohorts for users making their first in-sample contributions in 2020 (n = 58, 296,n¯i = 2, 915). Figure 17A revisits the findings from Figure3 by plotting the number of daily revisions from editors in each of these cohorts. The revision activity from the “2019” editors are the most active, partially because they are the largest cohort, but also because it includes

Proc. ACM Hum.-Comput. Interact., Vol. 3, No. CSCW, Article TBD. Publication date: November 2020. TBD:18 Brian C. Keegan & Chenhao Tan

Fig. 18. Revision count-weighted average age (in Fig. 19. Revision count-weighted average edit count days) of registered editor accounts on the pandemic of registered editor accounts on the pandemic event event (blue), disease (orange), and virus (green) arti- (blue), disease (orange), and virus (green) articles. cles. highly-engaged editors. In contrast to this 2019 cohort, the number of revisions from later cohorts shows a consistent pattern of temporary engagement: these are the newcomers who make some contributions to in-sample articles but most are not found a week later. This replicates previous findings about the significant churn in editors of politicians’ biographies and campaign articles[82]. Figure 17B plots the daily percentage of editors making revisions drawn from each of these cohorts. Even the editors who began contributing during the mid-March surge had only temporary engagement with the article. Obviously revisions made before 2020 can only come from the 2019 cohort, but this 2019 cohort is responsible for more than 40% of the active users in 2020. Again, the 2020 cohorts’ temporary engagement manifests as a single or few days of contributions that become negligible within a week or two. These results reproduce similar findings about high levels of turnover in editor cohorts on Wikipedia articles [82], but they cast a new light on the findings from Figures 14 and 15 that implied a strong tendency for users to remain engaged day-over-day and week-over-week. The 2019 cohort of editors includes a subset of editors whose sustained and consistent revision activity over months is responsible for the majority of revisions and editors during the surges of activity (e.g., mid-March) as well as sustaining maintenance (e.g., late April). This finding highlights the critical role of established contributors.

Editor backgrounds What are the backgrounds of editors on the seed articles? There are many ways to explore this question and we do it through two analyses. For each registered editor who contributed to these three seed articles, their summary statistics including total global edit count and account creation date were retrieved from the “userinfo” API endpoint. These user-level statistics were joined with the revision histories to compute revision-count weighted aggregations: if an editor with an account age of 1,000 days and a 1,000 editcount made 10 revisions to a seed article on a day, their account age and edit count would be counted 10 times in that day’s average statistic. Because edit counts can be strongly skewed by editors using automated tools, we employ a normalization dividing the total revisions an editor made to any of the seed articles by their total edit count. This “edit count fraction” can be thought of as the total focus an editor had on these seed articles over their whole editing career: editors who contributed only to coronavirus-related topics and nothing else would have an edit count fraction of 1.0, editors made 50 revisions to articles about penguins than then made 50 revisions to coronavirus-related topics would have an edit count fraction of 0.50.

Feature Before After ANOVA Kruskal-Wallis Dates active 26.9 ± 29.5 29.3 ± 30.9 6.76, p = 0.009 9.4, p = .002 Latency 7.46 ± 29.5 3.79 ± 16.0 29.2, p < 0.001 0.0, p = 0.855 Namespaces 3.89 ± 3.1 4.13 ± 3.3 6.95, p = 0.008 5.0, p = 0.026 Revisions 366 ± 1903 386 ± 1117 0.21, p = 0.646 3.1, p = 0.077 Category revisions 8.46 ± 51.4 43.5 ± 178.8 90.5, p < 0.001 1727, p < 0.001 Neighbor revisions 9.80 ± 45.9 5.19 ± 21.0 21.36, p < 0.001 59.0, p < 0.001 Neither revisions 1147 ± 3390 337 ± 1050 132.8, p < 0.001 319, p < 0.001

Table 3. Summary of editors’ changes in revision behavior after their first revision to any of the seed articles. Mean and standard deviation are reported for each feature for activity before and after the first revision and statistical tests and p-values are reported for differences.

Fig. 20. Visualization of changes in editors’ revision activity across article types from Table3.

Figure 18 plots revision-count weighted average age of registered editors on the three seed articles. The early editors on the pandemic event article before February had relatively young accounts (less than 2,000 days) but the average account age increases to approximately 2,750 days with the first editing surge in late January, rises again to approximately 3,000 days with thesecond editing surge in March, and remains high thereafter. The other seed event articles have much higher average editor account ages over the time window but some similar increases in account age correlated with external events. The increases in account age reflect an influx of more established editors to these articles. Figure 19 plots the revision-count weighted average editor count fraction on the three seed articles. Before approximately March 1, the editors making contributions to these articles had high edit count fractions indicating a high degree of attention and effort focused on these coronavirus collaborations. The values for all three seed article decrease with the March surge as established editors whose efforts were previously focused elsewhere enter the collaboration, a finding consistent with the increase in account ages seen in Figure 18 as well as the persistence of the 2019 cohort of editors seen in Figure 17. Whether this hand-off of contribution between two distinct groups of editors, from the younger founders to an older guard, was accompanied by conflict or relief isa question for future research.

Behavioral change Did participating in these breaking news collaborations change editors’ contribution patterns? Using the set of editors who contributed to the three seed articles, their contribution histories from December 2018 to the present were retrieved from the Wikipedia API. Using these editors’ first

Proc. ACM Hum.-Comput. Interact., Vol. 3, No. CSCW, Article TBD. Publication date: November 2020. TBD:20 Brian C. Keegan & Chenhao Tan contributions to any of the three seed articles as a pseudo-treatment, we match the amount of time in each editors’ contribution history (across all articles and namespaces, not just our COVID-19 sample) from their first revision to their most recent revision with an equivalence amount oftime immediately before the first contribution: if an editor was active for 30 days after their first revision to one of the seed articles, we look at their entire contribution history 30 days before and after that first contribution. We perform simple statistical analyses using both ANOVA (“are meansthe same?”) and Kruskall-Wallis (“are medians the same?”) to test whether contributing to one of these seed articles changed users’ overall contribution behavior within matched time windows around their first contribution to a seed article. Table3 reports the results of these analyses. After their first revision to a seed article, editors make contributions on significantly more days, significantly less time elapses between their revisions, they contribute to more namespaces (articles, talk pages, administrative processes, etc.), but do not make significantly more revisions. This last result is surprising because participating inan engrossing breaking news collaborations could stimulate editors to contribute more frequently, if only because they have to make more and smaller edits (Figure5) and editor re-engagement increases through our time window (Figure 14). A follow-up analysis of editors’ changes in revision activity to category-related, neighbor-related, and unrelated articles revealed significant differences in revision activity with important implications. Editors’ revision activity on category-related articles increased significantly—five-fold—after their first revision to a seed article, but revision activity on both neighbor-related articles and unrelated articles decreased significantly: editors shift their focus from neighbor-related and unrelated articles to category-related articles but do not increase their activity after contributing to a seed article. One worrying implication of this profound decrease in activity on unrelated articles is that breaking news collaborations may be draining editors’ attention away from other parts of Wikipedia. Whether other editors step in to pick up the slack on these forsaken articles, these editors return back their previous editing routines in the medium term, or these editors burn out and take a break after participating in a breaking news collaboration remain open research questions. Additionally, the non-zero before values for category-related and neighbor-related articles imply editors both (1) contribute more to neighbor-related articles before contributing to seed articles and (2) contribute more to category-related articles after contributing to seed articles. This suggests an aggregate flow of editors recruiting themselves from both unrelated and neighbor-related articles tojointhe seed article collaborations and then moving on to other category-related articles afterwards.

NETWORK DYNAMICS Networks are aggregations of relational data and network dynamics capture how the nodes and edges within a network change over time. These types of networks are not exhaustive of all the relevant networks—discussion, citation, and categories are examples of other relationships that warrant future analysis—but we focus on two to understand the movement of editors and changes in article content. First, we explore the collaboration networks of editors making revisions to articles and how these structures changed as editors shifted their contributions across an expanding set of relevant articles. Second, we explore the hyperlink networks of articles linking to other articles and how these structures changed as new information came to light. We define each of these networks in more details in the sub-sections below.

Fig. 21. Cumulative collaboration network activity (logarithmically-binned) distributions.

Collaboration Collaboration networks capture the structure of editors making contributions to articles. These bipartite graphs4 are unique because two distinct types of nodes exist and relationships can only exist between different node types18 [ ]. In other words, editors cannot revise editors and articles cannot revise articles; only editors can revise articles. Bipartite graphs require distinct algorithms for analyzing their structure in light of this constraint [89]; we use the bipartite class within Python’s networkx library [60]. The dynamics of Wikipedia’s collaboration networks can be captured with two different strategies. The first strategy isa cumulative collaboration network capturing all the revision activity between editors and articles leading up to a given date. The second strategy is a snapshot collaboration network capturing only the revision activity between editors and articles on a given date. Because even significant changes in activity can be drowned out by the magnitude of previously accumulated behavior in the cumulative strategy, we visualize daily changes using the snapshot strategy in Figures 22 and 23. Figure 21 captures the distributions of connectivity and activity in the cumulative collaboration network. We employ logarithmically-binned values in combination with double-logged axes to identify structure in the “tails” of the distribution [100, 144]. There are “long-tailed” distributions in all three facets: most editors contribute to one article, most articles have few editors, and and most editors make a single revision (upper left of each facet) while some editors edit hundreds of articles, some articles have thousands of contributing editors, and some editors revise an article hundreds of times (lower right of each facet). While “long-tailed” distributions are ubiquitous in many complex systems like online social behavior [74, 101], these distributions are not necessarily generated by mechanisms like preferential attachment [4, 21, 25, 99, 102]. We emphasize the truncated tails in all three of these distributions are deviations from true power laws and imply an over-representation of high-activity users, pages, and revisions. Figure 22 captures the size of the daily snapshot collaboration networks. The activity here mirrors the dynamics seen with revision (Figures2 and3) and pageview (Figure 11 and 12) activity: an initial rise in late January with coverage related to the initial outbreak in Wuhan, a rapid increase from late February through early March when major outbreaks are reported in Europe and the United States, and a decline in activity from the middle of March to the end of May as interest wanes. There is a significant spike of collaboration activity on May 4 when the primary seed articlewas renamed from “2019–20 coronavirus pandemic” to “COVID-19 pandemic” and required updating thousands of articles. Figure 23 captures the “cohesion” of the daily snapshot collaboration networks using two types of features: average bipartite clustering coefficients89 [ , 107] and fraction of nodes in the largest

4Also known as two-mode or affiliation networks.

Proc. ACM Hum.-Comput. Interact., Vol. 3, No. CSCW, Article TBD. Publication date: November 2020. TBD:22 Brian C. Keegan & Chenhao Tan

Fig. 23. Changes in the clustering coefficients and Fig. 22. Size of the “snapshot” collaboration net- LCC fraction in the “snapshot” collaboration networks. works. connected component (LCC fraction). The editor clustering coefficient captures the extent to which editors share articles in common and the article clustering captures the extent to which articles share editors in common. The clustering coefficient values are relatively stable until mid-March when the production and consumption of information from these collaborations shifts. Editor clustering (pink line) declines slightly, suggesting editors are less likely to work together and instead focus their contributions on fewer articles. Article clustering (brown line) increases substantially, suggesting articles are more likely to share editors. These seemingly contradictory results can be reconciled by considering the dramatic changes in the size of the collaborations during this time: there are many more editors than articles, new editors joining and new articles being created simultaneously make it harder for all editors to contribute to many articles in common, but the contributions from a handful of editors working across multiple articles increases the article clustering. The increases in article re-engagement (Figure 14) as well as the increasing cross-article engagement among the seed articles (Figure 15) also reinforce this interpretation. The LCC fraction captures the extent to which the collaboration is fragmented across small and isolated engagements or coheres into a large and well-connected effort. The LCC fraction (black line) mirrors the dynamics seen in Figure 22 with a pre-outbreak low connectivity, Wuhan-era moderate connectivity, and a pandemic-era high connectivity phases. The LCC fraction peaks with the mid-March surge of interest with an influx of new editors and article creations and then again in early May with the renaming event and approximately 80% of editors and articles are indirectly connected together into a giant component. There are three distinct “steps” in the LCC fraction over time. The first step is a “pre-pandemic” phase ending in late January when articles are edited in relative isolation from one another: editors on the “World Health Organization” article in late December are unlikely to also revise the “2020 Democratic Party presidential primaries” article on the same day, so these articles and their editors belong to separate components in this analysis. In other words, the low LCC fraction at this time is the “shadow of the future” capturing the behavior of articles that do not yet know they will become linked together in the future through this pandemic event. The second step is an “early pandemic” phase from late January to mid-February when some articles begin to be edited together. The “shadow of the future” still looms and the relatively low LCC fraction continues to capture the extent to which many articles that later become related to the pandemic are not yet aware and so remain disconnected from the emerging coronavirus collaboration. The third step is the “peak pandemic” phase from late February through the end of May with a high LCC fraction as hundreds of previously unrelated articles are now updated

Fig. 24. Spring layout visualization of the largest connected component in the cumulative hyperlink network. Nodes are sized by in-degree and colored by first appearance in the network (bluer earlier, redder newer). Edges are weighted by the number of weeks they appeared in the network and colored by first appearance in the network (bluer earlier, redder newer). The pandemic (P), disease (D), and virus (V) articles are annotated along with 8 specific clusters. on the same day to reflect the effects of the pandemic. The spike in early May is an artifact ofthe renaming event.

Hyperlinks The hyperlink network captures the links from the body of an article to other Wikipedia articles. We only look at the hyperlinks within the combined set of category-related and neighbor-related articles. We purposely exclude links within templates and infoboxes because these are often repeated across articles (causing artificially dense networks) and more accurately captures the semantics of each article’s content. Because some articles link to out-of-date (e.g., “2019-20 Wuhan outbreak”) or non-standard article names that are then redirected to the canonical article title [69], we resolve each revision’s outlinks to the canonical article title. Finally, we also retrieved the historical content of each article in the sample on a weekly frequency to analyze how the creation of new pages and links affected the structure of the hyperlink network over time. Figure 24 is a spring layout visualization of the “cumulative” hyperlink network that locates nodes with similar connections closer together. Nodes and edges are colored by the date of creation, with bluer colors corresponding to parts of the network that already existed or were introduced earlier and redder colors corresponding the parts of the network that were introduced later. The article about the pandemic event (P, approximately 1 o’clock), disease (D, approximately 10 o’clock), and virus (V, also 10 o’clock) are annotated. The sampling strategy ensures that this network

Proc. ACM Hum.-Comput. Interact., Vol. 3, No. CSCW, Article TBD. Publication date: November 2020. TBD:24 Brian C. Keegan & Chenhao Tan

Fig. 25. Changes in the in-degree centrality (A), closeness centrality (B), eigenvector centrality (C), reciprocity (D), clustering (E), and average in-degree neighbor’s in-degree (F) of the pandemic (blue), disease (orange), virus (green), and mean (grey) articles within the weekly hyperlink networks. is relatively densely-connected but there are several sub-communities with particularly dense connections annotated 1 to 5 counter-clockwise: Group 1. Medical topics like “Infection”, “Pneumonia”, and “Coronavirus”. Group 2. Locality-specific pandemic articles like “COVID-19 pandemic in Italy”. Group 3. Geographic and historical articles like “Italy” and “World War II”. Group 4. Air transportation articles like “American Airlines” and “Boeing 737”. Group 5. Chinese social articles like “Community Party of China” and “Pinyin”. Group 6. News and social media articles like “The New York Times” and “Twitter”. Group 7. Impacted events like “2020 Kentucky Derby” and “2020 U.S. presidential election”. Group 8. Institutions like “University of Oxford” and “World Health Organization”. Other topical categories are also present in the network but outside coherent sub-communities: biographies of political leaders, scientists, and victims; releases and events cancelled due to the pandemic; universities and companies impacted by and/or responding to the pandemic. The hyperlinks within well-established sub-communities like groups 3–6 have little sign of change: articles like the “United Kingdom” and “World War II” and the links between them existed before the pandemic began. The pandemic article was the first of the core articles created and the rainbow ofhued links around it reflect the growth in the content of the article over this time window: older(bue and green) links going towards articles in Groups 3 and 5 and newer links (yellow and red) going towards articles in groups 1 and 2. The disease article was created four weeks later and is deeply embedded between groups and 2, reflecting the similarity in linking behavior Because of the difficulty parsing network visualization “hairballs”, we analyze how the pandemic event, disease, and virus articles’ positions in these networks changed over time. Figure 25 plots six different features and the values for the three seed articles compared to the average values of all articles in the hyperlink network. Summarizing the findings below: the seed articles have many connections and became increasingly prominent in the hyperlink network, but many of these connections are from newly-created but poorly-connected articles.

In-degree centrality (Figure 25A). In-degree centrality captures the number of other articles that link to an article. We report values normalized by the size of the network to capture the fraction of other nodes in the network connecting to the three seed articles. Unsurprisingly, all three seed articles’ in-degree centrality grows significantly from less than 1% of the nodes in the network connecting to them to 45% of articles connecting to them by the end of May as other articles update their content and include hyperlinks to these articles. The average in-degree centrality for the rest of the network declines from March through May as hundreds of new articles are created (Figure7) but receive few links from other articles in the network Closeness centrality (Figure 25B). Closeness centrality captures the average shortest path length an article has to every other article in the network. Lower closeness centrality implies greater “distance” or more other articles that need to be passed through to reach the rest of the network and higher closeness centrality implies less “distance” and more articles can be reached directly or through few other articles. All three seed articles’ closeness centrality increases substantially from late February through the end of May as these articles link to and are linked from other articles in the network. Eigenvector centrality (Figure 25C). Eigenvector centrality captures the “prestige” of an article having connections from other “prestigous” articles. The seed articles’ eigenvector centrality also increases significantly over the average article’s eigenvector centrality starting in late February with the second revision surge. Receiving links from high-profile articles like “United States” or “World Health Organization” increased the eigenvector centrality of the seed articles. Reciprocity (Figure 25D). Node-level reciprocity is the ratio of hyperlinks pointing both to and from an article to the total hyperlinks around an article. Greater reciprocity implies a stronger relationship between articles as both articles link to each other. The reciprocity of the seed articles grows quickly tracking the initial growth of article size (Figure8), stabilizes through late March, then declines as the burst of newly-created national and local-level pandemic articles (Figure7) are expanded with many more links coming into the seed articles (Figure 25A) but these seed articles not reciprocating by linking to minor articles like “COVID-19 pandemic in Easter Island”. The average reciprocity in the whole hyperlink network declines around the same time for similar reasons and the reciprocity for the seed articles is similar to the rest of the network. Clustering (Figure 25E). Clustering captures the fraction of an article’s hyperlink neighbors that are connected to each other. Greater clustering implies an article is embedded with a cohesive sub-community while lower clustering implies an article’s neighbors are poorly connected or the article is bridging between otherwise poorly connected sub-communties. The clustering for all three seed articles rises briefly in the earliest weeks following creation then falls dramatically as the articles become more prominent and surrounded by diverse topics that do not link to each other. The clustering for these seed articles is much less than the average value in the rest of the network, again reflecting the prominence of these articles in the network in connecting different topics together. Average neighbor in-degree (Figure 25F). The average neighbor in-degree is a measure of how well-connected the articles linking to an article are themselves. An article with few links from other articles (low in-degree) but those articles being highly-linked from other articles would have a high average neighbor in-degree. The values for the seed articles peak immediately after their creation and decrease significantly before stabilizing in March and April. This is a result of new child articles linking to the pandemic event, disease, and virus articles even if these articles receive few links from any other articles in the network.

Proc. ACM Hum.-Comput. Interact., Vol. 3, No. CSCW, Article TBD. Publication date: November 2020. TBD:26 Brian C. Keegan & Chenhao Tan

DISCUSSION How did Wikipedia’s coverage of the coronavirus pandemic evolve? We explored this research question through three levels of analysis: the dynamics of articles’ revision activity and content, the contribution behavior of editors, and the structure of collaboration and hyperlink networks. This analysis is an in-depth case study using quantitative methods to describe the emergent collaborations around a historic global disruption in nature. The individual results in the previous sections illuminate how information production and consumption on these articles changed profoundly and rapidly, contributing editors increased their engagement and production by shifting their attention, and networks of collaborators and content coalesce in the aftermath of events. These individual results provide overlapping, cumulative, and emergent evidence of more general themes enabling these high-tempo collaborations. We adopt approaches from qualitative research like open coding to identify emergent themes common across the quantitative results from each level of analysis. This combination of an in-depth case study structured as multiple levels of analysis, application of mixed quantitative methods across these levels, and inductive analysis of their results is a kind of “quantitative portrait.”Much like pointillist art aggregates individual points into coherent structures, a quantitative portrait aggregates simple data into intermediary descriptive results that then coalesce into more coherent themes. These themes also reveal strategies for engaging in quantitative portraiture like consistent annotation, derivative features, re-classifying sub-groups, and retroactive sampling. We summarize the themes in our portrait in Table4 and describe how we arrived at these themes and their implications for doing quantitative portraiture in more detail below.

Theme 1: Event-driven activity and derivative features The shocks of significant external events influence information production and consumption behaviors that ripple through our results. Significant changes in the scale, tempo, and similarity of activity happened in the immediate aftermath of announcements as people sought, created, and updated information. Much of the activity is centered on two similar events: the announcement of quarantines, first in China in late January and then in Europe and the United States in themiddleof March. These events drove surges of editing and pageview activity across our sample of thousands of articles when examined in the aggregate (Figures2,3, 11, 12). These twin events oriented our interpretations of subsequent analyses—as seen in our repeated use of vertical annotations in time series—focusing on more abstract features. Annotation practices across diverse descriptive results enabled us to identify anomalies when the behavior of articles, editors, or networks either “resisted” the expected disruptions from external events or activity bursted or trends changed in the absence of any salient event. The absence of bursts and shifts in trends were both anomalies and invitations for “zooming in” to the underlying data to understand the mechanisms at play through both close interpretations of the underlying digital trace data [53] as well as exploration of related features. This could take the form of comparison to other time series for fingerprints of similar anomalies, manually inspecting archived revisions on Wikipedia itself to understand the contexts, or developing derivative aggregations or features to interrogate the assumptions of the first analysis. One example of this type of thelast form of triangulation—analyzing derivative features—was testing whether the surprisingly high re-engagement statistics were robust across different sampling windows (Figures 14 and 15) and finding similar trends. Grounded in the hypothesis that major events should structure the activity and behavior of these collaborations, consistent annotation of those events in our results enabled us to navigate findings across levels of analysis.

Theme 1: Event-driven activity and derivative features. Theme 2: Core-periphery spillovers and expanding samples. Theme 3: Contradictory user engagement and reclassification. Theme 4: Shadow of the future and retrospective sampling.

Figure Description T1 T2 T3 T4 2 Revision peaks follow external events but slow down since April ✓ ✓ ✓ 3 Wide-spread and intense revision activity across articles ✓ ✓ ✓ 4 Revision activity has consistent dirurnal cycles ✓ ✓ 5 Revision persistence only minutes during most acute editing ✓ ✓ 6 Revision size varies in response to intensity and events ✓ ✓ 7 Surge of geographic child article creation to manage coverage ✓ 8 Rapid growth in length of articles, surpassing high-quality neighbors ✓ ✓ Article 9 Seed pages serve as parents of newly created pages ✓ ✓ 10 Substantial differences in content of articles week-over-week ✓ 11 Double peak in attention, category overtakes neighbor ✓ ✓ ✓ 12 Millions of daily pageviews to pandemic-related articles in March ✓ ✓ ✓ 13 Growth in number of collaborators tracks revision activity ✓ 14 High levels of editor retention day-over-day, week, and ever ✓ 15 Strong flows of editors between seed articles ✓ 16 High and low-tempo contribution behaviors consistent over time ✓ ✓ ✓ ✓ User 17 Pre-pandemic editors remain the most active 18 Ages of collaborators regress to the mean ✓ ✓ ✓ 19 Earnest novices replaced by grizzled veterans ✓ ✓ 20 Contribution behavior is disrupted after first seed edit ✓ ✓ ✓ 21 Strong skews in activity across levels ✓ ✓ 22 Size of collaborations tracks revisions ✓ ✓ ✓ 23 Increasing coherence of collaboration ✓ ✓ ✓ 24 Clusters emerge among the category- and neighbor-related pages ✓ ✓ Network 25 Seed articles’ network positions become prominent ✓ ✓ ✓ Table 4. Summary of findings from individual figures with thematic mapping (T1, T2, T3).

Theme 2: Core-periphery spillovers and expanding samples The multiple events unfolding across time as well as the proliferation of child articles about the regional, national, and sub-national effects of the pandemic created an unusually large “periphery” of articles surrounding the seed articles about a global pandemic, the disease, and the virus. Previous analyses of Wikipedia’s breaking news collaborations as well as crisis informatics hashtag-centered sampling both bake in assumptions that privilege artifacts around singular event. The interactions afforded by a platform and behaviors queriable via an API contribute significantly to this privileging of “core events” while ignoring the effects of the event on a broader ecosystem of related content and behavior. Incorporating peripheral articles—category-related and neighbor-related articles— alongside our analysis of core “seed” articles revealed similarities and differences meriting further investigation. Our sampling frame enabled to explore how the bursts of activity on the core articles were not isolated but also “spilled over” into the rest of the information ecosystem: editors made revisions across articles causing them to connect as networks and readers consumed information across articles causing their pageviews to spike in tandem. “Zooming out” from the core to the periphery enables us to see how distinct but similar articles behave differently following the same event orthe profound scale of activity in the seed articles are yet still dwarfed by behavior unfolding in parallel

Proc. ACM Hum.-Comput. Interact., Vol. 3, No. CSCW, Article TBD. Publication date: November 2020. TBD:28 Brian C. Keegan & Chenhao Tan around them. Grounded in the hypothesis that different types of similarity should reveal different kinds of behavior, our inclusive sampling approach identified two discrete sets of related articles that became a reliable basis for comparison with each other as well as with articles in the core (Figures3,8, 12). The similarities or differences between these distinct sets of core and peripheral articles enabled us to navigate findings across levels of analysis.

Theme 3: Contradictory user engagement and reclassification Because editors are the agents of so much of the change seen in the data, it is natural to ask who they are but the results paint a complex and conflicting picture. The tempo of editing articles can become so rapid that content is overwritten within minutes (Figure5) yet editors also do not deviate significantly from their diurnal patterns (Figures4 and 16). A majority of editors make one or two contributions (Figure 21) but more than 60% of editors on any given day already contributed before (Figure 14). These contradictions arise from problems like Simpson’s paradox that invite inquiry into how aggregations might be biased by different behaviors of unobserved sub-groups [2, 92]. In asking about unobserved sub-groups, we can also begin to reflect on the factors that bind users together into sub-groups and why those relationships might be hard to observe. For our questions around user engagement, we artificially stratified users by a seemingly arbitrary feature like temporal co-occurrence of first edits and yet recovered new perspectives (Figure 17). Grounded in the experience that conflicting evidence can be the product of confounded behavior, the results from these reclassifications of existing data clarify some of the observed contradictions: engagement can remain high with participation so strongly skewed to low-effort edits because there is a sub- population of committed editors whose commitment predates the collaboration and sustains it over time. All data analysis involves some form of coarsening, aggregation, and generalization but these introduce biases and leads to complexities. By revisiting our assumptions for aggregation, we opened new lines of inquiry to help in triangulating across levels of analysis and navigating conflicting results.

Theme 4: Shadow of the future and retrospective sampling In game theory, the “shadow of the future” that explains how we behave differently when we expect to have interactions in the future. In the case of the prisoner’s dilemma, you cooperate now so you are not punished for defecting in the future. Our sampling strategy is an interesting form of time travel in that we take what we know about the present (e.g., editors and articles linked together in the aftermath of a deadly global pandemic) and return to their past to effectively watch them get hit by a proverbial bus. But our units of analysis—articles and editors—are only in our sample because historic events they know nothing about at one time will eventually draw them together. Traditional methods of social inquiry do not have the luxury of high-resolution archival data that allow researchers to reconstruct the state of the world before a disruption so they can track the behavior of people through it. Analyses like the changes in content (Figure 10), cross-article re-engagements (Figure 15), or changes in network structure (Figure 23 and 25) allow us to revisit these past events and to trace what people did at the time in the face of ambiguity and even danger. Editors have many first contributions, but some significantly change their behavior for weeksor months (Table3 and Figure 20), which they do not know at the time. This retroactive sampling of users’ behavior before their actions has remarkable (and ironic) parallels to epidemiologists’ contact tracing. We started with a strategy for expanding the sample along a dimension of topical similarity, but temporal dimensions such as users’ contributions before engaging in a collaboration or reconstructing a high-fidelity previous state of the world bring in other kinds of similarities and dependencies that can be analyzed. Of course, reliable access to these kinds of high-resolution archival data requires increasingly precious data infrastructures [20, 49, 114].

Limitations and future work As with so much Wikipedia research, these analyses and findings use only data and behavior from the English Wikipedia. Because different language editions of Wikipedia exhibit significant differences in content and behavior [8, 64], our findings about how the English Wikipedia responds to crises may not generalize. As noted in the analyses above, page protection is a ubiquitous confounder in our analysis by severely limiting the kinds of users who can participate. But Wikipedia’s page protections are an under-appreciated empirical setting for studying content moderation strategies: in the absence of market demands for engagements or subscriptions, Wikipedia can employ blunter strategies to cope with bad behavior. While the quantitative portraiture method we have outlined here has a greater commitment to induction and triangulation than most quantitative methods admit, it nevertheless involves significant abstractions of social behavior into aggregated counts [71]. Qualitative methods like trace and virtual ethnography, interviews, and discourse analysis are still sorely needed to fully apprehend the dynamics animating the timing of changes to articles, the motivations of editors who self-select into these bizarre collaborations, and the values in play around debates about what kinds of content to include.

CONCLUSION Wikipedia has a dogged capacity to respond to current events over two decades. In the face of a historic global pandemic, enormous collaborations have self-organized to document information about the events of the pandemic, evidence about the disease, and the details about the virus as well as hundreds of more specific articles. Given the enormous space of issues, we performed a “quantitative portrait” using a variety of quantitative methods at multiple levels of analysis to understand the features of this remarkable collaboration. Using two dozen descriptive results, we identify four themes cutting across them and the implications these have for combining quantitative methods.

ACKNOWLEDGMENTS Wikimedia Foundation for the data.

Proc. ACM Hum.-Comput. Interact., Vol. 3, No. CSCW, Article TBD. Publication date: November 2020. TBD:30 Brian C. Keegan & Chenhao Tan

REFERENCES [1] Reham Al Tamime, Richard Giordano, and Wendy Hall. 2018. Observing Burstiness in Wikipedia Articles during New Disease Outbreaks. In Proceedings of the 10th ACM Conference on Web Science (WebSci ’18). Association for Computing Machinery, Amsterdam, Netherlands, 117–126. https://doi.org/10.1145/3201064.3201080 [2] Nazanin Alipourfard, Peter G. Fennell, and Kristina Lerman. 2018. Using Simpson’s Paradox to Discover Interesting Patterns in Behavioral Data. In Twelfth International AAAI Conference on Web and Social Media (ICWSM). AAAI. [3] Benjamin M. Althouse, Samuel V. Scarpino, Lauren Ancel Meyers, John W. Ayers, Marisa Bargsten, Joan Baumbach, John S. Brownstein, Lauren Castro, Hannah Clapham, Derek AT Cummings, Sara Del Valle, Stephen Eubank, Geoffrey Fairchild, Lyn Finelli, Nicholas Generous, Dylan George, David R. Harper, Laurent Hébert-Dufresne, Michael A. Johansson, Kevin Konty, Marc Lipsitch, Gabriel Milinovich, Joseph D. Miller, Elaine O. Nsoesie, Donald R. Olson, Michael Paul, Philip M. Polgreen, Reid Priedhorsky, Jonathan M. Read, Isabel Rodríguez-Barraquer, Derek J. Smith, Christian Stefansen, David L. Swerdlow, Deborah Thompson, Alessandro Vespignani, and Amy Wesolowski. 2015. Enhancing Disease Surveillance with Novel Data Streams: Challenges and Opportunities. EPJ Data Science 4, 1 (Dec. 2015), 1–8. https://doi.org/10.1140/epjds/s13688-015-0054-0 [4] Pierpaolo Andriani and Bill McKelvey. 2009. From Gaussian to Paretian Thinking: Causes and Implications of Power Laws in Organizations. Organization Science 20, 6 (Oct. 2009), 1053–1071. https://doi.org/10.1287/orsc.1090. 0481 [5] Ofer Arazy, Johannes Daxenberger, Hila Lifshitz-Assaf, Oded Nov, and Iryna Gurevych. 2016. Turbulent Stability of Emergent Roles: The Dualistic Nature of Self-Organizing Knowledge Coproduction. Information Systems Research 27, 4 (2016), 792–812. https://doi.org/10.1287/isre.2016.0647 [6] René M. Bakker. 2010. Taking Stock of Temporary Organizational Forms: A Systematic Review and Research Agenda. International Journal of Management Reviews 12, 4 (2010), 466–486. https://doi.org/10.1111/j.1468- 2370.2010.00281.x [7] Rene M. Bakker, Robert J. DeFillippi, Andreas Schwab, and Jörg Sydow. 2016. Temporary Organizing: Promises, Processes, Problems. Organization Studies (Sept. 2016). https://doi.org/10.1177/0170840616655982 [8] Patti Bao, Brent Hecht, Samuel Carton, Mahmood Quaderi, Michael Horn, and Darren Gergle. 2012. Omnipedia: Bridging the Wikipedia Language Gap. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ’12). ACM, New York, NY, USA, 1075–1084. https://doi.org/10.1145/2207676.2208553 [9] Samuel Barbosa, Dan Cosley, Amit Sharma, and Roberto M. Cesar. 2016. Averaging Gone Wrong: Using Time-Aware Analyses to Better Understand Behavior. In Proceedings of the 25th International Conference on World Wide Web (WWW ’16). International World Wide Web Conferences Steering Committee, Montréal, Québec, Canada, 829–841. https://doi.org/10.1145/2872427.2883083 [10] Batuhan Bardak and Mehmet Tan. 2015. Prediction of Influenza Outbreaks by Integrating Wikipedia Article Access Logs and Google Flu Trend Data. In 2015 IEEE 15th International Conference on Bioinformatics and Bioengineering (BIBE). 1–6. https://doi.org/10.1109/BIBE.2015.7367640 [11] Frank J. Barrett. 1998. Creativity and Improvisation in Jazz and Organizations: Implications for Organizational Learning. Organization Science 9, 5 (1998), 605–622. https://doi.org/10.1287/orsc.9.5.605 [12] B.A. Bechky. 2006. Gaffers, Gofers, and Grips: Role-Based Coordination in Temporary Organizations. Organization Science 17 (2006), 3–21. [13] B.A. Bechky and G.A. Okhuysen. 2011. Expecting the Unexpected? How Swat Officers and Film Crews Handle Surprises. The Academy of Management Journal (AMJ) 54 (2011), 239–261. [14] Shiko M. Ben-Menahem, Georg Von Krogh, Zeynep Erden, and Andreas Schneider. 2016. Coordinating Knowledge Creation in Multidisciplinary Teams: Evidence from Early-Stage Drug Discovery. Academy of Management Journal 59, 4 (Aug. 2016), 1308–1338. https://doi.org/10.5465/amj.2013.1214 [15] D Berkowitz. 1992. Non-Routine News and Newswork: Exploring a What-a-Story. Journal of Communication 42 (1992), 82–94. [16] Gregory A. Bigley and Karlene H. Roberts. 2001. The Incident Command System: High-Reliability Organizing for Complex and Volatile Task Environments. The Academy of Management Journal 44 (2001), 1281–1299. [17] Colin R. Blyth. 1972. On Simpson’s Paradox and the Sure-Thing Principle. J. Amer. Statist. Assoc. 67, 338 (June 1972), 364–366. https://doi.org/10.1080/01621459.1972.10482387 [18] Steve P. Borgatti and Martin G. Everett. 1997. Network Analysis of 2-Mode Data. Social Networks 19 (1997), 243–269. 3. [19] Sarah Boxer. 2004. Mudslinging Weasels Into Online History. The New York Times (Nov. 2004). [20] danah boyd and Kate Crawford. 2012. Critical Questions for Big Data. Information, Communication & Society 15, 5 (June 2012), 662–679. https://doi.org/10.1080/1369118X.2012.678878 [21] Anna D. Broido and Clauset Aaron. 2019. Scale-Free Networks Are Rare. Nature Communications 10, 1 (Dec. 2019). https://doi.org/10.1038/s41467-019-08746-5

[22] Susan L. Bryant, Andrea Forte, and Amy Bruckman. 2005. Becoming Wikipedian: Transformation of Participation in a Collaborative Online Encyclopedia. In Proceedings of the 2005 International ACM SIGGROUP Conference on Supporting Group Work (GROUP ’05). ACM, New York, NY, USA, 1–10. https://doi.org/10.1145/1099203.1099205 [23] Catriona M. Burke and Michael J. Morley. 2016. On Temporary Organizations: A Review, Synthesis and Research Agenda. Human Relations (March 2016). https://doi.org/10.1177/0018726715610809 [24] JA Cannon-Bowers, E Salas, and S Converse. 1993. Shared Mental Models in Expert Team Decision Making. In Environmental Effects on Cognitive Abilities, Jr. N. John Castellan (Ed.). Lawrence Erlbaum, 221–246. [25] Aaron Clauset, Cosma Shalizi, and Mark Newman. 2009. Power-Law Distributions in Empirical Data. SIAM Rev. 51, 4 (Nov. 2009), 661–703. https://doi.org/10.1137/070710111 [26] Camille Cobb, Ted McCarthy, Annuska Perkins, Ankitha Bharadwaj, Jared Comis, Brian Do, and Kate Starbird. 2014. Designing for the Deluge: Understanding & Supporting the Distributed, Collaborative Work of Crisis Volunteers. In Proceedings of the 17th ACM Conference on Computer Supported Cooperative Work & Social Computing (CSCW ’14). Association for Computing Machinery, Baltimore, Maryland, USA, 888–899. https://doi.org/10.1145/2531602. 2531712 [27] Noam Cohen. 2007. The Latest on Virginia Tech, From Wikipedia. The New York Times (2007). [28] Noam Cohen. 2008. Don’t Like Palin’s Wikipedia Story? Change It. The New York Times (Aug. 2008). [29] Giovanni Colavizza. 2020. COVID-19 Research in Wikipedia. bioRxiv (May 2020), 2020.05.10.087643. https: //doi.org/10.1101/2020.05.10.087643 [30] Tiago Cunha, David Jurgens, Chenhao Tan, and Daniel Romero. 2019. Are All Successful Communities Alike? Characterizing and Predicting the Success of Online Communities. In The World Wide Web Conference (WWW ’19). Association for Computing Machinery, San Francisco, CA, USA, 318–328. https://doi.org/10.1145/3308558. 3313689 [31] Jonathan Dee. 2007. All the News That’s Fit to Print Out. The New York Times (July 2007). [32] Sidney W. A. Dekker and David D. Woods. 2010. The High Reliability Organization Perspective. In Human Factors in Aviation (Second Edition), Eduardo Salas and Dan Maurino (Eds.). Academic Press, San Diego, 123–143. https: //doi.org/10.1016/B978-0-12-374518-7.00005-5 [33] Peter Denning, Jim Horning, David Parnas, and Lauren Weinstein. 2005. Wikipedia Risks. Commun. ACM 48, 12 (Dec. 2005), 152. https://doi.org/10.1145/1101779.1101804 [34] Dimitar Dimitrov, Philipp Singer, Florian Lemmerich, and Markus Strohmaier. 2017. What Makes a Link Successful on Wikipedia?. In Proceedings of the 26th International Conference on World Wide Web (WWW ’17). International World Wide Web Conferences Steering Committee, Perth, Australia, 917–926. https://doi.org/10.1145/3038912. 3052613 [35] Thomas E. Drabek and David A. McEntire. 2003. Emergent Phenomena and the Sociology of Disaster: Lessons, Trends and Opportunities from the Research Literature. Disaster Prevention and Management: An International Journal 12, 2 (Jan. 2003), 97–112. https://doi.org/10.1108/09653560310474214 [36] Eric M. Eisenberg. 1990. Jamming: Transcendence Through Organizing. Communication Research 17, 2 (1990), 139–164. https://doi.org/10.1177/009365090017002001 [37] Adam Clark Estes. 2011. The Wikipedia War of Paul Revere and Sarah Palin. The Atlantic (June 2011). [38] Etter, Michael Andreas and Finn Årup Nielsen. 2015. Collective Remembering of Organizations: Co-Construction of Organizational Pasts in Wikipedianull. Corporate Communications: An International Journal (Aug. 2015). https: //doi.org/10.1108/CCIJ-09-2014-0059 [39] Geoffrey Fairchild, Lalindra De Silva, Sara Y. Del Valle, and Alberto M. Segre. 2015. Eliciting Disease Datafrom Wikipedia Articles. In Ninth International AAAI Conference on Web and Social Media. [40] Samer Faraj, Sirkka L. Jarvenpaa, and Ann Majchrzak. 2011. Knowledge Collaboration in Online Communities. Organization Science 22, 5 (Feb. 2011), 1224–1239. https://doi.org/10.1287/orsc.1100.0614 [41] Samer Faraj and Steven L. Johnson. 2010. Network Exchange Patterns in Online Communities. Organization Science 22, 6 (2010), 1464–1480. https://doi.org/10.1287/orsc.1100.0600 [42] Samer Faraj and Lee Sproull. 2000. Coordinating Expertise in Software Development Teams. Management Science 46, 12 (2000), 1554–1568. https://doi.org/10.1287/mnsc.46.12.1554.12072 [43] Samer Faraj and Yan Xiao. 2006. Coordination in Fast-Response Organizations. Management Science 52, 8 (Aug. 2006), 1155–1169. https://doi.org/10.1287/mnsc.1060.0526 [44] M. Ferron and P. Massa. 2011. Collective Memory Building in Wikipedia: The Case of North African Uprisings. In 7th International Symposium on Wikis and Open Collaboration. ACM, 114–123. [45] M. Ferron and P. Massa. 2011. WikiRevolutions: Wikipedia as a Lens for Studying the Real-Time Formation of Collective Memories of Revolutions. International Journal of Communication (2011). [46] Ferron, Michela and Massa, Paolo. 2014. Beyond the Encyclopedia: Collective Memories in Wikipedia. Memory Studies 7, 1 (Jan. 2014), 22–45. https://doi.org/10.1177/1750698013490590

Proc. ACM Hum.-Comput. Interact., Vol. 3, No. CSCW, Article TBD. Publication date: November 2020. TBD:32 Brian C. Keegan & Chenhao Tan

[47] Andrew Flowers and Carl Bialik. 2016. The Most-Edited Wikipedia Pages Over The Last 15 Years. [48] Heather Ford. 2015. Infoboxes and Cleanup Tags: Artifacts of Wikipedia Newsmaking. Journalism: Theory, Practice & Criticism 16, 1 (Jan. 2015), 79–98. https://doi.org/10.1177/1464884914545739 [49] Deen Freelon. 2018. Computational Research in the Post-API Age. Political Communication 35, 4 (2018), 665–668. [50] Ruth García-Gavilanes, Anders Mollgaard, Milena Tsvetkova, and Taha Yasseri. 2017. The Memory Remains: Understanding Collective Memory in the Digital Age. Science Advances 3, 4 (April 2017), e1602368. https: //doi.org/10.1126/sciadv.1602368 [51] R. Stuart Geiger and Aaron Halfaker. 2013. Using Edit Sessions to Measure Participation in Wikipedia. In Proceedings of the 2013 Conference on Computer Supported Cooperative Work (San Antonio, Texas, USA) (CSCW ’13). Association for Computing Machinery, New York, NY, USA, 861–870. https://doi.org/10.1145/2441776.2441873 [52] R. Stuart Geiger and Aaron Halfaker. 2013. When the Levee Breaks: Without Bots, What Happens to Wikipedia’s Quality Control Processes?. In Proceedings of the 9th International Symposium on Open Collaboration (WikiSym ’13). ACM, New York, NY, USA, 6:1–6:6. https://doi.org/10.1145/2491055.2491061 [53] R. Stuart Geiger and David Ribes. 2011. Trace Ethnography: Following Coordination through Documentary Practices. In 2011 44th Hawaii International Conference on System Sciences. 1–10. https://doi.org/10.1109/HICSS.2011.455 [54] Stefan Geiß, Melanie Leidecker, and Thomas Roessing. 2016. The Interplay between Media-for-Monitoring and Media-for-Searching: How News Media Trigger Searches and Edits in Wikipedia. New Media & Society 18, 11 (Dec. 2016), 2740–2759. https://doi.org/10.1177/1461444815600281 [55] Nicholas Generous, Geoffrey Fairchild, Alina Deshpande, Sara Y. Del Valle, and Reid Priedhorsky. 2014. Global Disease Monitoring and Forecasting with Wikipedia. PLOS Computational Biology 10, 11 (Nov. 2014), e1003892. https://doi.org/10.1371/journal.pcbi.1003892 [56] Jeremy Ginsberg, Matthew H. Mohebbi, Rajan S. Patel, Lynnette Brammer, Mark S. Smolinski, and Larry Brilliant. 2009. Detecting Influenza Epidemics Using Search Engine Query Data. Nature 457, 7232 (Feb. 2009), 1012–4. [57] Sascha Göbel and Simon Munzert. 2018. Political Advertising on the Wikipedia Marketplace of Information. Social Science Computer Review 36, 2 (April 2018), 157–175. https://doi.org/10.1177/0894439317703579 [58] Scott A. Golder and Michael W. Macy. 2011. Diurnal and Seasonal Mood Vary with Work, Sleep, and Daylength Across Diverse Cultures. Science 333, 6051 (Sept. 2011), 1878–1881. https://doi.org/10.1126/science.1202775 [59] Katie Hafner. 2007. Seeing Corporate Fingerprints in Wikipedia Edits. The New York Times (Aug. 2007). [60] Aric A. Hagberg, Daniel A. Schult, and Pieter J. Swart. 2008. Exploring Network Structure, Dynamics, and Function Using NetworkX. In SciPy’08. [61] Stephen Harrison. 2020. The Coronavirus Is Stress-Testing Wikipedia’s Systems—and Editors. Slate Magazine (March 2020). [62] Stephen Harrison. 2020. Future Historians Will Need Access to Coronavirus Misinformation. https://slate.com/technology/2020/05/wikipedia-coronavirus-information-future-historians.html. [63] Stephen Harrison. 2020. How Wikipedia Fought Back Against a Ban in Turkey. Slate Magazine (Jan. 2020). [64] Brent Hecht and Darren Gergle. 2010. The Tower of Babel Meets Web 2.0: User-Generated Content and Its Applications in a Multilingual Context. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ’10). ACM, New York, NY, USA, 291–300. https://doi.org/10.1145/1753326.1753370 [65] James M. Heilman, Eckhard Kemmann, Michael Bonert, Anwesh Chatterjee, Brent Ragar, Graham M. Beards, David J. Iberri, Matthew Harvey, Brendan Thomas, Wouter Stomp, Michael F. Martone, Daniel J. Lodge, Andrea Vondracek, Jacob F. de Wolff, Casimir Liber, Samir C. Grover, Tim J. Vickers, Bertalan Meskó, and Michaël R. Laurent. 2011. Wikipedia: A Key Tool for Global Public Health Promotion. Journal of Medical Internet Research 13, 1 (2011), e14. https://doi.org/10.2196/jmir.1589 [66] James M. Heilman and Andrew G. West. 2015. Wikipedia and Medicine: Quantifying Readership, Editors, and the Significance of Natural Language. Journal of Medical Internet Research 17, 3 (2015), e62. https://doi.org/10.2196/ jmir.4069 [67] James M Heilman, Jacob De Wolff, Graham M Beards, and Brian J Basden. 2014. Dengue Fever: A Wikipedia Clinical Review. Open Medicine 8, 4 (Oct. 2014), e105–e115. [68] Kyle S. Hickmann, Geoffrey Fairchild, Reid Priedhorsky, Nicholas Generous, James M. Hyman, Alina Deshpande, and Sara Y. Del Valle. 2015. Forecasting the 2013–2014 Influenza Season Using Wikipedia. PLOS Computational Biology 11, 5 (May 2015), e1004239. https://doi.org/10.1371/journal.pcbi.1004239 [69] Benjamin Mako Hill and Aaron Shaw. 2014. Consider the Redirect: A Missing Dimension of Wikipedia Research. In Proceedings of The International Symposium on Open Collaboration (OpenSym ’14). Association for Computing Machinery, Berlin, Germany, 1–4. https://doi.org/10.1145/2641580.2641616 [70] Benjamin Mako Hill and Aaron Shaw. 2015. Page Protection: Another Missing Dimension of Wikipedia Research. In Proceedings of the 11th International Symposium on Open Collaboration (OpenSym ’15). Association for Computing Machinery, San Francisco, California, 1–4. https://doi.org/10.1145/2788993.2789846

[71] James Howison, Andrea Wiggins, and Kevin Crowston. 2011. Validity Issues in the Use of Social Network Analysis with Digital Trace Data. Journal of the Association for Information Systems; Atlanta 12, 12 (Dec. 2011), 767–797. [72] Sirkka L. Jarvenpaa and Ann Majchrzak. 2010. Vigilant Interaction in Knowledge Collaboration: Challenges of Online User Participation Under Ambivalence. Information Systems Research 21, 4 (2010), 773–784. [73] Dariusz Jemielniak. 2014. Common Knowledge?: An Ethnography of Wikipedia. Stanford University Press. [74] Steven L. Johnson, Samer Faraj, and Srinivas Kudaravalli. 2014. Emergence of Power Laws in Online Communities: The Role of Social Mechanisms and Preferential Attachment. MIS Quarterly 38, 3 (Sept. 2014), 795–A13. [75] Gerald C. Kane and Sam Ransbotham. 2016. Content as Community Regulator: The Recursive Relationship Between Consumption and Contribution in Open Collaboration Communities. Organization Science (Sept. 2016). https: //doi.org/10.1287/orsc.2016.1075 [76] Nattiya Kanhabua, Tu Ngoc Nguyen, and Claudia Niederée. 2014. What Triggers Human Remembering of Events? A Large-Scale Analysis of Catalysts for Collective Memory in Wikipedia. In IEEE/ACM Joint Conference on Digital Libraries. 341–350. https://doi.org/10.1109/JCDL.2014.6970189 [77] Brian Keegan. 2015. Emergent Social Roles in Wikipedia’s Breaking News Collaborations. In Roles, Trust, and Reputation in Social Media Knowledge Markets. Springer, 57–79. [78] Brian Keegan, Darren Gergle, and Noshir Contractor. 2011. Hot off the Wiki: Dynamics, Practices, and Structures in Wikipedia’s Coverage of the Tohoku¯ Catastrophes. In Proceedings of the 7th International Symposium on Wikis and Open Collaboration (WikiSym ’11). ACM, New York, NY, USA, 105–113. https://doi.org/10.1145/2038558.2038577 [79] Brian Keegan, Darren Gergle, and Noshir Contractor. 2012. Staying in the Loop: Structure and Dynamics of Wikipedia’s Breaking News Collaborations. In Proceedings of the Eighth Annual International Symposium on Wikis and Open Collaboration (WikiSym ’12). ACM, New York, NY, USA, 1:1–1:10. https://doi.org/10.1145/2462932.2462934 [80] Brian Keegan, Darren Gergle, and Noshir Contractor. 2013. Hot Off the Wiki Structures and Dynamics of Wikipedia’s Coverage of Breaking News Events. American Behavioral Scientist 57, 5 (May 2013), 595–622. https://doi.org/10. 1177/0002764212469367 [81] Brian C. Keegan. 2013. A History of Newswork on Wikipedia. In Proceedings of the 9th International Symposium on Open Collaboration (WikiSym ’13). Association for Computing Machinery, Hong Kong, China, 1–10. https: //doi.org/10.1145/2491055.2491062 [82] Brian C. Keegan. 2019. The Dynamics of Peer-Produced Political Information During the 2016 U.S. Presidential Campaign. Proceedings of the ACM on Human-Computer Interaction 3, CSCW (Nov. 2019), 33:1–33:20. https: //doi.org/10.1145/3359135 [83] Brian C. Keegan and Jed R. Brubaker. 2015. ’Is’ to ’Was’: Coordination and Commemoration in Posthumous Activity on Wikipedia Biographies. In Proceedings of the 18th ACM Conference on Computer Supported Cooperative Work & Social Computing (CSCW ’15). Association for Computing Machinery, Vancouver, BC, Canada, 533–546. https: //doi.org/10.1145/2675133.2675238 [84] Katherine C. Kellogg, Wanda J. Orlikowski, and JoAnne Yates. 2006. Life in the Trading Zone: Structuring Coordination across Boundaries in Postbureaucratic Organizations. Organization Science 17, 1 (2006), 22–44. [85] Katherine J. Klein, Jonathan C. Ziegert, Andrew P. Knight, and Yan Xiao. 2006. Dynamic Delegation: Shared, Hierarchical, and Deindividualized Leadership in Extreme Action Teams. Administrative Science Quarterly 51, 4 (2006), 590–621. https://doi.org/10.2189/asqu.51.4.590 [86] Robert E. Kraut, Paul Resnick, Sara Kiesler, Moira Burke, Yan Chen, Niki Kittur, Joseph Konstan, Yuqing Ren, and John Riedl. 2012. Building Successful Online Communities: Evidence-Based Social Design. MIT Press. [87] Michael E Kummer. 2018. Attention in the Peer Production of User Generated Content-Evidence from 93 Pseudo- Experiments on Wikipedia. Available at SSRN 3431249 (2018). [88] Daniel Lamprecht, Kristina Lerman, Denis Helic, and Markus Strohmaier. 2017. How the Structure of Wikipedia Articles Influences User Navigation. New Review of Hypermedia and Multimedia 23, 1 (Jan. 2017), 29–50. https: //doi.org/10.1080/13614568.2016.1179798 [89] Matthieu Latapy, Clémence Magnien, and Nathalie Del Vecchio. 2008. Basic Notions for the Analysis of Large Two-Mode Networks. Social Networks 30 (2008), 31–48. https://doi.org/10.1016/j.socnet.2007.04.006 1. [90] Michaël R. Laurent and Tim J. Vickers. 2009. Seeking Health Information Online: Does Wikipedia Matter? Journal of the American Medical Informatics Association 16, 4 (July 2009), 471–479. https://doi.org/10.1197/jamia.M3059 [91] David Lazer, Ryan Kennedy, Gary King, and Alessandro Vespignani. 2014. The Parable of Google Flu: Traps in Big Data Analysis. Science 343, 6176 (March 2014), 1203–1205. https://doi.org/10.1126/science.1248506 [92] Kristina Lerman. 2018. Computational Social Scientist Beware: Simpson’s Paradox in Behavioral Data. Journal of Computational Social Science 1, 1 (Jan. 2018), 49–58. https://doi.org/10.1007/s42001-017-0007-4 [93] Brendan Luyt. 2016. Wikipedia, Collective Memory, and the Vietnam War. Journal of the Association for Information Science and Technology 67, 8 (Aug. 2016), 1956–1961. https://doi.org/10.1002/asi.23518

Proc. ACM Hum.-Comput. Interact., Vol. 3, No. CSCW, Article TBD. Publication date: November 2020. TBD:34 Brian C. Keegan & Chenhao Tan

[94] Ann Majchrzak, Sirkka L. Jarvenpaa, and Andrea B. Hollingshead. 2007. Coordinating Expertise Among Emergent Groups Responding to Disasters. Organization Science 18, 1 (Feb. 2007), 147–161. https://doi.org/10.1287/orsc. 1060.0228 [95] Thomas W. Malone and Kevin Crowston. 1994. The Interdisciplinary Study of Coordination. ACM Comput. Surv. 26, 1 (March 1994), 87–119. https://doi.org/10.1145/174666.174668 [96] David J. McIver and John S. Brownstein. 2014. Wikipedia Usage Estimates Prevalence of Influenza-Like Illness in the United States in Near Real-Time. PLOS Computational Biology 10, 4 (April 2014), e1003581. https://doi.org/10. 1371/journal.pcbi.1003581 [97] Marcus Messner and Jeff South. 2011. Legitimizing Wikipedia: How US National Newspapers Frame and Usethe Online Encyclopedia in Their Coverage. Journalism Practice 5 (2011), 145–160. [98] D Meyerson, KE Weick, and RM Kramer. 1996. Swift Trust and Temporary Groups. Trust in organizations: Frontiers of theory and research 166 (1996), 195. [99] Staša Milojević. 2010. Modes of Collaboration in Modern Science: Beyond Power Laws and Preferential Attachment. Journal of the American Society for Information Science and Technology 61, 7 (2010), 1410–1423. [100] Staša Milojević. 2010. Power law distributions in information science: Making the case for logarithmic binning. Journal of the American Society for Information Science and Technology 61, 12 (2010), 2417–2425. https://doi.org/ 10.1002/asi.21426 [101] Alan Mislove, Massimiliano Marcon, Krishna P. Gummadi, Peter Druschel, and Bobby Bhattacharjee. 2007. Mea- surement and Analysis of Online Social Networks. In Proceedings of the 7th ACM SIGCOMM Conference on Internet Measurement (San Diego, California, USA) (IMC ’07). Association for Computing Machinery, New York, NY, USA, 29–42. https://doi.org/10.1145/1298306.1298311 [102] Michael Mitzenmacher. 2004. A Brief History of Generative Models for Power Law and Lognormal Distributions. Internet Mathematics 1, 2 (2004), 226–251. https://doi.org/10.1080/15427951.2004.10129088 [103] D. Montgomery. 2011. Minutes after Virginia Earthquake, It Was on Wikipedia. The Washington Post (2011). [104] Sabine Niederer and José van Dijck. 2010. Wisdom of the Crowd or Technicity of Content? Wikipedia as a Sociotech- nical System. New Media & Society 12, 8 (Dec. 2010), 1368–1387. https://doi.org/10.1177/1461444810365297 [105] Wendy Norris, Amy Voida, Leysia Palen, and Stephen Voida. 2019. ’Is the Time Right Now?’: Reconciling Sociotemporal Disorder in Distributed Team Work. Proceedings of the ACM on Human-Computer Interaction 3, CSCW (Nov. 2019), 98:1–98:29. https://doi.org/10.1145/3359200 [106] Gerardo A. Okhuysen and Beth A. Bechky. 2009. Coordination in Organizations: An Integrative Perspective. The Academy of Management Annals 3, 1 (Jan. 2009), 463–502. https://doi.org/10.1080/19416520903047533 [107] Tore Opsahl. 2013. Triadic Closure in Two-Mode Networks: Redefining the Global and Local Clustering Coefficients. Social Networks 35, 2 (May 2013), 159–167. https://doi.org/10.1016/j.socnet.2011.07.001 [108] Dave Osthus, Ashlynn R. Daughton, and Reid Priedhorsky. 2019. Even a Good Influenza Forecasting Model Can Benefit from Internet-Based Nowcasts, but Those Benefits Are Limited. PLOS Computational Biology 15, 2 (Feb. 2019), e1006599. https://doi.org/10.1371/journal.pcbi.1006599 [109] Leysia Palen and Kenneth M. Anderson. 2016. Crisis Informatics: New Data for Extraordinary Times. Science 353, 6296 (July 2016), 224–225. https://doi.org/10.1126/science.aag2579 [110] Leysia Palen and Sophia B. Liu. 2007. Citizen Communications in Crisis: Anticipating a Future of ICT-Supported Public Participation. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ’07). ACM, New York, NY, USA, 727–736. https://doi.org/10.1145/1240624.1240736 [111] Leysia Palen and Sarah Vieweg. 2008. The Emergence of Online Widescale Interaction in Unexpected Events: Assistance, Alliance & Retreat. In Proceedings of the 2008 ACM Conference on Computer Supported Cooperative Work (CSCW ’08). ACM, New York, NY, USA, 117–126. https://doi.org/10.1145/1460563.1460583 [112] Christian Pentzold. 2009. Fixing the Floating Gap: The Online Encyclopaedia Wikipedia as a Global Memory Place. Memory Studies 2, 2 (May 2009), 255–272. https://doi.org/10.1177/1750698008102055 [113] Christian Pentzold, Esther Weltevrede, Michele Mauri, David Laniado, Andreas Kaltenbrunner, and Erik Borra. 2017. Digging Wikipedia: The Online Encyclopedia as a Digital Cultural Heritage Gateway and Site. Journal on Computing and Cultural Heritage 10, 1 (April 2017), 1–19. https://doi.org/10.1145/3012285 [114] Jessamy Perriam, Andreas Birkbak, and Andy Freeman. 2019. Digital Methods in a Post-API Environment. International Journal of Social Research Methodology 0, 0 (2019), 1–14. [115] Emily Porter, P. M. Krafft, and Brian Keegan. 2020. Visual Narratives and Collective Memory across Peer-Produced Accounts of Contested Sociopolitical Events. ACM Transactions on Social Computing 3, 1 (Feb. 2020), 4:1–4:20. https://doi.org/10.1145/3373147 [116] Reid Priedhorsky, Dave Osthus, Ashlynn R. Daughton, Kelly R. Moran, Nicholas Generous, Geoffrey Fairchild, Alina Deshpande, and Sara Y. Del Valle. 2017. Measuring Global Disease with Wikipedia: Success, Failure, and a Research Agenda. In Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing

(CSCW ’17). Association for Computing Machinery, Portland, Oregon, USA, 1812–1834. https://doi.org/10.1145/ 2998181.2998183 [117] Farah Qaiser. 2020. Like Zika, The Public Is Heading To Wikipedia During The COVID-19 Coronavirus Pandemic. Forbes (March 2020). [118] E. L. Quarantelli and Russell R. Dynes. 1977. Response to Social Crisis and Disaster. Annual Review of Sociology 3 (1977), 23–49. [119] Sam Ransbotham, Gerald C. Kane, and Nicholas H. Lurie. 2012. Network Characteristics and the Value of Collaborative User-Generated Content. Marketing Science 31, 3 (May 2012), 387–405. https://doi.org/10.1287/mksc.1110.0684 [120] Joseph Michael Reagle. 2010. Good Faith Collaboration: The Culture of Wikipedia. MIT Press. [121] Manoel Horta Ribeiro, Kristina Gligorić, Maxime Peyrard, Florian Lemmerich, Markus Strohmaier, and Robert West. 2020. Sudden Attention Shifts on Wikipedia Following COVID-19 Mobility Restrictions. arXiv:2005.08505 [cs] (May 2020). [122] Ramón Rico, Miriam Sánchez-Manzanares, Francisco Gil, and Cristina Gibson. 2008. Team Implicit Coordination Processes: A Team Knowledge-Based Approach. The Academy of Management Review 33, 1 (2008), 163–184. https: //doi.org/10.2307/20159381 [123] Marcel Salathé, Linus Bengtsson, Todd J. Bodnar, Devon D. Brewer, John S. Brownstein, Caroline Buckee, Ellsworth M. Campbell, Ciro Cattuto, Shashank Khandelwal, Patricia L. Mabry, and Alessandro Vespignani. 2012. Digital Epidemi- ology. PLoS Computational Biology 8, 7 (July 2012), e1002616. https://doi.org/10.1371/journal.pcbi.1002616 [124] Mauricio Santillana, André T. Nguyen, Mark Dredze, Michael J. Paul, Elaine O. Nsoesie, and John S. Brownstein. 2015. Combining Search, Social Media, and Traditional Data Sources to Improve Influenza Surveillance. PLoS Computational Biology 11, 10 (Oct. 2015). https://doi.org/10.1371/journal.pcbi.1004513 [125] Thomas Shafee, Gwinyai Masukume, Lisa Kipersztok, Diptanshu Das, Mikael Häggström, and James Heilman. 2017. Evolution of Wikipedia’s Medical Content: Past, Present and Future. Journal of Epidemiology and Community Health (Aug. 2017), jech–2016–208601. https://doi.org/10.1136/jech-2016-208601 [126] Maha Shaikh and Emmanuelle Vaast. 2016. Folding and Unfolding: Balancing Openness and Transparency in Open Source Communities. Information Systems Research 27, 4 (2016), 813–833. https://doi.org/10.1287/isre.2016. 0646 [127] J. Danielle Sharpe, Richard S. Hopkins, Robert L. Cook, and Catherine W. Striley. 2016. Evaluating Google, Twitter, and Wikipedia as Tools for Influenza Surveillance Using Bayesian Change Point Analysis: A Comparative Analysis. JMIR Public Health and Surveillance 2, 2 (2016), e161. https://doi.org/10.2196/publichealth.5901 [128] Irina Shklovski, Leysia Palen, and Jeannette Sutton. 2008. Finding Community through Information and Com- munication Technology in Disaster Response. In Proceedings of the 2008 ACM Conference on Computer Supported Cooperative Work (CSCW ’08). Association for Computing Machinery, San Diego, CA, USA, 127–136. https: //doi.org/10.1145/1460563.1460584 [129] Rachel Siegel. 05\15\2019. Search Result Not Found: China Bans Wikipedia in All Languages. Washington Post (05\15\2019). [130] Denise A. Smith. 2020. Situating Wikipedia as a Health Information Resource in Various Contexts: A Scoping Review. PLOS ONE 15, 2 (Feb. 2020), e0228786. https://doi.org/10.1371/journal.pone.0228786 [131] Robert Soden and Leysia Palen. 2016. Infrastructure in the Wild: What Mapping in Post-Earthquake Nepal Reveals about Infrastructural Emergence. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI ’16). Association for Computing Machinery, San Jose, California, USA, 2796–2807. https://doi.org/10.1145/ 2858036.2858545 [132] Rebecca Solnit. 2009. A Paradise Built in Hell: The Extraordinary Communities That Arise in Disaster. Viking Penguin, New York. [133] Robert A. Stallings and E. L. Quarantelli. 1985. Emergent Citizen Groups and Emergency Management. Public Administration Review 45, Special (Jan. 1985), 93–100. https://doi.org/10.2307/3135003 [134] Kathleen M. Sutcliffe, Timothy J. Vogus, and Erik Dane. 2016. Mindfulness in Organizations: A Cross-Level Review. Annual Review of Organizational Psychology and Organizational Behavior 3, 1 (March 2016), 55–81. https://doi. org/10.1146/annurev-orgpsych-041015-062531 [135] Reham Al Tamime, Wendy Hall, and Richard Giordano. 2019. Uncertainty During New Disease Outbreaks in Wikipedia. Proceedings of the International AAAI Conference on Web and Social Media 13 (July 2019), 38–46. [136] Chenhao Tan. 2018. Tracing Community Genealogy: How New Communities Emerge from the Old. In Twelfth International AAAI Conference on Web and Social Media. [137] Yla Tausczik, Kate Faasse, James W. Pennebaker, and Keith J. Petrie. 2012. Public Anxiety and Information Seeking Following the H1N1 Outbreak: Blogs, Newspaper Articles, and Wikipedia Visits. Health Communication 27, 2 (Feb. 2012), 179–185. https://doi.org/10.1080/10410236.2011.571759

Proc. ACM Hum.-Comput. Interact., Vol. 3, No. CSCW, Article TBD. Publication date: November 2020. TBD:36 Brian C. Keegan & Chenhao Tan

[138] Jennifer Thom, Dan Cosley, and Geri Gay. 2010. What Do You Know?: Experts, Novices and Territoriality in Collaborative Systems. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ’10). ACM, New York, NY, USA, 1685–1694. https://doi.org/10.1145/1753326.1753578 [139] Jennifer Thom, Dan R. Cosley, and Geri Gay. 2009. What’s Mine Is Mine: Territoriality in Collaborative Authoring. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ’09). ACM, New York, NY, USA, 1481–1484. https://doi.org/10.1145/1518701.1518925 [140] Michele Tizzoni, André Panisson, Daniela Paolotti, and Ciro Cattuto. 2020. The Impact of News Exposure on Collective Attention in the United States during the 2016 Zika Epidemic. PLOS Computational Biology 16, 3 (March 2020), e1007633. https://doi.org/10.1371/journal.pcbi.1007633 [141] Marlon Twyman, Brian C. Keegan, and Aaron Shaw. 2017. Black Lives Matter in Wikipedia: Collective Memory and Collaboration around Online Social Movements. In Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing (CSCW ’17). Association for Computing Machinery, Portland, Oregon, USA, 1400–1412. https://doi.org/10.1145/2998181.2998232 [142] Melissa A. Valentine and Amy C. Edmondson. 2014. Team Scaffolds: How Mesolevel Structures Enable Role-Based Coordination in Temporary Groups. Organization Science (2014). https://doi.org/10.1287/orsc.2014.0947 [143] Andrew H. Van De Ven, Andre L. Delbecq, and Richard Koenig. 1976. Determinants of Coordination Modes within Organizations. American Sociological Review 41, 2 (1976), 322–338. https://doi.org/10.2307/2094477 [144] Yogesh Virkar and Aaron Clauset. 2014. Power-Law Distributions in Binned Empirical Data. The Annals of Applied Statistics 8, 1 (2014), 89–119. [145] KE Weick and KM. Sutcliffe. 2007. Managing the Unexpected: Resilient Performance in an Age of Uncertainty. Jossey-Bass Inc Pub. [146] Karl E. Weick and Karlene H. Roberts. 1993. Collective Mind in Organizations: Heedful Interrelating on Flight Decks. Administrative Science Quarterly 38, 3 (Sept. 1993), 357. [147] Karl E. Weick, Kathleen M. Sutcliffe, and David Obstfeld. 1999. Organizing for High Reliability: Processes of Collective Mindfulness. In Research in Organizational Behavior, Vol. 21, R. I. Sutton and B. M. Staw (Eds.). Elsevier Science/JAI Press, US, 81–123. [148] Robert West, Ashwin Paranjape, and Jure Leskovec. 2015. Mining Missing Hyperlinks from Human Navigation Traces: A Case Study of Wikipedia. In Proceedings of the 24th International Conference on World Wide Web (WWW ’15). International World Wide Web Conferences Steering Committee, Florence, Italy, 1242–1252. https://doi.org/10. 1145/2736277.2741666 [149] Gwen M. Wittenbaum, Garold Stasser, and Carol J. Merry. 1996. Tacit Coordination in Anticipation of Small Group Task Completion. Journal of Experimental Social Psychology 32, 2 (March 1996), 129–152. https://doi.org/10. 1006/jesp.1996.0006 [150] Taha Yasseri, Robert Sumi, and János Kertész. 2012. Circadian Patterns of Wikipedia Editorial Activity: A Demographic Analysis. PLoS One 7, 1 (Jan. 2012), e30091. https://doi.org/10.1371/journal.pone.0030091 [151] Dave Yates, Christian Wagner, and Ann Majchrzak. 2010. Factors Affecting Shapers of Organizational Wikis. Journal of the American Society for Information Science and Technology 61, 3 (2010), 543–554. https://doi.org/10.1002/ asi.21266 [152] Kai Zhu, Dylan Walker, and Lev Muchnik. 2018. Content Growth and Attention Contagion in Information Networks: A Natural Experiment on Wikipedia. Available at SSRN 3191128 (2018). [153] Christoph Zimmer, Sequoia I. Leuba, Reza Yaesoubi, and Ted Cohen. 2018. Use of Daily Internet Search Query Data Improves Real-Time Projections of Influenza Epidemics. Journal of The Royal Society Interface 15, 147 (Oct. 2018), 20180220. https://doi.org/10.1098/rsif.2018.0220

Received June 2020; revised June 2020; accepted TBD

Proc. ACM Hum.-Comput. Interact., Vol. 3, No. CSCW, Article TBD. Publication date: November 2020.