Supporting Volunteer Moderation Practices in Online Communities Joseph Seering CMU-HCII-20-107 September 2020 Human-Computer Interaction Institute School of Computer Science Carnegie Mellon University Pittsburgh, Pennsylvania 15213 [email protected] Thesis Committee: Geoff Kaufman (Chair) Human-Computer Interaction Institute Carnegie Mellon University Jason Hong Human-Computer Interaction Institute Carnegie Mellon University Bob Kraut Human-Computer Interaction Institute Carnegie Mellon University Michael Bernstein Department of Computer Science Stanford University Submitted in partial fulfillment of the requirements for the degree of Doctor of Philosophy Copyright © 2020 Joseph Seering. All rights reserved. KEYWORDS Content moderation, platforms, social computing, computer-supported cooperative work, social media, volunteer moderators, cooperative responsibility, Twitch, Reddit, Facebook, AI-mediated communication, social identity theory, rebukes, interpersonal moderation, governance, communities, online communities, self- governance, self-moderation, literature review, platforms and policies, commercial content moderation, human- centered design, metaphors, computational social science, thematic analysis, interviews, digital labor, hate speech, harassment, social networks. Abstract In this dissertation, I explore multiple levels of the content moderation ecosystem with a focus on platforms that rely extensively on volunteer user labor. These platforms, like Reddit, Twitch, and Facebook Groups, expect users to moderate their own communities, but reserve the right to intervene when communities or content therein violates sitewide standards for behavior. This thesis contains three parts. I begin with a high-level exploration of how platforms do and do not engage with volunteer community moderators. I build on the framework of cooperative responsibility to analyze the different ways platforms and users have found common ground on values, roles, and spaces for deliberation. Next, I focus in depth on the philosophies and mental models of the volunteer moderators, analyzing the metaphors they used both explicitly and implicitly to describe the work they do. Finally, I dive into the specifics of interpersonal language use in moderation, looking at how both interpersonal “rebukes” impact subsequent comment threads on Reddit and how changes of rules in communities on Reddit impact subsequent behavior of community members. For each of these linguistic pieces, I present results from a related experiment, in which I used a custom-built comment forum to test the impact of simulated “rebukes” and rules. This work shows the nuance of several core processes in user-driven moderation, ranging from the very high level organizational interactions to very low level linguistic features of user comments, and I argue that more attention toward understanding these processes in Computer-Supported Cooperative Work and related fields is needed. Contents 1 Introduction: The Content Moderation Ecosystem5 1.1 Prelude......................................5 1.2 Three Levels of Content Moderation......................7 1.3 Dissertation Outline...............................8 2 Foundations: Two Perspectives in Moderation Research 11 2.1 Research in Content Moderation......................... 11 2.2 Two Perspectives in Moderation Research................... 13 2.2.1 The Platforms and Policies Perspective................. 13 2.2.2 The Communities Perspective: Users’ Intra-group Moderation.... 20 3 Organizational Perspectives: Cooperative Responsibility and Content Moderation 41 3.1 Introduction.................................... 41 3.2 Cooperative Responsibility in Online Governance............... 43 3.3 Methods...................................... 45 3.4 Analysis...................................... 48 3.4.1 Reddit and Cooperative Responsibility................. 48 3.4.2 Twitch and Cooperative Responsibility................. 60 3.4.3 Facebook Groups and Cooperative Responsibility........... 69 3.5 Discussion and Implications........................... 75 3.5.1 Platforms................................. 75 1 3.5.2 Implications for Cooperative Responsibility Theory.......... 79 4 Metaphors for Moderation 83 4.1 Introduction.................................... 83 4.2 Prior work..................................... 86 4.2.1 Metaphors and social behaviors..................... 86 4.2.2 Moderation in online spaces....................... 88 4.3 Methods...................................... 91 4.4 Results....................................... 93 4.4.1 Nurturing and Supporting Communities................ 94 4.4.2 Overseeing and Facilitating Communities................ 95 4.4.3 Fighting for Communities........................ 97 4.4.4 Managing Communities......................... 99 4.4.5 Governing and Regulating Communities................ 101 4.4.6 Establishing Face Validity: Feedback from Interviewees........ 102 4.5 Threads for future research............................ 103 4.6 Threads for Design................................ 105 4.7 Conclusions.................................... 109 5 Linguistic Factors in Rebukes and Rules 115 5.1 Introduction.................................... 115 5.2 Related work................................... 117 5.3 Study 1: Impact of Rebukes on Reddit..................... 118 5.3.1 Data collection and model building................... 120 5.3.2 Results................................... 123 5.4 Study 2: Rebukes in a controlled comment thread............... 126 5.5 Study 3: Rules on Reddit............................ 129 5.5.1 Data collection and analysis....................... 130 5.6 Study 4: Rules in a controlled comment thread................ 138 5.7 Conclusions.................................... 141 2 6 Discussion and Directions for Future Work 145 6.1 A (Mildly) Radical Vision and Some Pitfalls.................. 147 6.1.1 Radical Visions.............................. 147 6.1.2 Pitfalls of community self-governance.................. 150 6.2 Guiding Questions for Future Research..................... 152 6.3 Concluding thoughts............................... 168 3 4 1 Introduction: The Content Moderation Ecosystem 1.1 Prelude1 In May of 1978, the “CommuniTree #1” online Bulletin Board System (BBS) launched in the San Francisco Bay area [6, pp. 88–92], [7]. Built from the CommuniTree Group’s idea to structure online conversation in threaded, tree-style structures based around core “conference” topics, it was the most successful entry into the very new space of online social spaces; while the first set of these virtual bulletin boards, developed in the mid-to-late 1970s, only displayed messages either in alphabetical order or in the order messages were posted [7], CommuniTree #1’s tree-style design allowed for conversations to move fluidly in multiple directions. The CommuniTree #1 platform and its subsequent iterations were infused with its creators’ philosophy of the grand power of social technology – the first discussion thread (called a “conference”) opened with the bold statement, “We are as gods and might as well get good at it”. The participants (mostly academics, researchers, and computing hobbyists) saw themselves “not primarily as readers of bulletin boards or participants in a novel discourse but as agents of a new kind of social experiment” [6, p. 90]2. In 1982, Apple entered into an 1A modified version of this chapter combined with parts of the second and sixth chapters hasbeen accepted to CSCW 2020 with Joseph Seering as solo author. 2Reflecting the “Digital Utopianism” [8] of this era of technologists, a technical manual written for CommuniTree by Dean Gengle was dedicated to “R. Buckminster ‘Bucky’ Fuller / The first global shaman of our species.” [3, p. iii] 5 agreement with the United States government to provide schools with Apple computers as a substitute for paying taxes, which caused a huge influx of teenage, mostly male users into virtual spaces previously reserved for the intellectual elite. Upon discovering CommuniTree, these students filled the board’s allotted disk space with “every word they could thinkof that meant shitting or fucking” [7], an onslaught for which existing users were completely unprepared. CommuniTree had been launched with minimal moderation tools; an “anti- censorship” philosophy was written directly into its code, with features that prevented system operators from proactively filtering messages as they came in, made it difficult toremove messages once they were entered, and granted any user access to commands that controlled the host computer, so the students’ incursions forced system operators to completely purge the system almost daily. Within a few months, CommuniTree was dead. The online, self-governing utopia that was CommuniTree lasted for less than half a decade. Relying almost entirely on the goodwill of its homogenous user-base, it had managed to survive and even thrive, but when confronted by a new set of users with different values and goals it collapsed. This may be one of the earliest major failures of online moderation documented in research literature, and, at least in hindsight, was a major blow to the dream that the internet could function simply as a ‘marketplace of ideas’ where better perspectives would naturally rise to the top. Today, popular media is full of examples of problematic behaviors, including extensive harassment leading users to delete their accounts,3 or adopt defensive behaviors [2,9]. A 2017 PEW study found that 4 in 10 Americans had