crafting online harassment and abuse reporting policies Amanda Levendowski Internet Law & Policy Foundry Fellow Data & Society Databite December 17, 2015 1 CRAFTING HARP what’s the problem? 2 CRAFTING HARP “We suck at dealing with abuse and trolls on the platform and we’ve sucked at it for years. It’s no secret and the rest of the world talks about it every day. We lose core user after core user by not addressing simple trolling issues that they face every day.” - Dick Costolo, CEO of Twitter 3 CRAFTING HARP 40% 4 CRAFTING HARP According to a 2014 survey, over 40% of online adults report being targets one or more types of harassment: called offensive names 7% purposefully embarassed 8% 27% stalked 6% 8% sexually harassed 22% physically threatened sustained harassment Maeve Duggan, Online Harassment, PEW Research Internet Project (Oct. 22, 2014), http://www.pewinternet.org/2014/10/22/online-harassment/ 5 CRAFTING HARP WHO 6 CRAFTING HARP There may be a correlation between the types of people who engage in harassment and where that behavior is likely to occur: co-worker 7% former partner 10% family 12% friend 23% acquaintance 24% unknown 26% stranger 38% 7 CRAFTING HARP WHERE 8 CRAFTING HARP There may be a correlation between the types of people who engage in harassment and where that behavior is likely to occur: online dating site 6% co-worker 7% forum 10% former partner 10% family 12% email 16% friend 23% online gaming 16% acquaintance 24% the comments 22% unknown 26% stranger 38% social media 66% 9 platform permeability matrix CRAFTING HARP OPEN social media comments chat services forums BY USER BY PLATFORM email online communities text messaging CLOSED 10 CRAFTING HARP HOW 11 taxonomy of harassment CRAFTING HARP dogpiling doxxing gaslighting hate speech jaqing off revenge porn sealioning surveillance swatting 12 taxonomy of harassment CRAFTING HARP dogpiling Looping unwanted third parties into an online conversation doxxing gaslighting hate speech jaqing off revenge porn sealioning surveillance swatting 13 taxonomy of harassment CRAFTING HARP dogpiling doxxing Exposing target’s personal, often sensitive, information gaslighting hate speech jaqing off revenge porn sealioning surveillance swatting 14 taxonomy of harassment CRAFTING HARP dogpiling doxxing Attempting to create doubt in and of gaslighting target’s memory or perception hate speech jaqing off revenge porn sealioning surveillance swatting 15 taxonomy of harassment CRAFTING HARP dogpiling doxxing gaslighting Degrading interactions based identity hate speech and/or characteristics jaqing off revenge porn sealioning surveillance swatting 16 taxonomy of harassment CRAFTING HARP dogpiling doxxing gaslighting hate speech Using leading questions to advance jaqing off accusations or harmful interactions revenge porn sealioning surveillance swatting 17 taxonomy of harassment CRAFTING HARP dogpiling doxxing gaslighting hate speech jaqing off Distributing sexually explicit images revenge porn without the consent of the pictured individual sealioning surveillance swatting 18 taxonomy of harassment CRAFTING HARP dogpiling doxxing gaslighting hate speech jaqing off revenge porn sealioning Popping into a conversation with endless questions or demands surveillance swatting 19 taxonomy of harassment CRAFTING HARP dogpiling doxxing gaslighting hate speech jaqing off revenge porn sealioning surveillance Using technology to track target’s online behavior swatting 20 taxonomy of harassment CRAFTING HARP dogpiling doxxing gaslighting hate speech jaqing off revenge porn sealioning surveillance Sending a SWAT team to target’s swatting home, work or other location 21 taxonomy of harassment CRAFTING HARP dogpiling doxxing gaslighting hate speech jaqing off revenge porn sealioning surveillance Sending a SWAT team to target’s swatting home, work or other location 22 CRAFTING HARP WHY 23 CRAFTING HARP conceptualizing solutions 24 CRAFTING HARP Strategies for combatting harassment and abuse in online communities can be conceptualized as two approaches: LEGAL AND REGULATORY: creating new laws, amending or repurposing existing laws, relying on agency intervention to govern communities 25 CRAFTING HARP Strategies for combatting harassment and abuse in online communities can be conceptualized as two approaches: LEGAL AND REGULATORY: creating new laws, amending or repurposing existing laws, relying on agency intervention to govern communities NORM AND RULE: retooling, revising, and reimagining existing community standards and community management 26 CRAFTING HARP Strategies for combatting harassment and abuse in online communities can be conceptualized as two approaches: LEGAL AND REGULATORY: creating new laws, amending or repurposing existing laws, relying on agency intervention to govern communities NORM AND RULE: retooling, revising, and reimagining existing community standards and community management 27 CRAFTING HARP Strategies for combatting harassment and abuse in online communities can be conceptualized as two approaches: LEGAL AND REGULATORY: creating new laws, amending or repurposing existing laws, relying on agency intervention to govern communities NORM AND RULE: retooling, revising, and reimagining existing community standards and community management 28 CRAFTING HARP norm and rule-based approach • (con)textual algorithims • community governance • censorship • filtration • moderation • proprietary management systems • pseudonymity or real-name requirements • settings customization 29 CRAFTING HARP strong community norms: the key to effective HARP 30 CRAFTING HARP To craft effective HARP, platforms must resist the temptation of the politician’s fallacy: 1. We must do something. 2. This is something. 3. Therefore, we must do this. 31 CRAFTING HARP NOPE 32 CRAFTING HARP Effective HARP feature carefully crafted community norms, which are reinforced by platforms that: INVEST in developing community expectations 33 CRAFTING HARP Effective HARP feature carefully crafted community norms, which are reinforced by platforms that: INVEST in developing community expectations EMPOWER users to shape interactions 34 CRAFTING HARP Effective HARP feature carefully crafted community norms, which are reinforced by platforms that: INVEST in developing community expectations EMPOWER users to shape interactions ENFORCE against disrupters and violators 35 CRAFTING HARP case in point: league of legends 36 CRAFTING HARP “As we spend more and more of our time online, we need to acknowledge that online harassment and toxicity is not an impossible problem, and that it is a problem worth spending time on.” - Jeffrey Lin, Riot Games’ lead designer of social systems Jeffrey Lin, “Doing Something About the ‘Impossible Problem’ of Abuse in Online Games,” Re/Code (July 7, 2015) (op-ed), http://recode.net/2015/07/07/doing-something-about-the-impossible-problem-of-abuse-in-online-games/ 37 diagnosing the community CRAFTING HARP Riot Games began by asking: “Who is responsible for bad behavior in our community?” 13% persistently negative players generally inoffensive players 87% 38 adopting norms and rules CRAFTING HARP • (con)textual algorithims • community governance • proprietary management systems • settings customization 39 adopting norms and rules CRAFTING HARP • (con)textual algorithims • community governance • proprietary management systems • settings customization relying on a tribunal system that learns from community feedback 40 adopting norms and rules CRAFTING HARP • (con)textual algorithims • community governance • proprietary management systems • settings customization encouraging community to review and vote on reported misbehavior 41 adopting norms and rules CRAFTING HARP • (con)textual algorithims • community governance • proprietary management systems • settings customization REVISING REPORTING designing a machine-learning MECHANISMS & CREATING tribunal & reporting mechanisms TRIBUNAL INFRASTRUCTURE 42 adopting norms and rules CRAFTING HARP • (con)textual algorithims • community governance • proprietary management systems • settings customization recalibrating default chat settings & enhancing user control 43 adopting norms and rules CRAFTING HARP • (con)textual algorithims • community governance • proprietary management systems • settings customization recalibrating default chat settings & enhancing user control 44 CRAFTING HARP 40% 45 tangible community impact CRAFTING HARP Outcomes from Riot’s HARP overhaul: 40% drop in verbal abuse 35% increase in positive chat Lin, Re/Code; see also Laura Hudson, Curbing Online Abuse Isn’t Impossible. Here’s Where We Start., WIRED (May 14, 2014), http://www.wired.com/2014/05/fighting-online-harassment/ 46 CRAFTING HARP legal and regulatory: approaches to revenge porn 47 CRAFTING HARP Many advocates and targets of revenge porn believe that new criminal laws are necessary to combat revenge porn: 48 CRAFTING HARP “[U]nlawful to intentionally disclose, display, distribute, publish, advertise or offer a photograph, videotape, film or digital recording of another person in a state of nudity or engaged in specific sexual activities if the person knows or should have known that the depicted person has not consented to the disclosure.” from Arizona’s H.B. 2515 Arizona’s Revenge Porn Law Isn’t a Solution – It’s a Different Kind of Problem, VICE News (Sept. 29, 2014) https://news.vice.com/article/arizonas-revenge-porn-law-isnt-a-solution-its-a-different-kind-of-problem 49 CRAFTING HARP An informal 2013 survey of revenge porn images identified the following categories of origin for those images: manipulated