Crafting Online Harassment and Abuse Reporting Policies

crafting online harassment and abuse reporting policies

Amanda Levendowski Internet Law & Policy Foundry Fellow Data & Society Databite December 17, 2015

1 CRAFTING HARP

what’s the problem?

2 CRAFTING HARP

“We suck at dealing with abuse and trolls on the platform and we’ve sucked at it for years. It’s no secret and the rest of the world talks about it every day. We lose core user after core user by not addressing simple trolling issues that they face every day.”

- Dick Costolo, CEO of Twitter

3 CRAFTING HARP 40%

4 CRAFTING HARP

According to a 2014 survey, over 40% of online adults report being targets one or more types of harassment:

called oﬀensive names

7% purposefully embarassed 8% 27% stalked 6% 8% sexually harassed

22% physically threatened

sustained harassment

Maeve Duggan, Online Harassment, PEW Research Internet Project (Oct. 22, 2014), http://www.pewinternet.org/2014/10/22/online-harassment/ 5 CRAFTING HARP

WHO

6 CRAFTING HARP

There may be a correlation between the types of people who engage in harassment and where that behavior is likely to occur:

co-worker 7% former partner 10%

family 12%

friend 23% acquaintance 24%

unknown 26%

stranger 38%

7 CRAFTING HARP

WHERE

8 CRAFTING HARP

There may be a correlation between the types of people who engage in harassment and where that behavior is likely to occur:

online dating site 6% co-worker 7% forum 10% former partner 10%

family 12% email 16%

friend 23% online gaming 16% acquaintance 24% the comments 22% unknown 26%

stranger 38% social media 66%

9 platform permeability matrix CRAFTING HARP

OPEN

social media comments

chat services forums BY USER BY PLATFORM

online communities text messaging CLOSED

10 CRAFTING HARP

HOW

11 taxonomy of harassment CRAFTING HARP

dogpiling

doxxing

gaslighting

hate speech

jaqing oﬀ

revenge porn

sealioning

surveillance

swatting

12 taxonomy of harassment CRAFTING HARP

dogpiling Looping unwanted third parties into an online conversation doxxing

gaslighting

hate speech

jaqing oﬀ

revenge porn

sealioning

surveillance

swatting

13 taxonomy of harassment CRAFTING HARP

dogpiling

doxxing Exposing target’s personal, often sensitive, information gaslighting

hate speech

jaqing oﬀ

revenge porn

sealioning

surveillance

swatting

14 taxonomy of harassment CRAFTING HARP

dogpiling

doxxing Attempting to create doubt in and of gaslighting target’s memory or perception

hate speech

jaqing oﬀ

revenge porn

sealioning

surveillance

swatting

15 taxonomy of harassment CRAFTING HARP

dogpiling

doxxing

gaslighting

Degrading interactions based identity hate speech and/or characteristics

jaqing oﬀ

revenge porn

sealioning

surveillance

swatting

16 taxonomy of harassment CRAFTING HARP

dogpiling

doxxing

gaslighting

hate speech Using leading questions to advance jaqing oﬀ accusations or harmful interactions

revenge porn

sealioning

surveillance

swatting

17 taxonomy of harassment CRAFTING HARP

dogpiling

doxxing

gaslighting

hate speech

jaqing oﬀ Distributing sexually explicit images revenge porn without the consent of the pictured individual sealioning

surveillance

swatting

18 taxonomy of harassment CRAFTING HARP

dogpiling

doxxing

gaslighting

hate speech

jaqing oﬀ

revenge porn

sealioning Popping into a conversation with endless questions or demands surveillance

swatting

19 taxonomy of harassment CRAFTING HARP

dogpiling

doxxing

gaslighting

hate speech

jaqing oﬀ

revenge porn

sealioning

surveillance Using technology to track target’s online behavior swatting

20 taxonomy of harassment CRAFTING HARP

dogpiling

doxxing

gaslighting

hate speech

jaqing oﬀ

revenge porn

sealioning

surveillance

Sending a SWAT team to target’s swatting home, work or other location

21 taxonomy of harassment CRAFTING HARP

dogpiling

doxxing

gaslighting

hate speech

jaqing oﬀ

revenge porn

sealioning

surveillance

Sending a SWAT team to target’s swatting home, work or other location

22 CRAFTING HARP

WHY

23 CRAFTING HARP

conceptualizing solutions

24 CRAFTING HARP

Strategies for combatting harassment and abuse in online communities can be conceptualized as two approaches:

LEGAL AND REGULATORY: creating new laws, amending or repurposing existing laws, relying on agency intervention to govern communities

25 CRAFTING HARP

Strategies for combatting harassment and abuse in online communities can be conceptualized as two approaches:

LEGAL AND REGULATORY: creating new laws, amending or repurposing existing laws, relying on agency intervention to govern communities

NORM AND RULE: retooling, revising, and reimagining existing community standards and community management

26 CRAFTING HARP

Strategies for combatting harassment and abuse in online communities can be conceptualized as two approaches:

LEGAL AND REGULATORY: creating new laws, amending or repurposing existing laws, relying on agency intervention to govern communities

NORM AND RULE: retooling, revising, and reimagining existing community standards and community management

27 CRAFTING HARP

Strategies for combatting harassment and abuse in online communities can be conceptualized as two approaches:

LEGAL AND REGULATORY: creating new laws, amending or repurposing existing laws, relying on agency intervention to govern communities

NORM AND RULE: retooling, revising, and reimagining existing community standards and community management

28 CRAFTING HARP norm and rule-based approach

• (con)textual algorithims • community governance

• censorship • ﬁltration • moderation • proprietary management systems • pseudonymity or real-name requirements • settings customization

29 CRAFTING HARP

strong community norms: the key to eﬀective HARP

30 CRAFTING HARP

To craft eﬀective HARP, platforms must resist the temptation of the politician’s fallacy:

1. We must do something. 2. This is something. 3. Therefore, we must do this.

31 CRAFTING HARP

NOPE

32 CRAFTING HARP

Eﬀective HARP feature carefully crafted community norms, which are reinforced by platforms that:

INVEST in developing community expectations

33 CRAFTING HARP

Eﬀective HARP feature carefully crafted community norms, which are reinforced by platforms that:

INVEST in developing community expectations

EMPOWER users to shape interactions

34 CRAFTING HARP

Eﬀective HARP feature carefully crafted community norms, which are reinforced by platforms that:

INVEST in developing community expectations

EMPOWER users to shape interactions

ENFORCE against disrupters and violators

35 CRAFTING HARP

case in point: league of legends

36 CRAFTING HARP

“As we spend more and more of our time online, we need to acknowledge that online harassment and toxicity is not an impossible problem, and that it is a problem worth spending time on.”

- Jeﬀrey Lin, Riot Games’ lead designer of social systems

Jeﬀrey Lin, “Doing Something About the ‘Impossible Problem’ of Abuse in Online Games,” Re/Code (July 7, 2015) (op-ed), http://recode.net/2015/07/07/doing-something-about-the-impossible-problem-of-abuse-in-online-games/

37 diagnosing the community CRAFTING HARP

Riot Games began by asking: “Who is responsible for bad behavior in our community?”

13% persistently negative players generally inoﬀensive players 87%

38 adopting norms and rules CRAFTING HARP

• (con)textual algorithims • community governance

• proprietary management systems

• settings customization

39 adopting norms and rules CRAFTING HARP

• (con)textual algorithims • community governance

• proprietary management systems

• settings customization

relying on a tribunal system that learns from community feedback

40 adopting norms and rules CRAFTING HARP

• (con)textual algorithims • community governance

• proprietary management systems

• settings customization

encouraging community to review and vote on reported misbehavior

41 adopting norms and rules CRAFTING HARP

• (con)textual algorithims • community governance

• proprietary management systems

• settings customization

REVISING REPORTING designing a machine-learning MECHANISMS & CREATING tribunal & reporting mechanisms TRIBUNAL INFRASTRUCTURE

42 adopting norms and rules CRAFTING HARP

• (con)textual algorithims • community governance

• proprietary management systems

• settings customization

recalibrating default chat settings & enhancing user control

43 adopting norms and rules CRAFTING HARP

• (con)textual algorithims • community governance

• proprietary management systems

• settings customization

recalibrating default chat settings & enhancing user control

44 CRAFTING HARP 40%

45 tangible community impact CRAFTING HARP

Outcomes from Riot’s HARP overhaul:

40% drop in verbal abuse

35% increase in positive chat

Lin, Re/Code; see also Laura Hudson, Curbing Online Abuse Isn’t Impossible. Here’s Where We Start., WIRED (May 14, 2014), http://www.wired.com/2014/05/ﬁghting-online-harassment/

46 CRAFTING HARP

legal and regulatory: approaches to revenge porn

47 CRAFTING HARP

Many advocates and targets of revenge porn believe that new criminal laws are necessary to combat revenge porn:

48 CRAFTING HARP

“[U]nlawful to intentionally disclose, display, distribute, publish, advertise or offer a photograph, videotape, film or digital recording of another person in a state of nudity or engaged in specific sexual activities if the person knows or should have known that the depicted person has not consented to the disclosure.”

from Arizona’s H.B. 2515

Arizona’s Revenge Porn Law Isn’t a Solution – It’s a Diﬀerent Kind of Problem, VICE News (Sept. 29, 2014) https://news.vice.com/article/arizonas-revenge-porn-law-isnt-a-solution-its-a-diﬀerent-kind-of-problem 49 CRAFTING HARP

An informal 2013 survey of revenge porn images identiﬁed the following categories of origin for those images:

manipulated images 12%

36% hacked images

42%

uploaded images

Why One Mom’s Investigation Might Actually End Revenge Porn, On the Media (Dec. 6, 2013), http://www.onthemedia.org/story/why-one-moms-investigation-might-actually-stop-revenge-porn/ 50 CRAFTING HARP

An informal 2013 survey of revenge porn images identiﬁed the following categories of origin for those images:

manipulated images 12%

36% hacked images

42%

uploaded images

An informal 2013 survey of revenge porn images identiﬁed the following categories of origin for those images:

manipulated images 12%

36% hacked images

42%

uploaded images

An informal 2013 survey of revenge porn images identiﬁed the following categories of origin for those images:

manipulated images 12% ? hacked images

42%

uploaded images

54 CRAFTING HARP Some of the most powerful sites on the web have made it easier to remove or limit access to revenge porn:

Facebook Google Instagram Microsoft PornHub reddit Tumblr Twitter

55 CRAFTING HARP

QUESTIONS