A Natural Experiment on Hate Speech After Terrorist Attacks

The breakdown of antiracist norms: A natural experiment on hate speech after terrorist attacks

Amalia Alvarez-Benjumea´ a,1 and Fabian Wintera

aMax-Planck Research Group “Mechanisms of Normative Change,” Max-Planck-Institute for Research on Collective Goods, 53113 Bonn, Germany

Edited by Douglas S. Massey, Princeton University, Princeton, NJ, and approved August 3, 2020 (received for review April 24, 2020)

Terrorist attacks often fuel online hate and increase the expres- affect online hate speech is important to prevent a toxic online sion of xenophobic and antiminority messages. Previous research environment and promote open conversations. has focused on the impact of terrorist attacks on prejudiced atti- As of now, however, little is known about the mechanisms tudes toward groups linked to the perpetrators as the cause of causing this increase. It is well established through observational this increase. We argue that social norms can contain the expres- studies that terrorist attacks have a profound impact on xenopho- sion of prejudice after the attacks. We report the results of a bic attitudes (12–14), particularly those that occurred in national combination of a natural and a laboratory-in-the-field (lab-in- territory (15), which has led many scholars to assume that the the-field) experiment in which we exploit data collected about rise in online hate results from the change in attitudes (1, 6). The the occurrence of two consecutive Islamist terrorist attacks in attitudinal change argument states that terrorist attacks increase Germany, the Wurzburg¨ and Ansbach attacks, in July 2016. The xenophobic attitudes and antiimmigrant sentiment because peo- experiment compares the effect of the terrorist attacks in hate ple perceive terrorist attacks carried out by out-group members speech toward refugees in contexts where a descriptive norm as intergroup threat (7, 15, 16). This leads to an increase in prej- against the use of hate speech is evidently in place to contexts udice (17, 18) and results in an increase in hate speech as a direct in which the norm is ambiguous because participants observe consequence of the change in attitudes. antiminority comments. Hate toward refugees, but not toward Focusing solely on a change in individual attitudes misses a other minority groups, increased as a result of the attacks only crucial point: Hate speech is a communicative act and, as such, in the absence of a strong norm. These results imply that attitudi- it is regulated by social norms. Social norms play a decisive nal changes due to terrorist attacks are more likely to be voiced if role in containing the public expression of prejudice (19–21), norms erode. such as xenophobic, racist, and discriminatory remarks. Previous research found that norms have a direct effect on behavior that terrorist attacks | social norms | online hate | prejudice | refugees is independent of individual attitudes (22). Because of the independent effects of norms and individual attitudes, an increment of antiminority sentiment after terrorist attacks or a legitimiza- n 18 July 2016, a 17-y old armed with an ax attacked tion of preexisting prejudice will materialize in more hateful Opassengers on board a train heading to Wurzburg¨ in the comments only if the social norm allows it. southern part of Germany. Six days later, on 24 July, another We empirically test the role of social norms in containing the attacker injured several people and killed himself when he det- expression of online hate after terrorist attacks using a com- onated a backpack bomb in Ansbach, near Nuremberg, in the bination of a natural experiment and a laboratory-in-the-field first Islamist terrorist suicide attack in Germany. Both attacks (lab-in-the-field) experiment. Between two waves of data col- were later claimed by the Islamic State (IS). The two consecu- lection in a lab-in-the-field experiment on hate speech in an tive terrorist attacks hit Germany at the peak of the so-called experimental online discussion forum, the Wurzburg¨ and Ans- “European refugee crisis.” During this period, civil wars in Syria bach terrorist attacks took place consecutively in Germany. We and Iraq caused a massive displacement of people fleeing war analyze the impact of these terrorist attacks on hate speech in and political instability, pushing large numbers of refugees to the surrounding countries and Europe. The situation fueled an already heated public discussion on German policies on Significance immigration. After terrorist attacks, hatred often follows suit (1–5). The Surges in hateful and xenophobic content online are often effect is particularly noticeable when the attacker is character- found after terrorist attacks. We find that this effect is ized as a member of a social or religious minority, as exemplified highly dependent on the local context and the respective by the wave of anti-Muslim hate crimes that followed the 9/11 social norms. Prejudiced attitudes are likely to be voiced only terrorist attacks (4–6), the increase in violence against refugees if the perceived social acceptability of expressing prejudice linked to Islamist attacks in Germany (7), or the escalation of increases. Since antihate norms play an important role in hate speech on Twitter after an Islamist attack in the United containing the expression of prejudice, understanding how Kingdom (8). More generally, formal and informal norms of terrorist attacks may impact the strength of the social norm “civic behavior” seem to erode after such attacks, and behavior is essential to understanding societal responses to terrorist that was not acceptable before becomes more frequent in the attacks. aftermath. We explain the erosion of civic behavior by focusing on one Author contributions: A.A.-B. and F.W. designed research, performed research, analyzed of the most immediate public reactions to terrorist attacks that data, and wrote the paper.y can usually be observed in social media: The expression of prej- The authors declare no competing interest.y udice gains traction in online environments (1, 2). We refer to This article is a PNAS Direct Submission.y this as hate speech, which is speech intended to promote hatred This open access article is distributed under Creative Commons Attribution-NonCommercial- on the basis of race, religion, ethnicity, or sexual orientation (9). NoDerivatives License 4.0 (CC BY-NC-ND).y Widespread hate speech may cause anger, frustration, or res- 1 To whom correspondence may be addressed. Email: [email protected] ignation (10) and pushes people out of the public debate (11), This article contains supporting information online at https://www.pnas.org/lookup/suppl/ thus harming the free exchange of opinions and ideas in the doi:10.1073/pnas.2007977117/-/DCSupplemental.y long run. Therefore, understanding how dramatic events might First published September 1, 2020.

22800–22804 | PNAS | September 15, 2020 | vol. 117 | no. 37 www.pnas.org/cgi/doi/10.1073/pnas.2007977117 Downloaded by guest on October 2, 2021 Downloaded by guest on October 2, 2021 lae-ejmaadWinter and Alvarez-Benjumea was sample the voluntar- Participants and Germany. German in residents in to restricted entirely strictly conducted was comments. the in iment of 2,133 is characteristics reference of for general crowdsourcing total workforce the a a on via 135 information in recruited and Demographic resulting attacks were marketplace, terrorist attacks internet the terrorist before the experiment after our for Experiment. pants the the in of Participants Commission Ethics was of the approval by circumstances ethical Bonn. as of collection for unforeseen University rights data requirement the exper- gender after the on to waived the collection, refugees comments Due in data on use group. the treatment and comments control the group consider the as treated we attacks the Hence, as the further jargon. to see (for imental refer situation (28) We political values the 1). the Western to on refugees and terrorism around discussion security narrative Islamist a refugees. threatening fed in and and as surge refugees rights of the increase gender recent linked topics: opinion social Public two on discussions in forum social online described the selected constructing are The discuss for steps to forum. chronological invited (the online topics were participants purpose-built environ- controlled where carefully a yet ment realistic in a as designed data was forum our collected We Experiment Natural a Lab-in-the-Field and a Experiment of Combination A Design: Experimental norm. social to the attributed of solely effect be differ- the therefore can Potential prejudice conditions. express expressed across any in to similar ences are that willingness attacks the ensures the in after or conditions it prejudice into individual in randomization increases that stress to of expression the the ambiguous expect on effect left We deterrent a prejudice. deliberately observed. have is to are norm norm descriptive comments the observed xenophobic be which hate can because in comments of hateful use contexts no the to because against strong norm is descriptive speech a the com- where We in specific contexts observe. vary participants the pare the thus in comments the conditions is of experimental composition prejudice Our they of about (25–27). that expression context expectations expect the changes might appropriate norm participants how it, descriptive many in The 24). If (23, engage disapprove. behavior do. never this of they others participants The approve If other they what strong. prejudice, that by assume of as logically expression signaled may presented the is ambiguous in is engage norm left norm others the is the of norm which the strength in which conditions in to conditions sentiment. compare antirefugee of We expression the limits hate of expression forum the in them. comments by category affected attacks, this be the not Because should to group. comments unrelated selected comparison completely We a is refugees. as on rights hate comments as gender on well features on as impact forum we rights an online First, gender had The strong. on attacks remain refugees. terrorist to against the norm speech that from the assess refrain perceive to of will they need validity they if the but so challenged, if doing is attacks norms the speech after antihate attitudes prejudiced express test to setting perfect a these forums in hypothesis. these role our making crucial a 21), play (20, norms rules domains social formal few contexts, very online are many toward there in speech Since hate issues. investigate gender to and refugees built purpose forum online an ´ ocrigteattdnlcag ruet ti important is it argument, change attitudinal the Concerning public the against norm social the whether test we Second, to likely more be will people that is argument main Our .Tefrmfeatured forum The S1). Fig. Appendix, SI h exper- The S3. Table Appendix, SI oa f19dfeetpartici- different 139 of total A section Appendix, SI al .Nme fcmet N=213 e ieo data of experimental topic time attacks), and per terrorist condition, 2,133) the = after (N and comments (before of collection Number contrary, 1. Table the descriptive on the If, but strong. remained attacks, speech the hate to atti- against due since norms changed condition have strong-norm may the the tudes in by nonexistent be challenged should or refugees no- not against reduced comments the are hate in in norms increase rights The gender attack. gender since in hateful condition in not increase norm but an “nega- refugees, expect a toward we observes case, comments speaker this the In and environment. challenged tive” are norms comments if negative only into translate norma- attitudes the Negative on context: depends tive opinions changing of voicing the comments whether positive very the only groups. in showing respective emphasized hate the by further toward of norm the condition use descriptive signals strong-norm the the hate against The that use norm forum. regularity others descriptive online behavioral many a a of how existence of created perception comments This hostile the speech. all biased removed thus we and condition, weak-norm strengths. is an different the with speech conditions, In highlighted strong-norm is hate norm and descriptive using weak-norm antihate of the very In acceptability to descrip- ambiguous. hostile the specific from Therefore, any ranged comments signal positive. presented not signaled. the including did norm; is comments, that tive of norm examples, mix specific condi- speech a different no hate featured the condition condition in no-norm no-norm content The the forum In the of tions. summary to a language shows friendly norm. antihate from the comments, of con- of transgressions actual mix condition different Each a a condition. of no-norm, sisted strong-norm a a conditions: and experimental weak-norm, different three into ized Conditions. partici- Experimental per the comments from eight collected on was of comment information participants. total identifying a a further leave No with to pant. page, required forum see was design, participant each (for the Each (29) of 7). validity validity tion create the internal observa- on to increases discussion individual and selected a that independent previously ensures are (see had This tions could thread conditions. we Participants each different forum). comments online of the the the bottom of only the screenshot see at a and for comment discussions 1 a the Fig. leave with par- to every presented started, asked consecutively experiment the was Once ticipant instructions). the (see the join topics for to pictures social 4 about asked different comments (SI were portrayed avatar leave They that an to S2). and instructed name Fig. and user and conversations a 3 given were section and partici- Appendix, study to the consent in informed pate provided participants Forum. experiment, Experimental the in Participating col- experimental data attacks), of terrorist topic. the time and after by condition, or collected before comments (e.g., 1 of lection Table number We conditions. the marketplace. experimental shows online into an participants via experiment randomized the for registered ily onr 3 2 3 228 226 135 135 227 225 134 rights Gender 135 136 Total Refugees norm Strong rights Gender norm Weak Refugees norm No Condition h i fti td st rvd neprmna etof test experimental an provide to is study this of aim The PNAS 405 eoeteatcsAtrteattacks the After attacks the Before | etme 5 2020 15, September atcpnsi h ou eerandom- were forum the in Participants 677 225 | ttebgnigo the of beginning the At o.117 vol. al S1 Table Appendix, SI 393 123 section Appendix, SI sec- Appendix, SI | o 37 no. 658 204 | 22801

SOCIAL SCIENCES exam” or the use of hate terms, e.g., “They can continue walk- ing away from Europe. They are not just war refugees, 90 per cent are nothing but social parasites who do whatever they want here” (emphasis added). SI Appendix, section 5 contains examples of comments and their classiﬁcation. SI Appendix, Table S4 gives an overview about the descriptive statistics of the mean hate score. Results This section provides the main results of our experiment. SI Appendix provides further details regarding the statistical approach (SI Appendix, section 8) and the statistical models (SI Appendix, section 9), as well as auxiliary analyses (SI Appendix, section 10).

Hate toward Refugees but Not toward Other Groups Soared after the Terrorist Attacks. We ﬁrst check whether the attacks had an impact on hate speech against refugees in our online forum. For this purpose, we analyze the effect of the terrorist attacks on hate speech in the no-norm condition (N = 725 comments). As described before, the no-norm condition features a mix of comments containing examples of antiminority and xenophobic comments; therefore, the descriptive norm against the expression of hate in this condition is ambiguous. SI Appendix, Table S5 provides the results from random- effects multilevel regression models, which take account of the nested design of comments in participants. The results are also depicted in Fig. 2. The mean hate score increases by 0.56 points following the attacks (P = 0.004; see model 1 in SI Appendix, Table S5 and Fig. 2A). To give an intuition about what these coefﬁcients mean, we can look at how changes in the score are translated into changes in hostility in the comments. A change in one score point can be very noticeable when comparing two comments from a thread on the same topic. A comment with a score of 6 reads “Get rid of the funny get-up ...We are in Europe, or more precisely Germany. Whoever wants to live here has to adapt. Multiculturalism, sure, but not that much.” A comment in the same picture, but with a score of 7 reads “That’s a very spe- cial bird, a black barn owl ...” (in reference to a woman wearing a burqa). Fig. 1. Screenshot of the no-norm experimental condition (our translation; We next compare the postattacks change in hate speech in German in the original). Image credit: Getty Images/Pierre Crom. against refugees to postattacks changes in gender rights to isolate the effect of the terrorist attacks. We assume that the terrorist

social norms have no effect, we would expect hate speech to increase similarly for all conditions and topics. AB Construction of the Hate Score. Our main focus in this study is changes in hate speech, i.e., expressed prejudice toward minority groups displayed by the participants in their comments. We therefore constructed a hate score to measure change before and after the terrorist attacks and across the different conditions. To construct the hate score, we asked 577 external raters to rate how prejudiced the comments were several weeks after the data collection in both waves. The raters were recruited from the same population as the participants (SI Appendix, Table S3). Every rater was assigned a set of about 30 randomly chosen comments and then an average score was computed for each comment (in SI Appendix, Fig. S5, we conduct analyses of the ratings). The rating task was completed online. We asked the raters to rate the comment on the following scale: “Is the comment friendly or hostile toward the group represented in the Fig. 2. (A and B) Postattacks mean change estimates of hate score in com- picture? (Give a number from 1 to 9 where 1 means very friendly ments on refugees (A) and comments on gender rights (B) for the different levels of the descriptive norm: no norm (N = 257), weak norm (N = 271), and and 9 means very hostile).” Comments with lower scores, e.g., strong norm (N = 270). Estimates account for the nested structure of the 1 to 4, are therefore affable, with a cordial language, and often data with a random intercept multilevel linear regression with an effect for express a strong-norm opinion. On the other side of the spec- participants. The regressions are conducted separately for each treatment. trum, high scores such as 8 or 9 generally imply abusive language, The dashed line represents no change in hate after the attacks. Error bars e.g., “I cannot stand gay people. They should have a psychiatric represent the 95% conﬁdence interval of the estimates.

22802 | www.pnas.org/cgi/doi/10.1073/pnas.2007977117 Alvarez-Benjumea´ and Winter Downloaded by guest on October 2, 2021 Downloaded by guest on October 2, 2021 aesoei infiatysalri h eknr (β weak-norm the in smaller P significantly is (SI score rights hate gender i.e., attacks, not is the S5). attacks to Table the unrelated Appendix, after topics ( norm in descriptive attacks found the the of effect after deterrent and indistinguish- before only, statistically is comments able express they strong-norm prejudice of and amount the weak-norm with confronted (β condition norm or weak the Model refugees. either about in in comments 3 hate in condition in strong-norm difference no the is postattacks There and refugees. three on the pre- comments in score for 2 hate conditions in Fig. experimental change comments. average postattacks antiimmigrant the the contain and depicts to xenophobic norm descriptive of Follow- the expression forums. expect to we strong-norm condition argument, and no-norm our weak-norm ing the the in in effect comments of effect their on the attacks comparing by against terrorist prediction norm the this descriptive a test of We presence expression. the its in forum online the in significantly is zero. coefficient from estimated different the test group, and each treatment for each whether, for The separately estimate. conducted the of are intervals regressions confidence coeffi- 95% regression the lines represent and Terror- Markers cients after refugees. on Speech comments Hate for of Expression Attacks. the ist Contain Norms Social online the in refugees against speech forum. hate increased attacks the significant in marginally only is points dif- (P 0.36 the of Also difference refugees. in about ference comments postattacks for increase o h unie 0ht 5ho h itiuino h hate the of distribution Winter the and Alvarez-Benjumea of refugees 95th about to comments the 10th on across quantiles 3 varies change the Fig. score. for attacks mag- postattacks hate terrorist the the the of the how distribution We depicts this of show above. of effect percentiles and to different the score model of hate regression the nitude quantile 75th of the a distribution in the estimate the emerge as comments, conditions of extreme the quantile most between the differences by largest driven is between conditions difference Comments. attacks the Extreme postterrorist the on that Effect show we Greatest Finally, the Have all Norms Descriptive across consistent been have would from exclusively attitudes conditions. attitudes resulting in negative hate therefore increase in in attacks an increase increase terrorist an An the refugees. to toward attributed after solely forum in be the increase attacks cannot The in the conditions. speech strong-norm after hate and opinions weak-norm antiimmigrant the in and xenophobic ing descriptive the when only ambiguous. occurs are speech norms claim hate our in with increase after consistent the significant are that findings became These attacks. conditions terrorist main experimental the the the that means of which effects condition, no-norm the to compared both in offset is (β condition condition no-norm weak-norm the the in points 0.56 postattacks of the increase that models regression multilevel random-effects signif- statistically not is effect (P this became but icant rights hostile, gender more on points gen- comments 0.23 attacks, discussing the comments After in rights. found der not topics. unrelated certainly toward is hate increase on This effect an have not would attacks ´ 0 = fw etitteaayi opsatcscmet,temean the comments, postattacks to analysis the restrict we If increase not does refugees against speech hate that predict We h ecitv ompeetdpriiat rmexpress- from participants prevented norm descriptive The .Terslsspottefidn that finding the support results The S5). Table Appendix, SI 0 = al S5 Table Appendix, SI n h tognr odto (β condition strong-norm the and .04) e h neato emAtrAttacks After term interaction the see .063; 0 = i.2dpcstepsatcscagsi aescore hate in changes postattacks the depicts 2 Fig. n usatal mle hnte05 points 0.59 the than smaller substantially and .28) = −0.68, rvdscrooaigrslsfrom results corroborating provides = P −0.55, 0 = .I h atcpnsare participants the If .054). β P 0 = 0 = .014, n h strong- the and .11) = −0.62, P 0 = × . P .This 95). Refugees = 0 = −0.48, . 01) et u td miial ofim hsfidn n provides and con- finding online this an confirms in empirically norms attacks study of terrorist erosion Our after the text. explain conversations to the civic by reasoning challenged of our are apply norms trans- the We or if public event. only same, in norms the social materializes of stay change gressions increase, Attitudinal may decrease. the violations cues, even social norm these of on Depending prevalence to form the norms. how in social cues on descriptive such environment provides of others their of the behavior in the After cues and permissible. behave, for socially search becomes people also attacks, voiced speech publicly hate be will if prejudice only that argue social we However, certain increase groups. expression—toward might their found, attitudes—and research terror- empirical negative that previous containing acknowledge as for We attacks, attacks. norms ist terrorist social after of speech importance hate the shows study Our Discussion condition after condition no-norm strong-norm attacks. and the the weak-norm in the in likely likely less more and hate- becoming extremely from comments result attacks ful that the suggests after this changes Overall, average attacks. the the also after is comments frequent hateful less effect More became negative: the but conditions, quantiles highest strong-norm for larger weak-and Table Appendix, the (SI In attacks the S8). no- after hateful points, more the point 1.083 one in and than 1.023 score 90th of the the (P increase and respectively in estimated 80th an the points have attacks, 0.56 terrorist quantiles the of after Compared condition change distribution. norm average the of the in quantiles stronger to “hostile” is effect or this higher but the quantile distribution, dis- hate entire 75th increased the the attacks the along terrorist of speech from the condition, tail particularly no-norm the score, upper In on. hate the the to of closer tribution pronounced more become shown are and norm, results weak before The norm, no score norm. condition. conditions: strong hate experimental no-norm three the the all to for in compared attacks the attacks the the on in terrorist change indicated the the quantile after to repre- the corresponds line coefficient gray for Each Each coefficient intervals. the confidence sents 95% the with score with attacks terrorist of effect the level. of confidence for interval 95% coefficients confidence the regression no- represent quantile the lines the in vertical The score gray norm. hate The the strong condition. of and norm distribution norm, all preattacks weak for the norm, refugees is category no on reference norm: comments descriptive in the 95th of to levels 10th quantiles for attacks terrorist 3. Fig. After Attacks h siae a ewe h he odtostnsto tends conditions three the between gap estimated The − − − 3 2 1 0 1 2 h ltdpcsteetmtdmdlcefiinso h feton effect the of coefficients model estimated the depicts plot The PNAS < .505 .50.95 0.75 0.50 0.25 .A h 0hqatl omnsaemore are comments quantile 80th the At 0.05). | No etme 5 2020 15, September − Quantiles oftheHateScore Quantile Treatment Effects omWeak Norm − omStrong Norm | o.117 vol. − Norm | o 37 no. τ hquantile th | x 22803 axis.

SOCIAL SCIENCES additional evidence on how the expression of hate is regulated other instigators of antisocial behavior. Future research should by social norms. consider repeated interaction, nonanonymity, or heterogeneous We find that hate speech toward refugees increased after effects to expand the results. For the present purposes, the design the terrorist attacks only if participants could observe previous suffices to show the importance of descriptive norms in shaping hate comments. Online hate against refugees increases after the the reaction to terrorist attacks in online communities. attacks both compared with levels of hate against refugees before Our results suggest that supervisors of public discussions, the attacks and relative to the increase toward gender rights. either in the virtual domain or in the real world, may be well This increase is not found when the social norm is exogenously advised to implement measures to ensure a well-tempered atmo- manipulated to remain strong. Under the assumption of proper sphere. Particularly after events leading to a broad discussion randomization of individual attitudes into experimental condi- about the status of minorities, too many bad examples could lead tions, this difference can be solely attributed to the normative to an erosion of norms and embolden prejudiced citizens. As an context. unintended consequence, this may lead to the (self-)exclusion of Furthermore, we show that the terrorist attacks radicalize the marginalized groups from the discussion and in the worst case to already hateful comments under certain conditions, but have a breakdown of the whole debate. only a small effect on positive or moderate ones. Extremely hateful comments become more frequent after the attacks when the Data Availability. Anonymized comma-separated value (csv) code have been norm is ambiguous, but less frequent in the conditions with a deposited in Open Science Framework (DOI 10.17605/OSF.IO/3S5PK) (30). strong norm. Descriptive norms against hate speech thus act as a ACKNOWLEDGMENTS. We thank Emad Bahrami Rad for the programming bulwark that prevents extremely prejudiced opinions from being assistance. Valuable feedback in earlier drafts of this article was provided voiced. On the positive side, the vast majority of people would by Nan Zhang, Rima Maria Rahal, Delia Baldasarri, and Michael Mas;¨ the not be converted into spreaders of hate, just because others do members of the Max Planck Institute for Research on Collective Goods; the members of the Norms and Networks cluster at the University of Groningen; so. However, it also suggests that those who already hold nega- and conference participants at the 12th Annual Meeting of the Interna- tive beliefs about minorities seem to be encouraged to paint an tional Network of Analytical Sociologists (INAS 2019), the 6th International even darker picture of immigration in Western societies. Meeting on Experimental and Behavioral Social Sciences (IMEBESS 2019), Of course, this study paints a simplified picture of online com- the 3rd Cultural Transmission and Social Norms workshop (CTSN 2018), and the seminar of the Democracy, Elections, and Citizenship (DEC) research munities in which participants are chosen randomly, interact only group at the Autonomous University of Barcelona. Financial support of once, and are anonymous. Furthermore, we do not explore how the Max-Planck-Society for the Max-Planck Research Group “Mechanisms of different individuals may react differently, such as racist trolls or Normative Change” is gratefully acknowledged.

1. I. Awan, I. Zempi, ‘I will blow your face OFF’–Virtual and physical world anti-Muslim out-group discrimination and support for anti-immigration policies–evidence from 9 hate crime. Br. J. Criminol. 57, 362–380 (2017). European countries. Rev. Int. Psychol. Soc. 22, 203–233 (2009). 2. P. Burnap et al., Tweeting the terror: Modelling the social media reaction to the 17. B. M. Riek, E. W. Mania, S. L. Gaertner, Intergroup threat and outgroup attitudes: A Woolwich terrorist attack. Soc. Network Anal. Min. 4, 206 (2014). meta-analytic review. Pers. Soc. Psychol. Rev. 10, 336–353 (2006). 3. R. D. King, G. M. Sutton, High times for hate crimes: Explaining the temporal 18. I. Fritsche, E. Jonas, T. Kessler, Collective reactions to threat: Implications for inter- clustering of hate-motivated offending. Criminology 51, 871–894 (2013). group conflict and for solving societal crises. Soc. Issues Policy Rev. 5, 101–136 4. I. Disha, J. C. Cavendish, R. D. King, Historical events and spaces of hate: Hate crimes (2011). against Arabs and Muslims in post-9/11 America. Soc. Probl. 58, 21–46 (2011). 19. F. A. Blanchard, C. S. Crandall, J. C. Brigham, L. A. Vaughn, Condemning and con- 5. B. D. Byers, J. A. Jones, The impact of the terrorist attacks of 9/11 on anti-Islamic hate doning racism: A social context approach to interracial settings. J. Appl. Psychol. 79, crime. J. Ethn. Crim. Justice 5, 43–56 (2007). 993–997 (1994). 6. E. Hanes, S. Machin, Hate crime in the wake of terror attacks: Evidence from 7/7 and 20. A. Alvarez-Benjumea,´ F. Winter, Normative change and culture of hate: An 9/11. J. Contemp. Crim. Justice 30, 247–267 (2014). experiment in online environments. Eur. Sociol. Rev. 34, 223–237 (2018). 7. S. Jackle,¨ P. D. Konig,¨ Threatening events and anti-refugee violence: An empirical 21. K. Munger, Tweetment effects on the tweeted: Experimentally reducing racist analysis in the wake of the refugee crisis during the years 2015 and 2016 in Germany. harassment. Polit. Behav. 39, 629–649 (2017). Eur. Sociol. Rev. 34, 728–743 (2018). 22. E. L. Paluck, Reducing intergroup prejudice and conflict using the media: A field 8. M. L. Williams, P. Burnap, Cyberhate on social media in the aftermath of Woolwich: experiment in Rwanda. J. Pers. Soc. Psychol. 96, 574 (2009). A case study in computational criminology and big data. Br. J. Criminol. 56, 211–238 23. C. Horne, S. Mollborn, Norms: An integrated framework. Annu. Rev. Sociol. 46, 467– (2015). 487 (2020). 9. I. Gagliardone, D. Gal, T. Alves, G. Martinez, Countering Online Hate Speech (UNESCO 24. K. D. Opp, “What is always becoming what ought to be.”: How political action Publishing, 2015). generates a participation norm. Eur. Sociol. Rev. 20, 13–29 (2004). 10. S. Hinduja, J. W. Patchin, Offline consequences of online victimization: School 25. M. E. Tankard, E. L. Paluck, Norm perception as a vehicle for social change. Soc. Issues violence and delinquency. J. Sch. Violence 6, 89–112 (2007). Policy Rev. 10, 181–211 (2016). 11. C. West, “Words that silence? Freedom of expression and racist hate speech” in 26. C. Bicchieri, Norms in the Wild: How to Diagnose, Measure, and Change Social Norms Speech and Harm: Controversies over Free Speech, I. Maitra, M. K. McGowan, Eds. (Oxford University Press, New York, NY, 2016). (Oxford University Press, Oxford, 2012), pp. 222–248. 27. Bicchieri, C., & Dimant, E. Nudging with care: The risks and benefits of social 12. M. Walters, R. Brown, S. Wiedlitzka, Causes and motivations of hate crime. information. Public Choice, 1–22 (2019). https://ssrn.com/abstract=2918883. Accessed 12 February 2020. 28. E. Greussing, H. G. Boomgaarden, Shifting the refugee narrative? An automated 13. A. Echebarria-Echabe, E. Fernandez-Guede,´ Effects of terrorism on attitudes and frame analysis of Europe’s 2015 refugee crisis. J. Ethnic Migrat. Stud. 43, 1749–1774 ideological orientation. Eur. J. Soc. Psychol. 36, 259–265 (2006). (2017). 14. H. G. Boomgaarden, C. H. de Vreese, Dramatic real-world events and public opinion 29. H. Rauhut, F. Winter, “On the Validity of Laboratory Research in the Political and dynamics: Media coverage and its impact on public reactions to an assassination. Int. Social Sciences: The Example of Crime and Punishment” in Experimental Political Sci- J. Public Opin. Res. 19, 354–366 (2007). ence, B. Kittel, W. J. Luhan, R. B. Morton, Eds. (Palgrave Macmillan, London, 2012), 15. J. Legewie, Terrorist events and attitudes toward immigrants: A natural experiment. pp. 209–232. Am. J. Sociol. 118, 1199–1245 (2013). 30. A. Alvarez-Benjumea,´ F. Winter, Supplementary materials the breakdown of anti 16. B. Doosje, A. Zimmermann, B. Kupper,¨ A. Zick, R. Meertens, Terrorist threat and per- racist norms: A natural experiment on hate speech after terrorist attacks. Open ceived Islamic support for terrorist attacks as predictors of personal and institutional Science Framework. https://osf.io/3s5pk. Deposited 10 July 2020.

22804 | www.pnas.org/cgi/doi/10.1073/pnas.2007977117 Alvarez-Benjumea´ and Winter Downloaded by guest on October 2, 2021