Downloaded by guest on September 27, 2021 iia literacy digital lower. far where are India, use northern media of social area of rural sam- in rates largely representative a not a in (but respondents among of effects States ple no United find the we However, in India). later remained online discernment weeks educated in several highly increase This a measurable 17.5%). and (by 26.5%) India (by in States sample United the sam- in representative nationally ple a both among headlines and news mainstream false result, between a discernment As larger. improved significantly intervention head- were the news latter reduced the false on intervention and effects mainstream but this lines, both countries. to of 14 accuracy exposure in perceived people that the to indicate news false results spot Our to pro- how which on campaign, “tips” literacy vided States media modeled largest world’s United intervention the an the on of closely effectiveness in the assess exper- elections we media India, survey recent and digital preregistered around of from conducted role data iments the phe- Using explored this shortfalls. has have to literacy date antecedents studies to political few research and nomenon, While crit- psychological societies. a on modern is focused for online 2020 28, circulating challenge April Levi misinformation ical Margaret in Member Board belief Editorial by Widespread accepted and MA, 2019) Cambridge, 20, Technology, November of review Institute for Massachusetts (received Rand, G. David by Edited India 131029, Haryana Sonipat, University, 03755; NH Hanover, College, 84112; UT City, c a Reifler Jason e eerhsosta e epeaepeae oeffectively to prepared are people few that shows research users xii). p. skills vey 7, critical (ref. the fiction” of from all truth separate take intro- to will 1997 muster “It can influential subject. an the to to duction according publishing,” fever-swamp personal the in of creativity “Misinformation—and as theorists. easily as by —breeds early identified was mation contex- and high- content. between news skills low-quality distinguish the and effectively information to lack of required they knowledge quality because tual the online most evaluate encounter conditions, reliably they ideal to under complex struggle and Even people fragmented of (6). a ecosystem concept navigate competen- information the successfully and skills to news, the needed fake captures cies usefully and literacy media disinformation digital digital on ture political online literacy. to media digital vulnerability in human shortfalls another arti- misinformation: this on In focus (3–5). we factors cle, psychological and economic, political, 2). (1, countries problem of the number voters—a address a to misleading in actions other government are inspired and information has that false news” fear or “fake dubious so-called of that forms worry increasingly world S India and Guess States M. Andrew United the in news false and mainstream between increases discernment intervention literacy media digital A www.pnas.org/cgi/doi/10.1073/pnas.1920498117 eadR odSho fPbi oiy nvriyo ihgn n ro,M 48109-1045; MI Arbor, Ann Michigan, of University Policy, Public of School Ford R. Gerald eateto oiis rneo nvriy rneo,N 08544; NJ Princeton, University, Princeton Politics, of Department oeta 0yltr hs annsse rsin.Sur- prescient. seem warnings these later, y 20 than More misinfor- and literacy media digital between connection The litera- empirical emerging the in overlooked largely While on focused far thus has misinformation online into Research namtr oiia iifrain epeaon the for around ground People fertile be misinformation. to political proved inflammatory have platforms media ocial | e eateto oiia cec,Wsigo nvriyi t oi,S.Lus O63130-4899; MO Louis, St. Louis, St. in University Washington Science, Political of Department g,1 oilmedia social n elna Sircar Neelanjan and , a,1,2 g | ihe Lerner Michael , eateto oiis nvriyo xtr xtrE44J ntdKndm and Kingdom; United 4RJ, EX4 Exeter Exeter, of University Politics, of Department misinformation h,1 b,c,1 ejmnLyons Benjamin , b eateto oiia cec,Uiest fMcia,AnAbr I48109-1045; MI Arbor, Ann Michigan, of University Science, Political of Department d,1 o niermr n iifrain ia misinformation Viral misinformation. and rumors environment also online fertile has a democracy, use for largest Smartphone world’s (12). the mis- scale India, was massive made news a This low- at fake (11). people that election States, leading fears presidential United widespread US created 2016 the in phenomenon the media In social before on months years. widely the distributed recent were articles in online quality India and this States in encounter they claims dubious to context. people victim skills, literacy fall media frequently digital necessary the news Without other algorithmic feeds. via and misinformation politicians disseminate widely allowing to tra- 10), actors lack (9, that controls platforms obtain editorial media effectively ditional social increasingly have from information worldwide adults information new and US people news learn of Nonetheless, to 17% (8). only confidence online that and 2017 skills as Cen- the Research recently Pew as the found example, ter For world. digital the navigate doi:10.1073/pnas.1920498117/-/DCSupplemental. at online information supporting contains article This 2 1 article this at in available results is at available the is replicate plan preanalysis at to study repository necessary Dataverse the scripts at Editorial and available the are files by invited Data editor deposition: guest Data a is D.G.R. Submission. Direct Board. PNAS a is article This oeiaie ies . C BY-NC-ND) (CC 4.0 NoDerivatives License distributed under is article access open This interest.y competing research, no designed declare authors N.S. paper.y The the and wrote J.R., and B.N., data, analyzed J.M.M., research, B.L., performed M.L., A.M.G., contributions: Author owo orsodnemyb drse.Eal [email protected] Email: work.y addressed. this be to may equally correspondence contributed whom N.S. To and J.R., B.N., J.M.M., B.L., M.L., A.M.G., aeta eaieysot clbeitretoscudbe could interventions world. the scalable around misinformation short, fighting in effective relatively the indi- results both that encouraging largely in cate Our intervention India. and literacy of States media effectiveness United news the digital high-quality evaluates real-world and study a large-scale low- This between people’s online. and distinguish literacy to media ability digital documenting explain- between exists relationship evidence factor the rigorous to little plat- critical leading technology However, of design forms. misinformation, a the and online policy as education in in changes belief identified widespread been ing media online digital has the in deficit navigate literacy global effectively This to environment. prepared information are people Few Significance hs ocrshv eoeepcal ain nteUnited the in salient especially become have concerns These ao .Montgomery M. Jacob , y d eateto omncto,Uiest fUa,Sl Lake Salt Utah, of University Communication, of Department https://osf.io/97rnz . y f https://osf.io/u3sgc eateto oenet Dartmouth Government, of Department h . y eateto oiia cec,Ashoka Science, Political of Department e,1 raieCmosAttribution-NonCommercial- Commons Creative https://doi.org/10.7910/DVN/Q5QINN rna Nyhan Brendan , y https://www.pnas.org/lookup/suppl/ NSLts Articles Latest PNAS h ni td raayi plan preanalysis study India The . f,1 , | h US The . f10 of 1

POLITICAL SCIENCES spread via WhatsApp in India has reportedly provoked hatred ticism toward catchy headlines). Importantly, the success of this and ethnic violence (13). Moreover, online political misinforma- approach does not require readers to take burdensome steps like tion became a significant concern during the 2019 Indian general conducting research or thinking deeply about each piece of news election as political parties engaged in aggressive digital cam- they encounter (which is typically impossible in practice given paign efforts via short message service (SMS) and messaging the volume of stories that social media users encounter). Instead, applications like WhatsApp (14, 15). For instance, one analy- this intervention aims to provide simple decision rules that help sis found that over 25% of the news shared on during people distinguish between mainstream and false news, which we the election by the governing Bharatiya Janata Party (BJP) came call “discernment” following ref. 4. from dubious outlets (16). There are important reasons to be skeptical about the effec- Many nonprofits and governments are seeking to counter tiveness of this approach. Prior research has found that media these trends (and the related threat of foreign manipulation literacy interventions like this can help people think critically campaigns) by improving the digital media literacy of news con- about the media content they receive (43). However, prior stud- sumers (17–20). For instance, American universities increasingly ies focus mostly on offline health behavior; the extent to which teach media literacy to undergraduate students (21) and similar these interventions are effective for controversial political claims efforts are also being proposed at the kindergarten to grade 12 or online (mis)information is largely unknown. Moreover, such (22). Similarly, WhatsApp and the National Association of Soft- interventions may struggle to overcome people’s reliance on ware and Service Companies announced plans to train nearly heuristics such as familiarity and congeniality that news con- 100,000 people in India through in-person events and posts on sumers use to evaluate the credibility of online stories (44, 45). social media to spot misinformation (23). Finally, attempting to identify false news through close scrutiny Despite the attention and resources these initiatives have of a headline differs from the typical approach of professional received, however, little large-scale evidence exists on the effec- fact checkers, who usually use “lateral reading” of alternative tiveness of promoting digital media literacy as a response to sources to corroborate claims (46). online misinformation. Existing scholarly work related to digital We therefore conducted preregistered survey experiments in and media literacy is frequently qualitative in nature or focused both the United States and India examining the effectiveness on specific subpopulations and/or issues. Observational find- of presenting people with “tips” to help spot false news sto- ings are mixed (24, 25) and randomized controlled trials remain ries. [The US and India studies were each preregistered with rare (26). Evidence in Governance and Politics; see Materials and Meth- Two related but more specific approaches have been shown ods. All preregistered analyses are reported in this article or in to be somewhat effective in countering misinformation and the replication archive for the study (47).] Strikingly, our results are important to note, however. First, inoculation interventions indicate that exposure to variants of the Facebook media liter- have been employed to protect audiences against misleading acy intervention reduces people’s belief in false headlines. These content by warning of misinformation and either correcting spe- effects are not only an artifact of greater skepticism toward all cific false claims or identifying tactics used to promote it. This information—although the perceived accuracy of mainstream approach has been shown to reduce the persuasiveness of misin- news headlines slightly decreased, exposure to the intervention formation in specific domains (27–32). In addition, other studies widened the gap in perceived accuracy between mainstream and evaluate the effectiveness of providing warnings about specific false news headlines overall. In the United States, the effects misinformation (33, 34). of the treatment were particularly strong and remained statis- We therefore seek to determine whether efforts to promote tically measurable after a delay of approximately 3 wk. These digital media literacy can improve respondents’ ability to cor- findings suggest that efforts to promote digital media literacy can rectly evaluate the accuracy of online content across issues. Such improve people’s ability to distinguish between false and main- a finding would suggest that digital media literacy shortfalls are stream news content, a result with important implications for a key factor in why people fall victim to misinformation. In par- both scientific research into why people believe misinformation ticular, we consider the effects of exposure to Facebook’s “Tips online and policies designed to address the problem. to Spot False News,” which were developed in collaboration with Our main research hypotheses evaluate whether the media the nonprofit First Draft and subsequently promoted at the top literacy intervention reduces belief in false news stories (hypoth- of users’ news feeds in 14 countries in April 2017 and printed esis 1 [H1]), increases belief in mainstream news content (H2), in full-page newspaper advertisements in the United States, the and improves respondents’ ability to distinguish between them United Kingdom, France, Germany, Mexico, and India (35–40). (H3). We also consider three research questions (RQs) for which A variant of these tips was later distributed by WhatsApp (a our a priori expectations were less clear. First, past research Facebook subsidiary) in advertisements published in Indian and shows that the effects of many experimental treatments (e.g., in Pakistani newspapers in 2018 (41, 42). These tips are therefore persuasion and framing studies) decay quickly over time (48), almost surely the most widely disseminated digital media lit- although providing participants with novel information may have eracy intervention conducted to date. (The full treatments are more long-lasting effects (49). We therefore test the durability provided in SI Appendix, section A.) The US treatment, which of our treatment effect by leveraging a two-wave panel design was adapted verbatim from Facebook’s campaign, consists of 10 to tests its effects several weeks after the initial intervention strategies that readers can use to identify false or misleading (RQ1). Second, it is also possible that interventions may work stories that appear on their news feeds, whereas the India treat- only to make individuals more skeptical of noncongenial content ment, which uses adapted versions of messages shown in India by they are already inclined to dismiss, leaving their vulnerability Facebook and WhatsApp, presents 6. to ideologically consistent misinformation unchanged. We there- These interventions provide simple rules that can help individ- fore test for the heterogeneity of the treatment effects based uals to evaluate the credibility of sources and identify indicators on the partisan congeniality of the content (RQ2). Finally, we of problematic content without expending significant time or test whether the intervention changed self-reported intentions attention. For instance, one sample tip recommends that respon- to share false stories or subsequent online news consumption dents “[b]e skeptical of headlines,” warning that “If shocking behavior in the US sample where these measures were available claims in the headline sound unbelievable, they probably are.” (RQ3). Additional analyses exploring heterogenous treatment Such an approach should reduce reliance on low-effort processes effects and alternate outcomes are discussed below, but full mod- that frequently lead people astray (e.g., perceptions of cognitive els appear in SI Appendix, section C. These analyses include fluency) by teaching people more effective heuristics (e.g., skep- whether intuitive cognitive style or prior headline exposure

2 of 10 | www.pnas.org/cgi/doi/10.1073/pnas.1920498117 Guess et al. Downloaded by guest on September 27, 2021 Downloaded by guest on September 27, 2021 . onso -on cl itn otet[ITT]: treat to nearly (intent of decrease scale a 4-point show SE a 1 Table on in points study US 0.2 Results the articles. of news 1 false wave of from accuracy a perceived causes the intervention in literacy decrease media the to exposure randomized Experiment. Survey US Results headlines. “hyperpartisan” of treatment the credibility whether perceived as the well affects as effect, treatment the moderates us tal. et Guess wave in news mainstream had in intervention [SE belief literacy 1. on (ITT, Table media effect in 1 the negative columns to small third exposure and a that second These find the (H3). distin- in We articles successfully shown news are to false results and ability mainstream mainstream people’s between of increase guish accuracy and perceived (H2) the news increase would vention to compared con- (0.35 reach 0.18; significance not statistical does of difference compliers thresholds this among ventional but greater takers actually never is among websites than news false see to available; visits are par- data among Appendix behavioral conducted SI presurvey websites (analysis whom untrustworthy not for do visit ticipants who to those propensity to baseline compared statistically their dif- no in treatment find the fer take we who respondents Crucially, sub- that (SI B1). evidence modest significant the Fig. are B, However, differences section these Trump. of Appendix, most Donald of predispositions tak- toward magnitudes conspiracy never stantive feelings in than lower their parties scored and political also two Compliers in the ers. polarized toward politically more feelings politics, and in their identifiers, interested likely Republican graduates, more college knowledgeable, were older, Compliers (50). be B) to tak- (“never section (“compliers”) treatment , Appendix to it (SI assigned to if ers”) even assigned not would if who only those to treatment the take cessfully (ATT: scale nearly 4-point by a decreased P headlines on two-stage that false points estimate using of 0.3 we compute accuracy approach, perceived we this the which With ATT, regression. assignment. the least-squares random reports original the 1 is Table successfully instrument were our of [66%] and two-thirds condition treated) (approximately questions treatment tips follow-up the news of in the series respondents of treatment a content of answer variables the receipt correctly instrumental about to for ability an indicator the our using is model, (ATT), this In treated approach. treatment the average the on as known effect is which it, received actually every who force cannot we carefully. tips respondents, these of read the subset to offer respondent random interven- can tips” a news to we “fake literacy tion have While digital may the respondents. read which to some opportunity intervention, by the neglected of been effect true the understate hyperpar- C2). of Table C, accuracy section Appendix, perceived (ITT: the headlines tisan on intervention literacy † eesrl mle sn hsls tign ento ftetetuptake. treatment of definition stringent less this using smaller cor- in necessarily question try comprehension third each the answering by as rectly exploratory compliance additional define provide instead also that We study. results India the in reported analyses compliance hs eut eentpeeitrdbtwr siae omthtepreregistered the match to estimated were but preregistered not were results These < u ethptee rdce httemdaltrc inter- literacy media the that predicted hypotheses next Our suc- would who respondents of characteristics the compare We those on treatment the of effect the estimate also therefore We above described effects intent-to-treat the that is concern One 0 = 0 = 0.005). P .020; .026], 0 = β = .08). † eto A section , P −0.046 P < < .W bev iia fet ftemedia the of effects similar observe We 0.005). .Hwvr h eaieefcso the of effects negative the However, 0.01). u T fetetmtsare estimates effect ATT Our C7. Table C, section Appendix, SI [SE ossetwt u rthptei (H1), hypothesis first our with Consistent β o eal) h vrg ubro prior of number average The details). for = 0 = SE −0.176, .017], P < β 0 = ATT, 0.01; = .020; SE −0.299, P < β β (SI 0.005) = = 0 = −0.196, −0.071 .030; nrae icrmn ewe antemadflestories intervention false literacy and media the mainstream result, (ITT, between a described discernment news As increased false larger. of are accuracy above perceived the on intervention false all between accuracy viewed. perceived headlines in news difference mainstream all mean and and the respon- accurate” the as all in at difference calculated level at is the dent “not accuracy for news represents mainstream variable 1 versus dependent false where The perceived scale, accurate.” “very 4 represents to are 4 1 accuracy accuracy). a news news mainstream on and mainstream measured false and perceived false for variables for Dependent respondent paren- by in errors (OLS) (clustered standard squares theses robust least with ordinary coefficients least-squares are two-stage entries or Cell 2018). December to (November ATT effects ITT perceived on intervention literacy type media news US by of accuracy Effect 1. Table ogrsaitclymaual ywv .A eut the (β decayed result, magnitude 0.020; its a the false although in zero wave, As and from second no mainstream distinguishable 2. statistically was between on remained wave headlines difference content treatment by accuracy news literacy measurable perceived mainstream media statistically of the longer accuracy of perceived effect the negative the tion, otemdaltrc nevnini ae1 eraeo 7 of decrease of decrease a relative 1, a wave represents in effect This intervention points. literacy percentage media assigned the were the in who respondents 32% to among “very from 24% as to decreased condition headline accurate” control false United “somewhat accuracy. a or headline the rating accurate” perceived respondents in of of intervention indicator proportion literacy The binary a media using the States of effects treat (ITT, more 1 by wave attenuates to magnitude its P relative later, half weeks than present still is 0.035], .Temda nevlbtenwvsws2 ;the d; 20 was waves between inter- C, interval literacy section remains median 5th Appendix, headlines media The aver- (SI false C1). that the zero of Table waves from of accuracy between distinguishable perceived effect statistically delay the the a on After weeks, vention RQ1. several per aged 2 wave in news condition. respon- false control the and in to mainstream improvement compared between stories 26.5% distinguish to a ability represents dents’ relative effect In content. this of types terms, two these between distinguish better < *P i.1ilsrtstesbtniemgiueo h netto intent the of magnitude substantive the illustrates 1 Fig. nadto,w ettedrblt fteeteteteffects treatment these of durability the test we addition, In osat0.551*** 4,907 4,907 19,623 4,907 9,813 0.551*** 4,907 (respondents) N (headlines) 4,907 N 19,623 effects fixed Headline 4,907 9,813 Constant intervention literacy Media (respondents) N (headlines) N effects fixed Headline Constant intervention literacy Media to ATT, 0.005; < β P 5 **P 0.05, 95th P 0 = < < 0.05). .146 0. ecniernews1 o2 .Wieteeffect the While d. 29 to 16 was range percentile ,dmntaigta thle epnet to respondents helped it that demonstrating 005), < [SE 1 ***P 0.01, β = 0 = −0.121 .024], < 0 tosdd.Dt r rmwv 1 wave from are Data (two-sided). 0.005 −0.299*** −0.196*** [SE 000 006 (0.035) (0.026) (0.030) (0.024) (0.017) (0.020) P as antemfalse Mainstream False X X X X < 0 = ATT, 0.005; .028], β NSLts Articles Latest PNAS = .7* 0.223*** −0.071** 0.146*** −0.046** −0.080 P < β .I addi- In 0.005). 0 = 0 = [SE Mainstream− .223 SE .050; (0.016) (0.016) 0 = | [SE .019], f10 of 3 = =

POLITICAL SCIENCES Wave 1 Wave 2

60% 60%

40% 40%

20% 20%

0% 0% False news Mainstream news False news Mainstream news

Control Media literacy intervention

Fig. 1. Percentage of US respondents rating false and mainstream news headlines as somewhat accurate or very accurate. Respondents rated two and four headlines, respectively, in wave 1 and four and eight headlines, respectively, in wave 2. Headlines were selected randomly in wave 1, balanced by partisan congeniality, and presented in random order. Error bars are 95% confidence intervals of the mean.

approximately one-fourth in the percentage of people wrongly larger (Table 2), although the difference between the ITT and endorsing misinformation. Treatment effects continue to persist the ATT is larger for the Indian analysis because fewer respon- with this alternate measure—in wave 2, the intervention reduced dents answered all comprehension checks correctly in the Indian the proportion of people endorsing false headlines as accurate sample (28% in the online sample versus 66% in the United from 33 to 29%, a 4-percentage-point effect. By contrast, the pro- States).‡ Respondents to the online survey who received the portion of respondents who classified mainstream news as not treatment were nearly half of a response category more skeptical very accurate or not at all accurate rather than somewhat or very of false news stories (ATT: β = −0.470, SE = 0.097; P < 0.005). accurate decreased only from 57 to 55% in wave 1 and 59 to 57% As in the United States, we also find no support for H2, in wave 2. which predicted that exposure to the treatment would increase Finally, RQ2 explores whether the effects of the media literacy the perceived accuracy of mainstream news. Instead, the per- intervention are moderated by the partisan congeniality of the ceived accuracy of mainstream news decreased, although by less headlines people rated. We find no consistent evidence that the than the effect observed for false news (ITT, β = −0.071 [SE = effects of media literacy intervention are moderated by conge- 0.025], P < 0.01; ATT, β = −0.259 [SE = 0.095], P < 0.01). niality (SI Appendix, section C). In other words, the effects of the Results again mirror the US study for H3—respondents bet- intervention were not differentially concentrated among head- ter distinguished between mainstream and false articles (ITT, lines that were uncongenial to respondents—an encouraging null β = 0.063 [SE = 0.025], P < 0.05; ATT, β = 0.221 [SE = 0.088], result that echoes findings in recent studies (34, 51, 52). P < 0.05).§ While the magnitude of this effect is lower than for Additional results reported in SI Appendix, section C indi- the US sample, it translates to a 17.3% improvement in dis- cate that we have insufficient evidence to conclude that the cernment between mainstream and false news relative to the intervention had an effect on self-reported intention to share difference observed in the control condition. As we discuss in false news or behavioral measures of news consumption (SI more detail below, treatment effects cannot be distinguished Appendix, section C, Table C13). However, the intervention did from zero in the second wave (RQ1). significantly increase sharing intentions for mainstream news and While the analyses of the online samples from the United decrease sharing intentions for hyperpartisan news. This is con- States and India show substantially similar results, results from sistent with previous studies that have reported mixed effects of the face-to-face survey in India differ in important ways. As warning labels on sharing intent (33, 34). The mixed results we shown in Table 2, we find no evidence that the treatment observe for sharing intent may be attributable to the fact that increased the perceived accuracy of mainstream news articles belief accuracy questions appeared immediately before the shar- as predicted by H2. However, it did not reduce the perceived ing intent questions in the survey, which may prime accuracy accuracy of these headlines either as we found in the United concerns among respondents and thereby alter both real and States and online studies. In addition, unlike the other studies, self-reported sharing behavior. In addition, we find no measur- we find no evidence that the media literacy treatment system- able effect of the intervention on posttreatment visits to false atically affected beliefs in false news stories (H1) or discrimi- news, mainstream news, or fact-checking sites, although these nation between false and mainstream news (H3) among India effects can be estimated only among the subset of respondents face-to-face respondents. for whom we have behavioral data (SI Appendix, section C, Tables C14–15).

‡The analysis of compliers is presented in SI Appendix, section B. As SI Appendix, Fig. B2 India Survey Experiments. As detailed in Materials and Methods, shows, compliers in the India online sample (those who would take the treatment if we conducted separate online and face-to-face surveys in India of assigned) were more likely to be young, male, Hindu, and high caste; to have graduated different populations. For the online sample, we again find sup- from college; to use WhatsApp; and to have more political knowledge and interest than port for H1. The media literacy treatment significantly reduced never takers (respondents who would not take the treatment if assigned to receive it). We find no significant differences between these groups in the face-to-face sample (SI beliefs in false news stories (ITT: β = −0.126, SE = 0.026; Appendix, section B, Fig. B3). P < 0.005) in the first wave of a two-wave survey (Table 2). §In an exploratory analysis, we show that the result is robust to using an indicator for As with the US analysis, the ATT estimate was substantially false news headlines instead of headline fixed effects (SI Appendix, section D, Table D3).

4 of 10 | www.pnas.org/cgi/doi/10.1073/pnas.1920498117 Guess et al. Downloaded by guest on September 27, 2021 Downloaded by guest on September 27, 2021 ATT T effects ITT type news by accuracy perceived on intervention literacy media India of Effect 2. Table us tal. et Guess reliable statistically no found We (RQ1). time over persist would headlines). mainstream respondents—62% for 72% face-to-face and among headlines lev- false (belief overall for headlines higher mainstream or were false els either for not those did and treatment who literacy media survey was was the rela- face-to-face received there what between who the above, respondents average of noted on group), half difference as significant Finally, treatment only no headlines. was the false accuracy for in perceived observed 60% in decrease to tive controls (from intervention for literacy media 63% the survey online received by who main- skeptically respondents more although viewed group contrast, also By this were ). D7 stories among stream Table headlines D, percent- false section Appendix, of 19 (SI endorsement approximately in of points decline successfully—the age a treatment much indicates the was estimate received intervention ATT the who relative the of those in in effect for 10% 44% the greater to above, approximately noted group of As control decrease terms. the a in group, 49% accurate treatment from somewhat as accurate headlines very false or rating percent- survey, respondents the online reduced of the intervention age For literacy media surveys. the India to two exposure the issue to respondents this for consider (we samples the in further of demo- or modes compositions survey different graphic the reflect may studies between 0.163], = stories news false (ITT, and mainstream between accuracy perceived in 0. (ITT, stories news news false 0. of 0.035], ratings = between of for [SE face-to-face effects null the samples of the intervention online estimate reject (ITT literacy and can media face-to-face we the in that indicate difference D13, in no presented Table results, D, These section model. pooled a accuracy using perceived in difference mean the as level respondent variable the dependent at The accurate.” calculated “Very viewed. is represents and headlines 4 accuracy and news false news accurate” mainstream perceived all mainstream all for at and variables versus “Not false Dependent represents false all 1 accuracy). perceived where between news scale in mainstream 4 difference and to 1 the false a for for on respondent measured by are accuracy (clustered news parentheses mainstream in errors standard robust with n o h T siaefrrtnso mainstream of ratings for estimate [ ITT 266 the for and 005) *P e eerhqeto a hte n ramn effects treatment any whether was question research key A effect ITT the of magnitudes substantive the illustrates 2 Fig. samples two the between difference the assess directly We ediefie effects fixed Headline (respondents) N (headlines) N ediefie effects fixed Headline Constant intervention literacy Media (respondents) N (headlines) N Constant intervention literacy Media < β SE 5 **P 0.05, = 0 = Discussion). −0.057 P > .148], < .Ptnily h ifrne norresults our in differences the Potentially, 0.05). P 1 ***P 0.01, < S 0.039], = [SE β P ATT, 0.005; 0 = > ,btntfrteaeaedifference average the for not but 0.05), 7 [ .072 < 0 tosdd.Dt r rmwv Arlt a 09.Cl nre r L rtosaelatsurscoefficients least-squares two-stage or OLS are entries Cell 2019). May to (April 1 wave from are Data (two-sided). 0.005 −0.470*** −0.126*** as as esMisra esMainstream news Mainstream news False Mainstream−false news Mainstream news False SE P (0.097) (0.026) 17,031 17,031 3,177 3,177 − β 0 = X X > 0 = niedifference, online ATT, 0.05; .035], .428 P S 0.149], = [SE < −0.259** −0.071** ATT, 0.05; β (0.095) (0.025) niesml aet-aesample Face-to-face sample Online 17,163 17,163 3,182 3,182 = X X IAppendix, SI −0.194 β 0 = .118 P β [SE = < 0.361*** 0.361*** bu as esi h aesuy(T estimate: (ITT warning study general same a the of SE in effects news the false exceed about disputed greatly labels estimate: and were (34) the (ITT articles 0.005) to that checkers exposure indicate fact to of by used effect are initially sizes from estimated effect Facebook lessons the the the task, to ratings apply headline comparable to the to respondents intervention instruct the not does study pooled (see the 3 as Fig. results). well and model in as shown perceptions study are each accuracy results for The estimates news news. effect false false treatment and mainstream combined reduced between over- a discernment the of increased replicates study, studies pattern face-to-face three all India all from the data findings pooling in these estimate between observed Although discern not increased. to news are ability false respondents’ and news result, mainstream mainstream accu- for a perceived the diminished As the and are that headlines. on study headlines effects US false negative the of con- racy find both relatively study indicates, online a 3 India Fig. reveals As studies pattern. across sistent results our Comparing headlines the Discussion whether on depend partisanship. respondent observe to congenial literacy we were effects media the the congeniality that partisan that by survey moderated (RQ2; India were either effects intervention’s in evidence Appendix, able (SI D2). sample Table India D, ratings either section in accuracy respondents headline 2 affected wave treatment among the that evidence ¶ (0.088) (0.025) (0.017) (0.017) 0.221* 0.063* nteoln uvy h t o9t ecnierne ee1 o2 n 5t 6d, 26 to 15 in and described A d experiment addi- section message 29 four fact-check the to the include 14 from we headlines if were false unchanged ranges tional substantively percentile are results 95th 2 to wave 5th The respectively. the survey; online the in h einitrasbtenwvswr 0dfrtefc-ofc uvyad1 d 19 and survey face-to-face the for d 20 were waves between intervals median The 3,160 3,160 hs fet r losbtnieymaigu.Atog our Although meaningful. substantively also are effects These 0 = . .W antconclude cannot We D9). Table D, section Appendix, SI 034; eto ,TbeD10). Table D, section Appendix, (SI P < −0.035 −0.007 (0.024) (0.113) 13,712 13,712 .Acmaio fefc ie ihother with sizes effect of comparison A 0.05). 3,267 3,267 X X ¶ ial,w i o n ttsial reli- statistically find not did we Finally, (0.113) (0.024) 13,969 13,969 eto F section Appendix, SI 3,314 3,314 0.011 0.002 β X X = SE −0.236, NSLts Articles Latest PNAS 0 = 0.237*** 0.237*** β (0.030) (0.138) (0.021) (0.021) 3,140 3,140 0.028 0.006 = .036; IAppendix , SI −0.079, | o full for −false f10 of 5 P <

POLITICAL SCIENCES Online Face−to−face

60% 60%

40% 40%

20% 20%

0% 0% False news Mainstream news False news Mainstream news

Control Media literacy intervention

Fig. 2. Percentage of India respondents rating false and mainstream news headlines as somewhat accurate or very accurate in wave 1. Respondents rated six of each type of headline. The headlines were balanced by partisan congeniality and presented in random order. Error bars are 95% confidence intervals of the mean.

randomized media literacy interventions in SI Appendix, section face-to-face participants because WhatsApp use was so rare (SI E, Table E1 shows that our US study has the largest measured Appendix, section D9). effect size to date on ratings of false headlines (d = 0.20) and Finally, we consider the potential trade-off between increased that the India online study (d = 0.11) falls in the upper middle of skepticism toward false news headlines and decreased belief the distribution. Moreover, effect sizes are substantially larger in mainstream news headlines. Our results do indicate that for respondents who were successfully treated with the media increased skepticism of false news headlines may come at the literacy intervention. expense of decreased belief in mainstream news headlines—the Despite the strength of the combined estimates, the effec- media literacy intervention reduced the perceived accuracy of tiveness of the intervention varied across samples. First, the these headlines in both the US and India online surveys. How- intervention may have been more unfamiliar or difficult to ever, the magnitude of the decrease in the perceived accuracy understand for Indian respondents, who successfully received of mainstream news headlines ranges from under 25% (United the treatment at a much lower rate than those in the United States) to just over half (India online sample) of the estimated States. Additional evidence suggests that respondents may have size of the decrease in the perceived accuracy of untrustworthy applied the intervention differently. Specifically, the US data news headlines in wave 1 of our surveys. As a result, respondents’ show that the negative effects of the media literacy interven- overall ability to distinguish between mainstream and untrust- tion on perceived headline accuracy were greater for headlines worthy news increases by more than 26% in the US sample and from untrustworthy, hyperpartisan, and unfamiliar mainstream 17% in the highly educated online Indian sample. Moreover, we sources that respondents in the control group found less plausi- observe no measurable decrease in the perceived accuracy of ble to begin with (r = 0.79; SI Appendix, section C, Table C11 mainstream news headlines in wave 2 of any of our surveys. and Fig. C2). This finding helps explain the observed nega- A related concern is that the intervention could reduce the tive effect of the media literacy intervention on the perceived overall accuracy of people’s beliefs given that they typically con- accuracy of mainstream news overall.# By contrast, no such sume much more information from mainstream sources than relationship between baseline headline accuracy and media lit- from untrustworthy ones (53). To address this concern, we use eracy intervention effects is observed in the Indian online data US Pulse web metering data to estimate the overall change that (r = −0.09; SI Appendix, section D8), suggesting respondents the intervention would hypothetically induce in people’s ability became more skeptical across the board. to accurately discern credible news given exposure rates for US Problems applying the intervention may have been particularly participants to different types of news sources (see SI Appendix, acute for respondents in the face-to-face sample. This group is section G for details). Because Americans’ news consumption quite dissimilar from both the highly educated online sample is concentrated among the high-prominence mainstream out- in India and the US sample on a number of important indi- lets for which the intervention may have had a small positive cators (SI Appendix, section B). In particular, participants in effect#, these calculations indicate that individuals would reach the face-to-face study had much less experience with the task valid accuracy beliefs for 64.6% of stories in the treatment group of evaluating news headlines online—only 11% reported using compared to 62.9% in the control group. Moreover, the percent- WhatsApp compared with 90% for the online sample in India. age of “false positives”—stories they encounter from dubious Correspondingly, an exploratory analysis shows the effects of the sources and believe to be true—would decrease from 6.1% intervention were similar among WhatsApp users across sam- of all stories consumed in the control group to 4.9% in the ples; however, these effects are imprecisely estimated among treatment group. Conclusion

#An exploratory analysis of whether source prominence moderates the effects of The findings we present provide important evidence that short- the media literacy intervention shows that the negative effects we observe for the falls in digital media literacy are an important factor in why perceived accuracy of mainstream news headlines were concentrated among stories people believe misinformation that they encounter online. We from low-prominence sources. By contrast, we find that the intervention appeared to increase the perceived accuracy of stories from high-prominence sources (SI Appendix, find that a simple, scalable media literacy intervention can section C9). decrease the perceived accuracy of false news content and help

6 of 10 | www.pnas.org/cgi/doi/10.1073/pnas.1920498117 Guess et al. Downloaded by guest on September 27, 2021 Downloaded by guest on September 27, 2021 eea aet hudb fee.Frt h fetszswere sizes effect the First, offered. be misinformation, should against fight caveats the in several tool valuable a be could tions encounter they issues. content or topics news more different of users on credibility helping the stories, news gauge reducing false accurately at of effective accuracy be be perceived could can con- the which scale intervention By at brief content. disseminated a news inexpensively that show real-world results of these array trast, when diverse discernment improve a not evaluating Inoc- may (27–30), misinformation 44). domains (34, refuting specific scale preemptively in to in difficult effective but approaches while practice ulation, policy in that propose effective factors others are that identify while (4), misinformation intervention in mislead- resist belief or Some implications. false for real-world counter important explanations help with finding to a strategy news, ing effective an be could not does respondents’ with observe predispositions. align question political we in claims the improvement performance whether the on rating depend Moreover, accuracy in India. headline news and mainstream in States factual United from the it both distinguish better to people perceived in difference the for level respondent main- them. the and between false accuracy at for level and headline news the at stream estimated are sizes Effect intervals. 3. Fig. us tal. et Guess lhuhteerslssgetta iia ieayinterven- literacy digital that suggest results these Although campaigns literacy media that suggest further results Our aaaefo ae1 fetszsaepotdwt 5 confidence 95% with plotted are sizes Effect 1. wave from are Data -.2 SIndiaonline Combined India face-to-face US Mainstream news False news 0 .2 aetedmgahcadpltclatiue fteU ouain(32% population US the of approxi- attributes to political algorithm and weighting demographic and the matching mate YouGov’s by selected were e eddb h uvycmayYuo hrl fe h 08US 2018 2019, 3, the January after to 2018 shortly 2018, 14, 27, YouGov December December to 2, company 20 wave November survey 1, (wave the elections midterm by fielded vey dur- national place after took immediately All and India. during campaigns. in interest electoral conducted political high were of place two periods took and ing survey States One United intervention. the literacy in media embedded an included Collection. Data the Methods reduce and improve Materials to thereby way and cost-effective news democracies. and of false health vital information to a be vulnerability to people’s panacea, prove a not may are effects. literacy they digital lasting literacy online improve potentially digital to and efforts real-world While significant a have that can date intervention to demonstration pelling the positive correct (54). wider to media help yield social treated on could are see who they population those misinformation lit- the instance, media for of if, the subset effects improving a everyone, of reach these eracy if not Even do effectiveness contexts. social interventions or in frequency behavior the accuracy-promoting lit- of increase media Finally, can how and interventions content. whether eracy news examine evaluating further should learn for researchers people are heuristics helping observe effective versus concerns they more accuracy effects increased the to whether due interven- identifying such which operate, through tions to mechanism seek the should scholars understand larger addition, better In have effects. potentially durable by more could used and/or which approach checkers), reading” fact “lateral the professional These as literacy contexts. (such digital intensive models electoral more training of and tests include countries, should approaches evaluations these samples, of (53). this effectiveness different on the news test relying in and than false default rather a of to as interventions model evaluations different point of comparative access effects conduct the that important should company an scholars actual technology an However, as used a served we by that has globally is study deployed the intervention of advantage One research. behavior. media social news, to hyperpartisan changes share of possibility to the main- intent suggesting share decreased to and intent news respondents’ stream evidence increased find do treatment we the However, that habits alter. to information difficult because real-world and insufficient ingrained perhaps changed are have news, treat- intervention false the we the of that consumption to Finally, conclude attention effectiveness.) to and of evidence its company ITT lack our the reduces how between when ment difference illustrates link (The estimates News feeds. ATT their False the in Spot it to to have offered exposed to Tips likely the were are users ignored measurable participants news Facebook but real-world mainstream treated many small of intervention; all a accuracy Fourth, perceived have the stories. did on it effect headlines, negative news false the mainstream although and between Third, discernment to basis. overall recurring educators improved a intervention and on journalists, lessons companies, these reinforce media remaining the social longer suggesting for study), no online need and India States the in United measurable the statistically in (diminish- magnitude time in false over ing decayed in effects belief the eliminate Second, not headlines. news did intervention simple this modest; k mtl k h ae bv eetterne nwihalwv n ae2 wave and 1 wave all which in ranges the reflect above collected. dates were The responses wk. 3 imately epnet oktesresi re ihadlybtente vrgn approx- averaging them between delay a with order in surveys the took Respondents nteUie tts ecnutdatowv niepnlsur- panel online two-wave a conducted we States, United the In com- most the are findings these though, knowledge, our To future for directions of number a suggest also findings Our ecnutdtowv ae uvy frsodnsthat respondents of surveys panel two-wave conducted We NSLts Articles Latest PNAS N = ,283). 4, k Respondents N | = f10 of 7 ,907; 4,

POLITICAL SCIENCES college graduates, 45% male, median age 50 y; 46% identify as Democrats nically factual but present slanted facts in a deceptive manner. We selected and 36% as Republicans). A subset of these respondents were members of these stories from hyperpartisan sources identified in prior work (58) (SI the YouGov Pulse panel and voluntarily provided behavioral data on their Appendix, section H). This process resulted in 8 mainstream, 4 false, and online information consumption as well (see SI Appendix, section A for more 4 hyperpartisan headlines. In wave 1, respondents were shown 8 headlines details). (a randomly selected headline from the two available for each possible com- US data collection was approved by the Institutional Review Boards (IRBs) bination of news type, valence, and source prominence), while respondents at the University of Michigan (HUM00153414), Washington University in St. in wave 2 were shown all 16 headlines. Headlines were presented as they Louis (201806142), and Princeton University (10875). University of Exeter would appear on the Facebook news feed to replicate a typical decision accepted the University of Michigan IRB approval. All subjects gave con- environment. Specifically, respondents were shown the article previews that sent to participate in our study. The US study preanalysis plan is available at are automatically generated by Facebook when a link is entered into the https://osf.io/u3sgc. news feed that featured a headline, a photo, the news source’s web domain, For India, we conducted two separate two-wave panel studies, one online and in certain cases a byline or story text snippet. Respondents were asked and the other face to face. Both surveys were conducted in Hindi. Respon- to rate the accuracy of each headline. dents were excluded if indicated they mostly or always give humorous In the India surveys, we adopted the same approach in asking respon- or insincere answers to survey questions (which amounted to 7–8% of dents to evaluate the accuracy of headlines that varied across several responses in the online survey by wave compared to less than 1% in the dimensions: valence (congenial to BJP supporters versus congenial to BJP face-to-face survey; this exclusion represents a deviation from our preregis- opponents) and accuracy (true articles from mainstream sources†† versus tration, but the results in Table 2 are robust to including these respondents). false articles as identified by fact checkers). Nationalism is also commonly In the online survey, we collected survey data from a national convenience linked to misinformation in India (59). The issue was particularly salient sample of Hindi-speaking Indians recruited via Mechanical Turk and the when the India surveys were conducted (a time of escalating tensions Internet Research Bureau’s Online Bureau survey panels (wave 1, April 17 to between India and Pakistan), so we also asked respondents to rate the May 1, 2019, N = 3, 273; wave 2, May 13 to 19, 2019, N = 1, 369). The India accuracy of true and false headlines relevant to nationalist concerns in face-to-face survey was conducted by the polling firm Morsel in Barabanki, the country (either India–Pakistan or Hindu–Muslim relations). Unlike the Bahraich, Domariyaganj, and Shrawasti, four parliamentary constituencies US study (where the same headlines were used in both waves 1 and 2 in the state of Uttar Pradesh where Hindi is the dominant language (wave to test for prior exposure effects), we used different sets of headlines in 1, April 13 to May 2, 2019, N = 3, 744; wave 2, May 7 to 19, 2019, N = 2,695). each wave. Finally, 4 additional false headlines were included in the second These locations were chosen, in part, due to their higher levels of religious wave based on fact checks conducted between the two waves.‡‡ In total, polarization, which we anticipated might increase demand for and belief in respondents rated 12 headlines in wave 1 (6 false and 6 true) and 16 in online misinformation and rumors. The representative random sample for wave 2 (10 false and 6 true). Respondents were presented with the headline the India face-to-face survey was drawn from the public voter registration in text format in the online survey, while enumerators read the headlines list for these constituencies and was administered orally by trained enumer- to respondents in the face-to-face survey. In both cases, participants were ators to account for low literacy rates. Relative to the face-to-face survey, asked to evaluate the accuracy of all headlines they were presented in online respondents were more likely to be male (72% versus 64%), younger each wave. (median age 30 y versus 37 y), more educated (76% college graduates ver- sus 6%), higher caste (42% low caste versus 74% in the face-to-face sample), Analysis. Our primary analyses are pooled OLS models predicting percep- more active on social media (90% use WhatsApp versus 11%), more inter- tions of headline accuracy on a four-point scale that ranges from not at ested in politics (3.7 versus 2.9 on a 4-point scale), more knowledgeable all accurate to very accurate. These models were estimated at the headline about politics (providing correct responses to an average of 2.8 vs. 2.1 of level with fixed effects for each headline. Although we attempted to choose four true–false questions about Indian politics), and slightly less likely to stories that were balanced in their face validity, the headlines differed in support the BJP (42% versus 46%) (SI Appendix, section D, Table D1). plausibility because the actual stories were not constructed by researchers. India data collection was approved by the IRBs at the University of We therefore use the fixed effects to account for baseline differences in Michigan (HUM00160358), Ashoka University, and Morsel Research and perceived accuracy between headlines. Because respondents rated multi- Development (HIRB000007598). University of Exeter accepted the Univer- ple headlines, we also compute cluster-robust standard errors. In addition sity of Michigan IRB approval. All subjects gave consent to participate in our to the pooled OLS models, we also examine the difference in accuracy study. The India study preanalysis plan is available at https://osf.io/97rnz. beliefs between mainstream and false headlines at the respondent level Our study contexts can be viewed as a most-different case comparison by calculating a respondent-level measure of the difference in mean lev- among democracies (55). India and the United States are broadly consid- els of perceived accuracy between mainstream and false headlines. Higher ered the poorest and richest countries, respectively, in terms of income per scores on this scale indicate better ability to discern between stories of capita among longstanding large democracies (ref. 56, p. 42). As a result, different types. India is likely to have lower levels of education and media literacy than Congeniality is a binary variable that is coded at the headline level for the United States, which raises questions about the efficacy of any media partisans to indicate whether a story is consistent with a respondent’s par- intervention. The two studies we conduct within India further refine this tisan leanings (e.g., a Democrat evaluating a story that is favorable to a comparison, allowing us to evaluate the effects of the media literacy inter- Democrat would take the value of 1). Uncongenial is coded as the opposite. vention among both an online sample that has demographics that are more The baseline category is reserved for headline ratings by pure nonparti- similar to the United States and a face-to-face sample in one of the poorest sans. To determine the partisanship of respondents in the US survey, we regions in India. Our study can thus provide evidence about the efficacy of used the standard two-question party identification battery (which includes a media literacy intervention across democratic contexts that differ in levels leaners) to classify respondents as Democrats or Republicans. Because India of income, education, and digital media use. has a multiparty system, we classified respondents there as BJP supporters if they reported liking the BJP more than any other party (on a four-point News Headline Rating Task. The main outcome of interest in all three sur- scale) and as a BJP opponent if they liked any other party more than veys was the perceived accuracy of mainstream and false news headlines. To the BJP. construct this measure, we asked respondents to evaluate the accuracy of The key explanatory variable of interest is exposure to the media lit- a number of headlines on a 4-point scale ranging from very accurate (4) to eracy intervention, which was adapted from an intervention deployed by not at all accurate (1). All of the headlines were published by actual news Facebook and WhatsApp around the world, including in Hindi-language sources or circulated on Facebook or WhatsApp within 6 mo of the respec- newspapers in India (see SI Appendix, section A for details and the exact tive survey, and a portion of the headlines were rated as false by at least text). We randomly assigned respondents in wave 1 of the US and India one third-party fact-checking organization. The order of the headlines was studies with probability 0.5 to be exposed to a set of tips for distinguishing randomized within wave for each respondent. All headlines are shown in SI false news stories from mainstream stories. In the US survey experiment, 10 Appendix, section H1. In the US survey, respondents evaluated 16 different headlines that varied across multiple dimensions: news type (mainstream versus hyper- partisan versus false), valence (pro-Democrat versus pro-Republican), and ††Mainstream news sources included ZeeNews, Washington Post, India, prominence among mainstream sources (high versus low). We define high- IndiaToday, Nikkei Asian Review, Reuters, and Bloomberg. prominence mainstream sources as those that more than 4 in 10 Americans ‡‡These additional headlines were part of a parallel study; further details are provided recognize in Pew polling (57). Hyperpartisan stories are those that are tech- in SI Appendix, section A).

8 of 10 | www.pnas.org/cgi/doi/10.1073/pnas.1920498117 Guess et al. Downloaded by guest on September 27, 2021 Downloaded by guest on September 27, 2021 8 .J baoiz tptemnplto fdmcayoln.NwYr Times, York Helmus C. New T. online. 19. democracy of manipulation the Stop Abramowitz, endorsement. J. theory M. conspiracy and 18. literacy media News Maksl, A. Ashley, S. Craft, S. 17. Narayanan V. 16. 1 Atlantic, The election. India’s endangering is Misinformation Bansal, S. Poonam, S. fact- 15. say elections, 2019 the during 12 intensified has Wired, crisis fake-news India. India’s in Thakar, A. violence 14. and news fake fuels WhatsApp sowing How is McLaughlin, news T. fake 13. believe Americans Many Holcomb, J. Mitchell, A. Barthel, outperformed M. stories 12. news election fake how shows analysis This Face- Silverman, including C. media, 11. social using adults U.S. of Share Anderson, M. Perrin, A. 10. 0 .Ptl ni a ulchat rss tscle aenw.NwYr ie,2 April 29 Times, York New news. fake called It’s crisis. health public a has India Patil, S. 20. us tal. et Guess frequent through them internalized already have or either life who real in takers”—people tips “always the effectively saw are classify treatment to we relative receiving respondents low as some is Additionally, (although guessing rates). effect random compliance true under estimated outcome the the checks, an to comprehension such relative of the estimates likelihood of ATT the our all diminish on will correctly com- which guessed as misclassified they India be may because the respondents of pliers in fraction effects respectively, small understate A 28%, likely reasons. and estimates two ATT for 24% Our and surveys). online survey States and United online face-to-face the US in one the higher had in substantially respondents (66% was face-to-face treatment correctly; of had answer Receipt participants to chance). (online chances try three first to the up on correctly the questions comprehension and sample full the using of estimates receipt determine ITT below. ATT to both tips calculate asked of We was not were group treatment. treatment respondents each would and The after that task questions possible). rating URLs comprehension when headline as length the their before such adapted Face- administered reducing cues being by and after (omitting published 3 present those format of be from groups face-to-face tips 2 the 6 in and presented surveys, for 3 were India of WhatsApp groups the and 2 In book in 4. verbatim of presented group were one Facebook by published tips .J ostr .Bso,H he oilmdauecniust iei developing in rise to continues 2019. use media October Social Chwe, Center, H. Bishop, Research C. Pew Poushter, J. gaps. 9. readiness Digital Horrigan, B. J. era. 8. digital the news Gilster, in P. skills Fake survival 7. for framework Lazer, conceptual A D. literacy: Digital Swire-Thompson, Eshet, Y. B. 6. Friedland, L. Joseph, K. is news Grinberg, fake N. partisan to 5. Susceptibility biased: not Lazy, election. Rand, 2016 G. the D. in Pennycook, news G. fake and 4. media world. Social the Gentzkow, M. around Allcott, actions H. anti-misinformation 3. to guide A Flamini, D. Funke, D. 2. countries. 27 across study A trust: and post-truth bubbles, filter news, Fake Jackson, C. 1. o acltn h T,rcito ramn sdfie saseigall answering as defined is treatment of receipt ATT, the calculating For ad nEsenEurope Eastern in ganda 2019. July 30 https://www.nytimes.com/2017/12/11/opinion/fake-news-russia- Accessed kenya.html. 2017. December 11 Public Computational Communication on Project 2019.2, Memo 2019). (Data UK, campaign” Oxford, Propaganda, election Indian the ing 2019. https://www.theatlantic.com/international/archive/2019/04/india- July 30 Accessed misinformation-election-fake-news/586123/. 2019. April 2019. July https://qz.com/india/1609763/alt-news-boom- 30 2019. Accessed August live-on-fake-news-detection-amid-indian-election/. 3 India, Quartz checkers. 2019. October 9 Accessed https://www.wired.com/story/how-whatsapp-fuels-fake-news-and- violence-in-india/. 2018. December 2020. May 27 Accessed: americans-believe-fake-news-is-sowing-confusion. https://www.journalism.org/2016/12/15/many- (2016). Center Research Pew confusion. 2016. https://www.buzzfeed.com/ December 16 2016. Accessed November term=.ohXvLeDzK#.cwwgb7EX0. 16 Buzzfeed, facebook. craigsilverman/viral-fake-election-news-outperformed-real-news-on-facebook?utm on news real https:// 2019. April 10 2019. Center, November including-facebook-is-mostly-unchanged-since-2018/. Research 20 Accessed Pew 2018. since www.pewresearch.org/fact-tank/2019/04/10/share-of-u-s-adults-using-social-media- unchanged mostly is book, 2018. 2019. June 19 November Center, 20 Accessed Research in-developing-countries-but-plateaus-across-developed-ones/. Pew ones. developed https://www.pewresearch.org/global/2018/06/19/social-media-use-continues-to-rise- across plateaus but countries 2019. November 19 Accessed https://www.pewresearch.org/internet/2016/09/20/digital-readiness-gaps/. election. Hypermedia presidential Multimedia Educ. J. US 2016 the during (2019). on reasoning. motivated by than reasoning (2018). of 39–50 lack by explained better Perspect. 2019. August 14 https://www.poynter.org/ifcn/anti-misinformation-actions/. Accessed (2019). Institute Poynter 2019. August https://www.ipsos.com/en-us/news-polls/Fake-News-Study. 14 2018. Accessed September 5 Ipsos, tl cesd3 uy2019. July 30 Accessed https://www.nytimes.com/2019/04/29/opinion/india-elections-disinformation. html. 2019. –8(2017). 1–28 31, iia Literacy Digital tal., et Nw n nomto vrFcbo n htApdur- WhatsApp and Facebook over information and “News al., et usa oilMdaIflec:UdrtnigRsinPropa- Russian Understanding Influence: Media Social Russian 8–0 (2017). 388–401 2, Rn oprto,2018). Corporation, (Rand WlyCmue u,NwYr,N,1997). NY, York, New Pub, Computer (Wiley 316(2004). 93–106 13, Science Cognition 374–378 363, .Econ. J. 188, 9 .Tuaaa aeokfihsfk esoln ihfl-aepitnewspaper print full-page with online news fake The fights India. Facebook in Tsukayama, news H. fake 39. combat to ads newspaper to turns Facebook Srivastav, T. feed. 38. atop May news’ 8 Mashable, ads. ‘false newspaper print with spotting news fake fights for Facebook 6 Mezzofiore, tips G. 10 Facebook, 37. to misinformation. link puts against Facebook tool Constine, J. educational 36. new A Mosseri, A. 35. Clayton K. 34. 3 .Pnyok .Ba,E .Clis .G ad h mle rt fet Attaching effect: truth implied The Rand, G. D. Collins, T. E. resistance Bear, psychological A. confers Pennycook, game G. news 33. Fake Linden, Inoculating der van Lewandowsky, S. S. Roozenbeek, Leiserowitz, J. A. Cook, 32. J. Maibach, E. Linden, der van S. against public 31. the Inoculating Maibach, E. Maibach, E. Rosenthal, S. Leiserowitz, A. 30. inocula- through misinformation Neutralizing Ecker, confer K. U. to Lewandowsky, S. inoculation Cook, J. attitudinal 29. Using hate: against Vaccinating Test- Braddock, propaganda: theory K. conspiracy 28. to resistance Inducing Blumenthal, Miller, G. S. M. Banas, A. Baker, J. 27. G. Kavanagh, of J. identification Huguet, help literacy A. media 26. Does Liu, J. Mortensen, T. Jones-Jang, toward M. skepticism S. and 25. behaviors, media social literacy, News Tully, M. Vraga, K. E. 24. March Today, India news. 3 fake WQAD, about teach news. to collaborate fake NASSCOM and in WhatsApp lessons 23. get soon could students Illinois Conley-Keck, E. 22. 1 .Jzna olgstr fk es pdmcit ecal oet Washington moment. teachable a into epidemic news’ ‘fake turn Colleges Jazynka, K. 21. a eind odce,o nlzd l ocuin n n rosare errors any study and the conclusions how in All role analyzed. Face- own. no from or our played funding conducted, employees by its designed, supported and was was company admin- the study assistance but India for book, The Research Bureau surveys. Research Morsel the YouGov, Internet istering thank the We and comments. Development, Stan- and helpful and the to for Leuven, Universiteit University and and Katholieke ford Gothenburg; School, Uni- Stromback, Washington Kennedy of George Harvard Jesper University, University versity, Young Brigham the Oscarsson, Christine at Kaiser, participants at Henrik seminar Neggers, group Ben Young, to Resistance” Dannagal Yusuf Lai, Yamaya, Zeitzoff; “Knowledge Munger, Calvin Shun Thomas Vercellotti, to Kevin Tim and outstand- grateful Mayer, Thorson, for also Emily Jonathan Parashar Stedtnitz, are Foundational Sumit Levy, We Integrity and Ro’ee assistance. Facebook support innova- research the and funding ing research and for 2020 Awards 682758), Horizon Research (Grant Union’s program European tion the under Council ACKNOWLEDGMENTS. repository: Dataverse DVN/Q5QINN. the at available are article this Availability. our Data such, As them. effect. true on the effect understate no will estimates have ATT should intervention The experience. d.Wsigo ot 4Arl21.https://www.washingtonpost.com/news/the- 2017. newspaper-ads/?utm April 14 switch/wp/2017/04/14/facebook-fights-fake-news-online-with-full-page-print- Post, Washington ads. https://www.thedrum.com/news/2017/09/22/facebook- 2019. March 2017. 19 Accessed September turns-newspaper-ads-combat-fake-news-india. 22 Drum, 2019. March 19 Accessed uk/#PZqzuqCrwqqP. https://mashable.com/2017/05/08/facebook-fake-news-newspaper-ad-elections- 2017. 2019. March 19 https://techcrunch.com/2017/04/06/facebook-puts-link-to- Accessed 2017. 10-tips-for-spotting-false-news-atop-feed/. April 6 TechCrunch, 2017. May 23 Accessed misinformation/. https://newsroom.fb.com/news/2017/04/a-new-educational-tool-against- 2017. April media. social on stories false in (2019). belief 10.1007/s11109-019-09533-0 reducing Behav., in tags fact-check and warnings annst usto aenw edie nrae ecie cuayo headlines of accuracy perceived increases warnings. headlines news without fake of subset a to warnings misinformation. online against misinformation. against (2017). change. climate about misinformation influence. their reduces techniques One argumentation misleading Exposing tion: propaganda. extremist by (2019). persuasion to resistance Decay strategies. Truth metainoculation (2013). and Mitigating inoculation ing for Tool a as 2019). Education eracy don’t. literacies other but helps, (2019). literacy 10.1177/0002764219869406 Information news? fake media. (2019). social on information 2019. July 30 Accessed https://www.indiatoday.in/education-today/news/story/- 2019. and-nasscom-collaborate-to-teach-about-fake-news-1481882-2019-03-19. March 19 2019. July 30 Accessed fake-news/. https://wqad.com/2019/03/03/illinois-students-could-soon-get-lessons-in- 2019. March ot pi 07 https://www.washingtonpost.com/lifestyle/magazine/colleges- 99b4-9e613afeb09f 2017. April turn-fake-news-epidemic-into-a-teachable-moment/2017/04/04/04114436-fd30-11e6- 6 Post, 0779(2017). e0175799 12, elsltosfrfk es esrn h fetvns fgeneral of effectiveness the Measuring news? fake for solutions Real al., et aafie n cit eesr orpiaeterslsin results the replicate to necessary scripts and files Data 018/nc21.48(2020). 10.1287/mnsc.2019.3478 Sci., Manag. story.html?utm em.7692a0 cesd1 ac 2019. March 19 Accessed term=.c736f9621a30. etakDmcayFn,teErpa Research European the Fund, Democracy thank We Science 10.1080/1369118X.2019.1637445 Soc., Commun. Inf. agaeCommun. Palgrave 1114 (2017). 1141–1142 358, em.62763f cesd3 uy2019. July 30 Accessed term=.46c23796b30f. 10.1080/09546553.2019.1693370 Chall., Global u.Cmu.Res. Commun. Hum. NSLts Articles Latest PNAS 5(2019). 65 5, 1–23 Violence, Polit. Terrorism https://doi.org/10.7910/ xlrn ei Lit- Media Exploring RN Corporation, (RAND m ea.Sci., Behav. Am. 184–207 39, | f10 of 9 Polit. PLoS

POLITICAL SCIENCES 40. A. Al-Heeti, Facebook will fight with real newspaper ads (and more). CNet, 50. M. Marbach, D. Hangartner, Profiling compliers and non-compliers for instrumental 23 May 2018. https://www.cnet.com/news/facebook-is-fighting-misinformation-with- variable analysis. Polit. Anal., 10.1017/pan.2019.48 (2020). news-literacy-campaign-help-from-researchers/. Accessed 19 March 2019. 51. B. Swire-Thompson, U. K. Ecker, S. Lewandowsky, A. J. Berinsky, They might be a liar 41. , After lynchings in India, 10 tips from WhatsApp to spot fake news. 10 but they’re my liar: Source evaluation and the prevalence of misinformation. Polit. July 2018. https://www.thequint.com/news/india/after-lynchings-in-india-whatsapp- Psychol. 41, 21–34 (2020). offers-tips-to-spot-fake-news. Accessed 20 August 2019. 52. M. J. Aird, U. K. Ecker, B. Swire, A. J. Berinsky, S. Lewandowsky, Does truth matter to 42. Agence France Presse, WhatsApp running anti-fake news ads in Pakistan ahead of voters? The effects of correcting political misinformation in an Australian sample. R. elections. 19 July 2018. https://www.rappler.com/technology/news/207674-whatsapp- Soc. Open Sci. 5, 180593 (2018). anti-fake-news-campaign-pakistan-elections. Accessed 20 August 2019. 53. A. Guess, B. Nyhan, J. Reifler, Exposure to untrustworthy websites in the 2016 U.S. 43. E. W. Austin et al., The effects of increased cognitive involvement on college students’ election. Nat. Hum. Behav. 4, 472–480 (2020). interpretations of magazine advertisements for alcohol. Commun. Res. 29, 155–179 54. L. Bode, E. K. Vraga, See something, say something: Correction of global (2002). health misinformation on social media. Health Commun. 33, 1131–1140 44. G. Pennycook, T. D. Cannon, D. G. Rand, Prior exposure increases perceived accuracy (2018). of fake news. J. Exp. Psychol. Gen. 147, 1865–1880 (2018). 55. J. Seawright, J. Gerring, Case selection techniques in case study research: A menu of 45. L. K. Fazio, D. G. Rand, G. Pennycook, Repetition increases perceived truth equally for qualitative and quantitative options. Polit. Res. Q. 61, 294–308 (2008). plausible and implausible statements. Psychon. Bull. Rev. 26, 1705–1710 (2019). 56. A. Stepan, J. J. Linz, Y. Yadav, Crafting State-Nations: India and Other Multinational 46. S. Wineburg, S. McGrew, Lateral reading and the nature of expertise: Reading less Democracies (Johns Hopkins University Press, 2011). and learning more when evaluating digital information. Teach. Coll. Rec. 121, 1–40 57. A. Mitchell, J. Gottfried, J. Kiley, K. E. Matsa, Political polarization & media (2019). habits. Pew Research Center, 21 October 2014. https://www.pewresearch.org/wp- 47. A. Guess et al., Data from “A digital media literacy intervention increases discern- content/uploads/sites/8/2014/10/Political-Polarization-and-Media-Habits-FINAL- ment between mainstream and false news in the United States and India.” Harvard REPORT-7-27-15.pdf. Accessed 21 March 2019. Dataverse. https://doi.org/10.7910/DVN/Q5QINN. Deposited 29 May 2020. 58. G. Pennycook, D. G. Rand, Fighting misinformation on social media using crowd- 48. A. S. Gerber, J. G. Gimpel, D. P. Green, D. R. Shaw, How large and long-lasting are sourced judgments of news source quality. Proc. Natl. Acad. Sci. U.S.A. 116, 2521–2526 the persuasive effects of televised campaign ads? Results from a randomized field (2019). experiment. Am. Polit. Sci. Rev. 105, 135–150 (2011). 59. S. Chakrabarti, Nationalism a driving force behind fake news in India, research shows. 49. A. Coppock, E. Ekins, D. Kirby, The long-lasting effects of newspaper op-eds on public BBC News, 12 November 2018. https://www.bbc.com/news/world-46146877. Accessed opinion. Q. J. Polit. Sci. 13, 59–87 (2018). 9 September 2019.

10 of 10 | www.pnas.org/cgi/doi/10.1073/pnas.1920498117 Guess et al. Downloaded by guest on September 27, 2021