A Digital Media Literacy Intervention Increases Discernment Between Mainstream and False News: Evidence from the United States and India”

Home , Ajay Maken, Andrew Gillum

Online Appendix: Supporting Information for “A digital media literacy intervention increases discernment between mainstream and false news: Evidence from the United States and India”

www.pnas.org/cgi/doi/10.1073/pnas.1920498117 Table of contents

Appendix A: Study designs A.1 U.S. study design ...... A-1 A.1.1 Sample and randomization ...... A-1 A.1.2 Web-tracking/web consumption data ...... A-1 A.1.3 U.S. news tips treatment ...... A-2 A.2 India study design ...... A-3 A.2.1 India online sample design ...... A-3 A.2.2 India face to face sample design ...... A-4 A.2.3 India randomization ...... A-5 A.2.4 India news tips treatment ...... A-6 A.2.5 India fact-check experimental design ...... A-7 A.2.6 India headline rating task (outline) ...... A-7 Appendix B: Descriptive statistics for samples by compliance status Appendix C: Additional results from U.S. media literacy intervention C.1 ITT and ATT results for main text ...... C-1 C.2 Effect of intervention on hyperpartisan news ...... C-2 C.3 Balance tests and panel attrition ...... C-3 C.4 Effect of intervention on binary outcome ...... C-5 C.5 Effect of intervention with alternative treatment receipt definition ...... C-6 C.6 Effect of intervention with ordered probit ...... C-7 C.7 Heterogeneous effects by congeniality ...... C-8 C.8 Heterogeneous effects by prior exposure ...... C-10 C.9 Heterogeneous effects by source prominence and headline plausibility ...... C-11 C.10 Effects of intervention on sharing intention and news consumption ...... C-14 Appendix D: Additional results from India media literacy intervention D.1 Descriptive statistics for both India samples ...... D-1 D.2 ITT and ATT results for main text ...... D-2 D.3 Balance tests and panel attrition ...... D-5 D.4 Effect of intervention on binary outcome ...... D-8 D.5 Effect of intervention with ordered probit ...... D-9 D.6 Heterogeneous effects by congeniality ...... D-10 D.7 Robustness to inclusion of additional wave 2 fake headlines ...... D-12 D.8 Heterogeneous effects by headline plausibility ...... D-14 D.9 Moderating effect of WhatsApp use ...... D-16 D.10 Pooled analysis of India samples ...... D-18 Appendix E: Benchmarking effect sizes Appendix F: Pooled analysis (U.S. and India samples) Appendix G: Estimating the net effect of the media literacy intervention Appendix H: U.S. and India study news headlines ...... H-1 H.1 U.S. news headlines ...... H-1 H.2 India headlines (wave 1) ...... H-5 H.3 India headlines (wave 2) ...... H-7 A Study designs

A.1 U.S. study A.1.1 Sample and basic design Participants in the U.S. study were YouGov panel members who consented to participate in an online study (YouGov determines the specific eligibility and exclusion criteria for their panel). Researchers have no role in selecting the participants. This study was conducted among a representative sample of the U.S. population by YouGov, which recruits a large panel of opt-in respondents and then uses a weighting and matching algorithm to construct a final sample that mirrors the demographic composition of the U.S. population. Our participants closely resemble the U.S. population in both demographics and political attitudes and affiliations (see demographics reported in the main text). The experimental results we present do not use survey weights per Franco et al. (2017) and Miratrix et al. (2018). Respondents were randomly assigned to exposure to the media literacy intervention or to a control condition in which no information was shown (simple random assignment with 50% assignment probability via the YouGov platform). The text shown to respondents is provided in Figure A1. All respondents in Wave 1 were then shown 8 news articles for evaluation out of a possible 16: 4 pro- Democratic mainstream news articles (2 from low-prominence sources and 2 from high-prominence sources), 4 pro-Republican mainstream news articles (2 from low-prominence sources and 2 from high-prominence sources), 2 pro-Democratic false news articles, 2 pro-Republican false news articles, 2 pro-Democratic hyperpartisan sources, and 2 pro-Republican hyperpartisan sources. The 8 displayed were chosen by randomly selecting 1 out of each group of 2 listed above for each respondent. In Wave 2, all participants were exposed to all 16 possible articles, thus providing perceptions of accuracy for repeated and non-repeated articles. Headline stimuli are provided in Figures H1 and H2. (See the replication archive for this study for the full survey instrument, along with replication data and code: https://doi.org/10.7910/DVN/Q5QINN.) We find no evidence of imbalance on observable measures across conditions (Table C3). Sim- ilarly, attrition between waves was not significantly related to treatment status (Table C4), though participants who took part in wave 2 had higher levels of political knowledge, were older, and rated false news as less accurate compared to those who did not take part (Table C5).

A.1.2 Web consumption data A subset of our U.S. respondents opted in to provide anonymized web visit data via the YouGov Pulse panel. (See Guess, Nyhan and Reiﬂer 2020 for validation of this data source.) Pulse operates via passive metering technology: An app developed by the analytics ﬁrm Reality Mine collects non-password-protected data from real-time laptop/desktop clickstream activity. Users are clearly informed that they can pause data collection or remove the app at any time. In our U.S. sample, N = 1069 have valid pre-survey web data for which we construct measures of consumption for comparison purposes, while N = 1107 have valid post-treatment data which we use as outcomes. Per our preregistration, we measured post-treatment online information consumption by aggre- gating each respondent’s web visits for the seven calendar days after the one in which they completed the wave 1 survey. The lists we used to code each type of media are below:

A-1 • Mainstream news visit: One of AOL, ABC News, CBSNews.com, CNN.com, FiveThir- tyEight, FoxNews.com, Huﬃngton Post, MSN.com, NBCNews.com, NYTimes.com, Politico, RealClearPolitics, Talking Points Memo, The Weekly Standard, WashingtonPost.com, WSJ.com, or Wikipedia

• Fact-checking visit: PolitiFact, Snopes, or Factcheck.org; the Washington Post Fact Checker is excluded because it is part of the Washington Post, which is already a qualifying media outlet per above

• False news visit: Any visit to one of the 673 domains identiﬁed in Allcott, Gentzkow and Yu (2019) as a false news producer as of September 2018 excluding those with print versions (including but not limited to Express, the British tabloid) and also domains that were previously classiﬁed by Bakshy, Messing and Adamic (2015) as a source of hard news. In addition, we exclude sites that predominantly feature user-generated content (e.g., online bulletin boards) and political interest groups.

We computed a binary measure of exposure to the types of content above as well as a count of the total webpages visited from each category during the period. We coded false news visits as pro-Democrat (pro-Republican) if 60% or more of the visits by partisans (not including leaners) in our sample period were from Democrats (Republicans). Duplicate visits to webpages were not counted if they were successive (e.g., a page that was reloaded after ﬁrst opening it). URLs were cleaned of referrer information and other extraneous parameters before de-duplication. For more detail, see the processing steps described in Guess, Nyhan and Reiﬂer (2020).

A.1.3 U.S. news tip intervention

A-2 Figure A1: U.S. media literacy intervention

“Tips to Spot False News”

Be skeptical of headlines. False news stories often have catchy headlines in all caps with exclamation points. If shocking claims in the headline sound unbelievable, they probably are.

Look closely at the URL. A phony or look-alike URL may be a warning sign of false news. Many false news sites mimic authentic news sources by making small changes to the URL. You can go to the site to compare the URL to established sources.

Investigate the source. Ensure that the story is written by a source that you trust with a reputation for accuracy. If the story comes from an unfamiliar organization, check their “About” section to learn more.

Watch for unusual formatting. Many false news sites have misspellings or awkward layouts. Read carefully if you see these signs.

Consider the photos. False news stories often contain manipulated images or videos. Sometimes the photo may be authentic, but taken out of context. You can search for the photo or image to verify where it came from.

Inspect the dates. False news stories may contain timelines that make no sense, or event dates that have been altered.

Check the evidence. Check the author’s sources to conﬁrm that they are accurate. Lack of evidence or reliance on unnamed experts may indicate a false news story.

Look at other reports. If no other news source is reporting the same story, it may indicate that the story is false. If the story is reported by multiple sources you trust, it’s more likely to be true.

Is the story a joke? Sometimes false news stories can be hard to distinguish from humor or satire. Check whether the source is known for parody, and whether the story’s details and tone suggest it may be just for fun.

Some stories are intentionally false. Think critically about the stories you read, and only share news that you know to be credible.

[These tips are taken verbatim from the original tips published by Facebook (see https://www.facebook.com/ help/188118808357379).]

A.2 India study design We conducted two two-wave surveys in India. The methodology of the surveys is described below.

A.2.1 Online survey design The online survey was conducted via Qualtrics among a non-representative sample of Hindi-speaking participants recruited via Mechanical Turk and the Internet Research Bureau’s Online Bureau survey panel. These participants live across India and therefore varied in the electoral phase of the constituency in which they live. Data collection for the first wave of the survey began April 17, 2019 on Amazon Mechanical Turk and April 25, 2019 on the Opinion Bureau panel and was completed by May 1, 2019. The second wave was fielded from May 13–18, 2019. The timing of the completion of the first wave and the fielding of the second wave mirrors the phase 6 constituencies in the face-to-face survey. Balance tests suggest the randomization was successful (see Table D4a). Similarly, we find no evidence of differential attrition attributable to treatment status among wave

A-3 2 participants (see Table D6), though participants who took part in Wave 2 were more likely to be middle-aged, more educated, Hindu, BJP supporters, and had higher levels of political interest and knowledge relative to those who did not take part (Table D5a). Participant demographics are reported in the main text.

A.2.2 Face-to-face survey design The face-to-face survey was conducted among a representative sample of respondents in four parliamentary constituencies in the Indian state of Uttar Pradesh. The Indian national election is completed in multiple "phases"; each part of the country is assigned to one of seven polling dates corresponding to a phase. In 2019, the national election consisted of seven phases. For this study, we selected two phase 5 constituencies (polling date: May 6, 2019) and two phase 6 constituencies (polling date: May 12, 2019). In each case, we chose parliamentary constituencies in Uttar Pradesh in which the Muslim population was likely to be greater than 20% according to district-level data from the 2011 Indian Census. Using this procedure, we selected the constituencies of Barabanki and Bahraich in phase 5 and the constituencies of Domariyaganj and Shrawasti in phase 6. Wave 1 was fielded from April 13–May 1, 2019 among 3,744 respondents. Wave 2 was fielded May 7–18, 2019 among 2,980 respondents who previously took part in Wave 1. Respondents were recruited using a sampling methodology that blocks at the Assembly constituency level and clusters at the polling booth level. Two Assembly constituencies from each Lok Sabha constituency were randomly selected. For each Assembly constituency, 47 polling booths will be selected. Geographically contiguous polling booths are numbered in order, so we will select polling booths by dividing each Assembly Constituency into 94 evenly-sized groups, randomly selecting evens or odds, and selecting one polling booth for each even or odd group of polling booths. This methodology is intended to ensure low geographic spillovers across polling booths. Ten respondents will be interviewed from the list of registered voters for each polling booth. Voter lists are ordered by geographic contiguity of individuals’ addresses. We will select these respondents by dividing the list of registered voters into 20 evenly-sized groups. We will then randomly select evens or odds. (The even/odd selection process is also intended to reduce the likelihood of geographic spillovers as sequentially numbered voters are likely to be closest to each other.) For each of the ten selected groups of registered voters, we will then select ten respondents: one primary respondent and an additional nine alternates in case of failure to administer the survey. If surveyors are unable to administer the survey to ten respondents in a polling booth using this procedure, we will drop respondents from that polling booth and instead survey ten respondents using the procedure described above in an additional randomly selected polling booth from the same group. Data was collected by professional enumerators contracted by Morsel Research & Development Pvt. Ltd. using electronic tablets running the SurveyCTO mobile data collection platform. Enu- merators uploaded survey responses at least once per work day in order to verify data collection progress and quality. The survey instrument and all treatments were provided in Hindi. Balance tests suggest the randomization was successful (see Table D4b). Similarly, we find no evidence of differential attrition attributable to treatment status among wave 2 participants (see Table D6), though participants who took part in wave 2 were more likely to be Hindu and BJP supporters and less likely to be Muslim and BJP opponents than those who did not take part (Table D5b). Participant demographics are reported in the main text.

A-4 A.2.3 Media literacy intervention design Each respondent was randomly assigned to either the placebo or treatment condition for the media literacy intervention experiment (randomization occurs at the individual level in both the face-to- face and the online survey). Those assigned to the treatment condition were shown (online) or read (face-to-face) six tips for spotting false news adapted from the ads Facebook and WhatsApp published in Hindi-language newspapers in India. Respondents in the control condition were not exposed to any tips. The tips were read or shown in two groups of three with a comprehension question after each set of tips. (See Figure A2 for wording.) To increase the likelihood of respondents understanding the tips, respondents in the online survey who provided an incorrect answer to either comprehension question were asked to read the tips and answer the question again. They were allowed to proceed in the survey after answering the question correctly or trying three times — whichever comes ﬁrst. In the face-to-face survey, such a process was not undertaken due to concerns about respondent attention and attrition. Respondents who provide an incorrect answer to either comprehension question simply proceeded in the survey.

A-5 A.2.4 India news tip intervention

Figure A2: India media literacy intervention

“Tips to Spot False News”

Be skeptical of headlines. If shocking claims in the headline sound unbelievable, they probably are.

Investigate the source. Ensure that the story is written by a source that you trust with a reputation for accuracy. If the story comes from an unfamiliar organization, try to ﬁnd out more about who they are.

Check the evidence. Check the author’s sources to conﬁrm that they are accurate. Lack of evidence or reliance on unnamed experts may indicate a false news story.

Question information that upsets you. If you read something that makes you angry or afraid, ask whether it was shared to make you feel that way. And if the answer is yes, think twice before sharing it.

Think about whether the story is a joke. Sometimes false news stories can be hard to distinguish from humor or satire. Check whether the source is known for parody, and whether the story’s details and tone suggest it may be just for fun.

[These tips are adapted from original tips published by Facebook (see https://www.facebook.com/ help/188118808357379) and WhatsApp (see https://www.thequint.com/news/india/after- lynchings-in-india-whatsapp-offers-tips-to-spot-false-news).]

A-6 A.2.5 Fact-check message experiment design The second intervention in the India study invites respondents to receive updates about a topic via WhatsApp or other electronic means; respondents were randomly assigned to receive either an invitation to receive political fact-checks or placebo content about health and ﬁtness (the version of the experiment in the online study also included a pure control group). The messages were only oﬀered to respondents who use WhatsApp and opted in to receive the messages. Respondents in the placebo and treatment groups for the fact-check message experiment who opted in to receiving messages were sent approximately one per day between Wave 1 and Wave 2 on the following dates (the pure control group in the online study received no messages):

• Phase 5 face-to-face: April 23 plus April 27–May 4 (total: 9 messages via WhatsApp)1

• Phase 6 face-to-face: May 2–4 (total: 3 messages via WhatsApp)2

• Online: May 7–11 (total: 5 messages via email)3

(Results for the fact-check message experiment are presented in the “Additional preregistered anal- yses” document in the replication archive.)

A.2.6 India news headline stimuli Respondents in both surveys rated the accuracy of the same headlines in each wave (headlines diﬀered between waves). These were either shown to respondents online or read to them by enumerators in the face-to-face survey. Both Wave 1 and Wave 2 included both mainstream and false headlines that were either congenial to Bharatiya Janata Party (BJP) supporters or congenial to BJP opponents as well as headlines pertaining to nationalism issues (either India-Pakistan or Hindu- Muslim relations). Sample headlines are provided in Figure A3. Wave 2 also included four false headlines that had been fact-checked (these were drawn from the messages sent to respondents in the treatment group of the fact-check message experiment between Wave 1 and Wave 2).

1Due to a miscommunication with the survey firm conducting the face-to-face survey, an introduction message and one fact-check or health/fitness fact message were sent to all eligible phase 5 participants for whom WhatsApp numbers were available on April 23, 2019 before the researchers learned that phase 5 surveys were not complete. Once the phase 5 surveys were complete, we sent an additional message with the content from the message of April 23 to the respondents not included in the original message. 2Due to limits that were unexpectedly imposed by WhatsApp on bulk communication with our participants, only three messages could be sent to phase 6 respondents from the face-to-face survey who opted to receive them. We had originally planned to send an equal number of messages to phase 5 and 6 respondents, but were constrained by the upcoming election date and the WhatsApp restrictions mentioned above. 3Due to initial difficulties sending messages in Hindi to Mechanical Turk workers via email, the May 7 message had to be resent in the early morning hours of May 8 IST.

A-7 Figure A3: Sample India news headline stimuli

BJP-congenial headlines India’s mom-and-pop merchants lean toward Modi’s BJP in election (pro-BJP; mainstream) PM Modi visits Kumbh, ﬁrst head of state to visit sacred ceremony (pro-BJP; false) Rahul Gandhi greeted with “Modi Modi” slogans as he interacts with students in Pune (anti-INC; mainstream) Congress workers chant “Pakistan Zindabad” slogan in Rajasthan’s Rajsamand (anti-INC; false)

BJP-uncongenial headlines Congress party launches new theme song written by superstar Bollywood composer... (pro-INC; mainstream) Britain’s former Prime Minister Tony Blair says India’s future is safe in Rahul Gandhi’s hands (pro-INC; false) India’s job crisis is worse than people thought — and its government tried to squelch the data (anti-BJP; mainstream) BJP candidate’s trousers fall down while giving speech (anti-BJP; false)

Nationalist headlines After Pulwama, India to stop share of water ﬂowing to Pakistan (India-Pakistan relations; mainstream) Martyred Jawans’ bodies kept in garbage boxes after Naxal attack in Gadchiroli (India-Pakistan relations; false) Report ﬁnds that cow vigilantes killed 44 people in the last three years (Hindu-Muslim relations; mainstream) Saudi prince says he believes Kashmir is “Hindu land” (Hindu-Muslim relations; false)

Fact-checked headlines Yogi Adityanath ﬁlmed giving cash to people for voting for the BJP (false) Kanhaiya Kumar holds campaign rally with photo of the terrorist Afzal Guru (false)

A-8 B Descriptive statistics for sample by compliance status

B.1 Demographic and political measures by compliance status In the following ﬁgures, we show estimated means along various dimensions for both respondents who would take the tips treatment if and only if they are assigned to receive it (“compliers”) and those who would not do so under any circumstances (“never-takers”) computed following the procedure in Marbach and Hangartner (2020).

Figure B1: Proﬁle of compliers in U.S. sample

Affective polarization* ●

Democrat ●

Republican* ●

Trump feelings* ●

Media feelings ●

College* ●

Age 18−29* ●

Age 30−44* ●

Age 45−59* ●

Age 60+* ●

Conspiracy predispositions* ●

Political interest* ●

Political knowledge* ●

Male ●

0.0 0.2 0.4 0.6 0.8 1.0 Mean value

Sample Compliers ● Never−takers

Points show estimated means for the complete sample, respondents who would take the treatment only if assigned to it (“compliers”), and respondents who would not take the treatment even if assigned to it (“never-takers”) computed following Marbach and Hangartner (2020). All variables are rescaled to the [0,1] interval. Lines show 95% conﬁ- dence intervals based on bootstrapped standard errors. Asterisks on variable names indicates that the mean values for compliers and never-takers are signiﬁcantly diferent (p < .05).

B-1 Figure B2: Proﬁle of compliers in India online sample

Scheduled Caste* ●

Scheduled Tribe ●

Other Backward Class ●

General Caste* ●

Other Caste ●

College* ●

Political interest* ●

Political knowledge* ●

Age 18−30* ●

Age 31−45 ●

Age 46−60 ●

Age 61+* ●

Muslim ●

Hindu* ●

WhatsApp use* ●

Male* ●

0.0 0.2 0.4 0.6 0.8 1.0 Mean value

Sample Compliers ● Never−takers Points show estimated means for the complete sample, respondents who would take the treatment only if assigned to it (“compliers”), and respondents who would not take the treatment even if assigned to it (“never-takers”) computed following Marbach and Hangartner (2020). All variables are rescaled to the [0,1] interval. Lines show 95% conﬁ- dence intervals based on bootstrapped standard errors. Asterisks on variable names indicates that the mean values for compliers and never-takers are signiﬁcantly diferent (p < .05).

B-2 Figure B3: Proﬁle of compliers in India face-to-face sample

Scheduled Caste ●

Scheduled Tribe ●

Other Backward Class ●

General Caste ●

Other Caste ●

College ●

Political interest ●

Political knowledge ●

Age 18−30 ●

Age 31−45 ●

Age 46−60 ●

Age 61+ ●

Muslim ●

Hindu ●

WhatsApp use ●

Male ●

0.0 0.2 0.4 0.6 0.8 1.0 Mean value

B-3 B.2 Compliance by false/hyperpartisan website consumption Additionally, a key characteristic that could hypothetically be related to both treatment effectiveness and treatment compliance is the baseline propensity to consume information from false or hyperpartisan websites. Using the same procedure from Marbach and Hangartner (2020), we compare the measured number of visits to false news websites by compliers and never-takers in the U.S. sample before the survey period (N = 1069).1 On average, compliers visited more such sites (0.35 compared to 0.18), but the difference is not statistically distinguishable (0.165, p = 0.08). We measure total number of visits to false or hyperpartisan websites using YouGov Pulse data collected up to the day of each respondent’s wave 1 survey responses (not inclusive) in the post-election period (Nov. 12–Dec. 26, 2018). We define “false or hyperpartisan” websites as one of the 673 domains identified in Allcott, Gentzkow and Yu (2019) as a “fake news” producer as of September 2018 excluding those with print versions (including but not limited to Express, the British tabloid) and also domains that were previously classified by Bakshy, Messing and Adamic (2015) as a source of hard news. In addition, we excluded sites that predominantly feature user-generated content (e.g., online bulletin boards) and political interest groups.

1We report this comparison separately because of missing data induced by respondents who did not choose to share anonymized Pulse web consumption data as part of this study.

B-4 C Supplemental results from U.S. media literacy intervention

C.1 ITT and ATT estimates of literacy intervention by news type and wave

Table C1: Eﬀect of U.S. media literacy intervention on perceived accuracy by news type (Waves 1 and 2)

(a) Intent to treat eﬀects (ITT)

False news Mainstream news Mainstream−false Wave 1 Wave 2 Wave 1 Wave 2 Wave 1 Wave 2 Media literacy intervention -0.196*** -0.080*** -0.046** -0.029 0.146*** 0.050* (0.020) (0.019) (0.017) (0.016) (0.024) (0.020) Constant 0.551*** 0.594*** (0.016) (0.014) Headline ﬁxed eﬀects XXXX N (headlines) 9813 17121 19623 34237 N (respondents) 4907 4282 4907 4282 4907 4281

(b) Average treatment eﬀects on the treated (ATT)

False news Mainstream news Mainstream−false Wave 1 Wave 2 Wave 1 Wave 2 Wave 1 Wave 2 Media literacy intervention -0.296*** -0.121*** -0.072** -0.044 0.223*** 0.076* (0.031) (0.028) (0.026) (0.024) (0.035) (0.030) Constant 0.551*** 0.594*** (0.016) (0.014) Headline ﬁxed eﬀects XXXX N (headlines) 9813 17121 19623 34237 N (respondents) 4907 4282 4907 4282 4907 4281

* p < .05, ** p < .01, *** p < .005 (two-sided). Cell entries are OLS or two-stage least squares coefficients with robust standard errors in parentheses (clustered by respondent for false and mainstream news accuracy). Dependent variables for perceived false and mainstream news accuracy are measured on a 1–4 scale where 1 represents “Not at all accurate” and 4 represents “Very accurate.” The dependent variable for the difference in perceived false versus mainstream news accuracy is calculated at the respondent level as the mean difference in perceived accuracy between all false and all mainstream news headlined viewed.

C-1 C.2 Eﬀect of media literacy intervention on hyperpartisan news headlines

Table C2: Eﬀects of U.S. media literacy intervention on perceived accuracy of hyperpartisan news

Hyperpartisan news Wave 1 Wave 2 Media literacy intervention -0.176*** -0.075*** (0.020) (0.018) Headline ﬁxed eﬀects XX N (headlines) 9813 17121 N (respondents) 4907 4282

* p < .05, ** p < .01, *** p < .005 (two-sided). Data from November/December 2018. Cell entries are OLS co- eﬃcients with robust standard errors in parentheses (clustered by respondent). Dependent variables for perceived hyperpartisan news accuracy are measured on a 1–4 scale where 1 represents “Not at all accurate” and 4 represents “Very accurate.”

C-2 C.3 Balance tests and attrition across waves

Table C3: Balance statistics for U.S. media literacy intervention

Variable Control Media literacy intervention Diﬀerence N Female 0.550 0.542 -0.008 4,907 (0.498) (0.498) (0.014) Nonwhite 0.320 0.313 -0.007 4,907 (0.467) (0.464) (0.013) Age 18–29 0.164 0.177 0.013 4,907 (0.370) (0.382) (0.011) Age 30–44 0.264 0.250 -0.014 4,907 (0.441) (0.433) (0.012) Age 45–59 0.257 0.245 -0.012 4,907 (0.437) (0.430) (0.012) Age 60+ 0.315 0.327 0.013 4,907 (0.464) (0.469) (0.013) College 0.318 0.319 0.001 4,907 (0.466) (0.466) (0.013) Political interest 3.598 3.628 0.030 4,899 (1.168) (1.170) (0.033) Political knowledge 3.064 3.039 -0.026 4,907 (1.581) (1.589) (0.045) Republican 0.356 0.356 -0.000 4,907 (0.479) (0.479) (0.014) Democrat 0.454 0.471 0.017 4,907 (0.498) (0.499) (0.014)

* p < .05, ** p < .01, *** p < .005 (two-sided). Data are from Wave 1. Cell entries are means with robust standard errors in parentheses.

Table C4: Eﬀect of media literacy intervention on attrition in U.S. study

Wave 2 completion Media literacy intervention 0.0041 (0.0095) Constant 0.8708*** (0.0068) N (respondents) 4907

* p < .05, ** p < .01, *** p < .005 (two-sided). Cell entries are OLS coeﬃcients with robust standard errors in parentheses Dependent variable is completion of wave 2.

C-3 Table C5: Eﬀect of demographics on attrition in U.S. study

(1) (2) (3) (4) (5) (6) (7) (8) Mass media trust -0.0022 (0.0053) Media FT -0.0001 (0.0001) Republican 0.0086 (0.0137) Democrat -0.0029 (0.0133) Political knowledge 0.0146*** (0.0030) Nonwhite -0.0185 (0.0105) C-4 Female -0.0092 (0.0095) Age 30–44 0.0554*** (0.0176) Age 45–59 0.1092*** (0.0167) Age 60+ 0.1477*** (0.0156) Perceived false news accuracy (w1) -0.0183** (0.0066) Constant 0.8786*** 0.8777*** 0.8711*** 0.8282*** 0.8787*** 0.8779*** 0.7838*** 0.9082*** (0.0140) (0.0083) (0.0112) (0.0110) (0.0056) (0.0069) (0.0142) (0.0133) N (respondents) 4905 4705 4907 4907 4907 4907 4907 4907

* p < .05, ** p < .01, *** p < .005 (two-sided). OLS models with robust standard errors. Dependent variable is wave 2 completion. Age category indicators, political knowledge, and perceived accuracy of false news all remain significant after the Holm correction for family-wise error rate. C.4 Effect of intervention on binary outcome To provide intuition about the effects, Figure 1 in the main text presents results in an analysis where the outcome has been transformed into a binary variable. Under this coding scheme, the dependent variables for perceived false and mainstream news accuracy take the value of 0 for ratings of “Not at all accurate” or “Not very accurate” and 1 for ratings of “Somewhat accurate” or “Very accurate.” Table C6 shows the model associated with these results.

Table C6: Eﬀect of U.S. media literacy intervention on perceived accuracy by news type (binary)

(a) Intent to treat eﬀects (ITT)

False news Mainstream news Media literacy intervention -0.074*** -0.019* (0.009) (0.008) Headline ﬁxed eﬀects XX N (headlines) 9813 19623 N (respondents) 4907 4907

(b) Average treatment eﬀects on the treated (ATT)

False news Mainstream news Media literacy intervention -0.113*** -0.029* (0.014) (0.012) Headline ﬁxed eﬀects XX N (headlines) 9813 19623 N (respondents) 4907 4907

* p < .05, ** p < .01, *** p < .005 (two-sided). Data from Wave 1 (November/December 2018). Cell entries are OLS or two-stage least squares coeﬃcients with robust standard errors in parentheses (clustered by respondent). Dependent variables for perceived false and mainstream news accuracy take the value of 0 for ratings of “Not at all accurate” or “Not very accurate” and 1 for ratings of “Somewhat accurate” or “Very accurate.”

C-5 C.5 Eﬀect of intervention with alternative treatment receipt In the main text, we provide ATT estimates based on a model in which only only individuals who were able to answer a set of questions about the treatment on the ﬁrst try are considered to have received the treatment. However, respondents in the U.S. sample were allowed to try to answer these questions (after re-reading the media literacy intervention) three separate times. Here, we show results that instead treat anyone who was able to answer each of the questions on any of the three attempts as having received the treatment. The results are consistent with those in the main text.

Table C7: Eﬀect of U.S. media literacy intervention: Alternate treatment receipt deﬁnition

False news Mainstream news Mainstream−false Media literacy intervention -0.221*** -0.052** 0.165*** (0.023) (0.019) (0.026) Constant 0.551*** (0.016) Headline ﬁxed eﬀects XX N (headlines) 9813 19623 N (respondents) 4907 4907 4907

* p < .05, ** p < .01, *** p < .005 (two-sided). Data from November/December 2018. Cell entries are OLS coefficients with robust standard errors in parentheses (clustered by respondent for false and mainstream news accuracy). Dependent variables for perceived false and mainstream news accuracy are measured on a 1–4 scale where 1 represents “Not at all accurate” and 4 represents “Very accurate.” The dependent variable for the difference in perceived false versus mainstream news accuracy is calculated at the respondent level as the mean difference in perceived accuracy between all false and all mainstream news headlined viewed.

C-6 C.6 Eﬀect of intervention with ordered probit In the main text, we estimate the treatment eﬀects using an ordinary least squares model even though the outcome is a four-category variable. Here we show these results are robust to using an ordered probit model instead.

Table C8: Eﬀect of U.S. media literacy intervention on perceived accuracy by news type (ordered probit)

False news Mainstream news Media literacy intervention -0.235*** -0.047* (0.024) (0.018) N (headline) 9813 19623 N (respondent) 4907 4907

* p < .05, ** p < .01, *** p < .005 (two-sided). Data from Wave 1. Cell entries are ordered probit coeﬃcients with robust standard errors in parentheses, clustered by respondent for false and mainstream news accuracy. Dependent variables for perceived false and mainstream news accuracy are measured on a 1–4 scale where 1 represents “Not at all accurate” and 4 represents “Very accurate.”

C-7 C.7 Heterogeneous effects by congeniality One additional concern is that the media literacy intervention will only make individuals skeptical of uncongenial news content that they would already tend to disbelieve (RQ2). In Figure C1 and Table C9, we show that there is insufficient evidence to conclude that the congeniality of the headline moderated the effect of the digital literacy intervention.

Figure C1: Eﬀect of U.S. media literacy intervention on perceived accuracy of false news headline accuracy by partisan congeniality

(a) Partisan congeniality: Wave 1 (b) Partisan congeniality: Wave 2

Very accurate Very accurate

Somewhat accurate Somewhat accurate

Not very accurate Not very accurate

Not at all accurate Not at all accurate Uncongenial Congenial Uncongenial Congenial

Control Media literacy intervention Control Media literacy intervention

Average respondent ratings of two false news headlines viewed in wave 1 and four false news headlines viewed in wave 2. Headlines were selected randomly in wave 1, balanced by partisan congeniality, and presented in random order. Congeniality results are presented only for Democrats and Republicans (including leaners). Error bars are 95% conﬁdence intervals for the mean. Headline details and coding are provided in the “Survey instrument” document that will be included in the replication archive for this study.

C-8 Table C9: Eﬀects of U.S. media literacy intervention on perceived accuracy by news type and congeniality

False news Mainstream news Wave 1 Wave 2 Wave 1 Wave 2 Media literacy intervention -0.171*** -0.086*** -0.039 -0.043* (0.024) (0.023) (0.020) (0.019) Partisan congenial 0.502*** 0.543*** 0.549*** 0.580*** (0.026) (0.023) (0.020) (0.018) Media literacy intervention × congenial -0.072* 0.003 -0.027 0.023 (0.036) (0.032) (0.028) (0.026) Headline ﬁxed eﬀects XXXX N (headlines) 9813 17121 19623 34237 N (respondents) 4907 4282 4907 4282

C-9 C.8 Heterogeneous effects by prior exposure Previous research has found that individuals are more likely to believe in false stories simply as a function of prior exposure (Pennycook, Cannon and Rand 2018). We can test this in the U.S. sample because respondents in wave 1 were randomly exposed to only eight headlines while in wave 2 they received all 16. Specifically, we test to see if this prior exposure effect moderates the effect of the media literacy intervention. Table C10 shows that we have insufficient evidence to conclude that effect of the media literacy intervention was moderated by previous exposure.

Table C10: Eﬀects of U.S. media literacy intervention on perceived accuracy by news type and prior exposure

False news Mainstream news Wave 2 Wave 2 Media literacy intervention -0.062*** -0.031 (0.021) (0.018) Prior exposure 0.092*** 0.083*** (0.015) (0.011) Media literacy intervention × prior exposure -0.035 0.004 (0.021) (0.016) Headline ﬁxed eﬀects XX N (headlines) 17121 34237 N (respondents) 4281 4282

C-10 C.9 Heterogeneous effects by source prominence and headline plausibility We conducted an exploratory analysis to determine the relationship between the baseline plausibility of various news headlines and headline-specific treatment effects. Specifically, we assessed the perceived baseline accuracy of each headline using the average score given to that headline by the control group. In Table C11, we then calculate the effect of the media literacy intervention for each headline. These numbers are plotted together in Figure C2. Table C12 formally tests for the moderating role of source prominence on accuracy beliefs. We find that the media literacy intervention had a substantial negative effect on the perceived accuracy of headlines from low-prominence sources (e.g., Politico) and a smaller positive effect on the perceived accuracy of headlines from high-prominence scores (e.g., the Washington Post).

Figure C2: Eﬀect of media literacy intervention by baseline headline accuracy (U.S. sample)

-.1

-.2 Treatment effect literacy) (media

-.3 1.5 2 2.5 3 Perceived baseline accuracy

False Hyperpartisan Mainstream

Plot depicts mean accuracy ratings in the control condition and estimated media literacy intervention treatment eﬀects (ITT) for each headline in wave 1 of the U.S. study.

C-11 Table C11: Eﬀect of U.S. media literacy intervention on perceived accuracy (question-level)

(a) False and hyperpartisan headlines

Pro-D false 1 Pro-R hyper 2 Pro-D hyper 2 Pro-D hyper 1 Pro-R false 2 Pro-R false 1 Pro-D false 2 Pro-R hyper 1 Media literacy intervention -0.217*** -0.198*** -0.118*** -0.268*** -0.255*** -0.236*** -0.075 -0.123*** (0.035) (0.036) (0.039) (0.039) (0.036) (0.041) (0.040) (0.041) Constant 1.749*** 1.866*** 1.915*** 1.945*** 2.121*** 2.126*** 2.131*** 2.270*** (0.026) (0.026) (0.028) (0.029) (0.026) (0.030) (0.029) (0.029) N (respondents) 2472 2447 2479 2427 2445 2462 2434 2460 C-12 (b) Mainstream headlines

Pro-R 2 Pro-D 1 Pro-R 1 Pro-D 4 Pro-D 2 Pro-R 3 Pro-D 3 Pro-R 4 Media literacy intervention -0.195*** -0.174*** -0.036 -0.024 -0.176*** 0.144*** -0.010 0.097** (0.036) (0.040) (0.038) (0.043) (0.043) (0.035) (0.042) (0.035) Constant 2.297*** 2.319*** 2.438*** 2.526*** 2.543*** 2.787*** 2.795*** 2.933*** (0.025) (0.029) (0.026) (0.030) (0.030) (0.025) (0.030) (0.024) N (respondents) 2368 2436 2538 2468 2468 2402 2438 2505

* p < .05, ** p < .01, *** p < .005 (two-sided). Data are from Wave 1. Cell entries are OLS coeﬃcients with robust standard errors in parentheses. Dependent variables for perceived false and mainstream news accuracy are measured on a 1–4 scale where 1 represents “Not at all accurate” and 4 represents “Very accurate.” Headline details and coding are provided in the “Survey instrument” document that will be included in the replication archive for this study. Table C12: Eﬀects of U.S. media literacy intervention on perceived accuracy of mainstream news by source prominence

Mainstream news Wave 1 Media literacy intervention 0.050* (0.020) Low prominence source -0.360*** (0.016) Media literacy intervention × low prominence source -0.195*** (0.024) Constant 2.761*** (0.014) N 19623 N (respondent) 4907

* p < .05, ** p < .01, *** p < .005 (two-sided). Data from November/December 2018. Cell entries are OLS co- eﬃcients with robust standard errors in parentheses clustered by respondent. The dependent variable for perceived mainstream news accuracy is measured on a 1–4 scale where 1 represents “Not at all accurate” and 4 represents “Very accurate.”

C-13 C.10 Eﬀects of intervention on sharing intention and news consumption

Table C13: Eﬀects of U.S. media literacy intervention on sharing intention by news type

False news Mainstream news Hyperpartisan news Wave 1 Wave 2 Wave 1 Wave 2 Wave 1 Wave 2 Media literacy intervention -0.027 0.000 0.045* 0.027 -0.058*** 0.003 (0.020) (0.021) (0.021) (0.022) (0.020) (0.021) Headline ﬁxed eﬀects XXXXXX N (headline) 8592 14961 17180 29917 8592 14956 N (respondent) 4296 3741 4296 3741 4296 3740

* p < .05, ** p < .01, *** p < .005 (two-sided). Data from November/December 2018. Cell entries are OLS coeﬃ- cients with robust standard errors in parentheses clustered by respondent. Dependent variables for self-reported sharing intention are measured on a 1–4 scale where 4 represents a response of “Very likely” and 1 represents a response of “Not at all likely.”

C-14 In Tables C14 and C15, post-treatment news consumption is deﬁned according to our pre- analysis plan. For behavioral outcomes, we coded respondents’ Pulse data for the seven days after they completed the wave 1 survey as detailed in Section A.1.2.

Table C14: Eﬀects of U.S. media literacy intervention on subsequent news consumption (ITT)

False news Fact-checking Mainstream news Binary Count Binary Count Binary Count Media literacy interv. 0.0128 0.0258 0.0092 0.0401 0.0027 0.5858 (0.0154) (0.0873) (0.0127) (0.0542) (0.0319) (2.7810) Constant 0.0554*** 0.1971*** 0.0370*** 0.0945** 0.5154*** 11.4784*** (0.0104) (0.0621) (0.0086) (0.0349) (0.0227) (1.6597) N (respondent) 985 985 985 985 985 985

* p < .05, ** p < .01, *** p < .005 (two-sided). Data from November/December 2018. Cell entries are OLS coeﬃ- cients with robust standard errors in parentheses. Dependent variables indicate binary and pageview count measures of consumption of false news, fact-checking, and mainstream news websites on the day of survey completion after completing the survey and in the subsequent seven days.

Table C15: Eﬀects of U.S. media literacy intervention on subsequent news consumption (ATT)

False news Fact-checking Mainstream news Binary Count Binary Count Binary Count Media literacy interv. 0.0168 0.0337 0.0121 0.0524 0.0035 0.7657 (0.0200) (0.1139) (0.0166) (0.0707) (0.0416) (3.6306) Constant 0.0554*** 0.1971*** 0.0370*** 0.0945** 0.5154*** 11.4784*** (0.0104) (0.0620) (0.0085) (0.0348) (0.0226) (1.6580) N (respondent) 985 985 985 985 985 985

∗ p < .05, ∗∗ p < .01, ∗∗∗ p < .005 (two-sided). Data from November/December 2018. Cell entries are 2SLS coeﬃcients with robust standard errors in parentheses. Dependent variables indicate binary and pageview count measures of consumption of false news, fact-checking, and mainstream news websites on the day of survey completion after completing the survey and in the subsequent seven days.

C-15 D Additional results from India media literacy intervention

D.1 Descriptive statistics for both samples

Table D1: Descriptive statistics for online and face-to-face India surveys

Variable Online Face-to-face Diﬀerence N Female 0.283 0.363 0.080*** 7,017 (0.450) (0.481) (0.011) Low caste 0.418 0.741 0.323*** 7,017 (0.493) (0.438) (0.011) Age 18–29 0.478 0.330 -0.148*** 5,754 (0.500) (0.470) (0.013) Age 30–44 0.353 0.338 -0.015 5,754 (0.478) (0.473) (0.013) Age 45–59 0.062 0.208 0.146*** 5,754 (0.241) (0.406) (0.009) Age 60+ 0.107 0.123 0.017 5,754 (0.309) (0.329) (0.009) WhatsApp user 0.901 0.112 -0.790*** 7,017 (0.298) (0.315) (0.007) College 0.757 0.062 -0.695*** 7,017 (0.429) (0.241) (0.008) Muslim 0.105 0.265 0.159*** 7,017 (0.307) (0.441) (0.009) Hindu 0.720 0.727 0.006 7,017 (0.449) (0.446) (0.011) Political interest 3.744 2.859 -0.885*** 6,649 (1.091) (1.572) (0.033) Political knowledge 2.849 2.116 -0.733*** 7,017 (1.181) (1.141) (0.028) BJP support 0.423 0.461 0.038** 7,017 (0.494) (0.499) (0.012) BJP oppose 0.271 0.266 -0.005 7,017 (0.445) (0.442) (0.011)

* p < .05, ** p < .01, *** p < .005 (two-sided). Data are from Wave 1 (April/May 2019). Cell entries are means with robust standard errors in parentheses. Signiﬁcance tests were adjusted using the Holm correction for family-wise error rate.

D-1 D.2 ITT and ATT estimates by news type, wave, and sample

D-2 Table D2: Eﬀects of India media literacy intervention on perceived accuracy by news type

(a) Intent to treat eﬀects (ITT)

Online sample Face-to-face sample False news Mainstream news Mainstream−false False news Mainstream news Mainstream−false Wave 1 Wave 2 Wave 1 Wave 2 Wave 1 Wave 2 Wave 1 Wave 2 Wave 1 Wave 2 Wave 1 Wave 2 Media literacy intervention -0.126*** 0.007 -0.071** -0.062 0.063* -0.070 -0.007 -0.013 0.002 -0.013 0.006 -0.007 (0.026) (0.041) (0.025) (0.039) (0.025) (0.040) (0.024) (0.029) (0.024) (0.032) (0.030) (0.042) Constant 0.361*** 0.507*** 0.237*** 0.773*** (0.017) (0.029) (0.021) (0.029)

Headline ﬁxed eﬀects XXXXXXXX N (headlines) 17031 11497 17163 6844 13712 13881 13969 8421 N (respondents) 3160 1291 3160 1291 3160 1291 3140 2295 3140 2295 3140 2295

D-3 (b) Average treatment eﬀects on the treated (ATT)

Online sample Face-to-face sample False news Mainstream news Mainstream−false False news Mainstream news Mainstream−false Wave 1 Wave 2 Wave 1 Wave 2 Wave 1 Wave 2 Wave 1 Wave 2 Wave 1 Wave 2 Wave 1 Wave 2 Media literacy intervention -0.470*** 0.021 -0.259** -0.192 0.221* -0.206 -0.035 -0.056 0.011 -0.057 0.028 -0.030 (0.097) (0.126) (0.095) (0.121) (0.088) (0.120) (0.113) (0.121) (0.113) (0.138) (0.138) (0.173) Constant 0.361*** 0.507*** 0.237*** 0.773*** (0.017) (0.029) (0.021) (0.029)

Headline ﬁxed eﬀects XXXXXXXX N (headlines) 17031 11497 17163 6844 13712 13881 13969 8421 N (respondents) 3160 1291 3160 1291 3160 1291 3140 2295 3140 2295 3140 2295

* p < .05, ** p < .01, *** p < .005 (two-sided). Data from Waves 1 and 2 (April/May 2019), extra factchecked headlines omitted from Wave 2. Cell entries are OLS coefficients with robust standard errors in parentheses (clustered by respondent for false and mainstream news accuracy). Dependent variables for perceived false and mainstream news accuracy are measured on a 1–4 scale where 1 represents “Not at all accurate” and 4 represents “Very accurate.” The dependent variable for the difference in perceived false versus mainstream news accuracy is calculated at the respondent level as the mean difference in perceived accuracy between all false and all mainstream news headlined viewed. Table D3: Effect of india media literacy intervention on perceived accuracy by news type (false news indicator)

Online sample Face-to-face sample Media literacy intervention -0.072*** 0.001 (0.025) (0.024) False news -0.344*** -0.249*** (0.016) (0.019) Media literacy × false news -0.055* -0.011 (0.024) (0.027) Constant 2.745*** 3.057*** (0.018) (0.017) Headline ﬁxed eﬀects N (headlines) 34194 27681 N (respondents) 3199 3441

* p < .05, ** p < .01, *** p < .005 (two-sided). Data from Wave 1 (April/May 2019). Cell entries are OLS with robust standard errors in parentheses (clustered by respondents for false and mainstream news accuracy). Dependent variables for perceived false and mainstream news accuracy are measured on a 1–4 scale where 1 represents “Not at all accurate” and 4 represents “Very accurate.”

D-4 D.3 Balance tests and attrition across waves

Table D4: Balance statistics for media literacy intervention (a) Online sample Variable Control Media literacy intervention Diﬀerence N Female 0.285 0.280 -0.005 3,273 (0.452) (0.449) (0.016) Low caste 0.420 0.415 -0.005 3,273 (0.494) (0.493) (0.017) Age 18–29 0.490 0.465 -0.025 3,273 (0.500) (0.499) (0.017) Age 30–44 0.346 0.361 0.016 3,273 (0.476) (0.481) (0.017) Age 45–59 0.063 0.061 -0.003 3,273 (0.244) (0.239) (0.008) Age 60+ 0.100 0.113 0.013 3,273 (0.301) (0.317) (0.011) College 0.765 0.749 -0.016 3,273 (0.424) (0.434) (0.015) Muslim 0.114 0.096 -0.018 3,273 (0.318) (0.295) (0.011) Hindu 0.726 0.715 -0.011 3,273 (0.446) (0.452) (0.016) Political interest 3.710 3.779 0.069 3,188 (1.110) (1.070) (0.039) Political knowledge 2.877 2.821 -0.056 3,273 (1.172) (1.191) (0.041) BJP supporter 0.427 0.418 -0.010 3,273 (0.495) (0.493) (0.017) BJP opponent 0.258 0.285 0.027 3,273 (0.437) (0.451) (0.016) (b) Face-to-face sample Variable Control Media literacy intervention Diﬀerence N Female 0.368 0.358 -0.010 3,744 (0.482) (0.479) (0.016) Low caste 0.737 0.744 0.007 3,744 (0.440) (0.436) (0.014) Age 18–29 0.227 0.210 -0.018 3,744 (0.419) (0.407) (0.014) Age 30–44 0.221 0.227 0.006 3,744 (0.415) (0.419) (0.014) Age 45–59 0.130 0.147 0.017 3,744 (0.336) (0.354) (0.011) Age 60+ 0.084 0.079 -0.005 3,744 (0.277) (0.270) (0.009) College 0.063 0.060 -0.003 3,744 (0.243) (0.238) (0.008) Muslim 0.271 0.258 -0.013 3,744 (0.445) (0.438) (0.014) Hindu 0.719 0.734 0.015 3,744 (0.449) (0.442) (0.015) Political interest 2.833 2.887 0.054 3,461 (1.574) (1.569) (0.053) Political knowledge 2.085 2.149 0.064 3,744 (1.140) (1.142) (0.037) BJP supporter 0.451 0.471 0.020 3,744 (0.498) (0.499) (0.016) BJP opponent 0.275 0.257 -0.018 3,744 (0.447) (0.437) (0.014)

* p < .05, ** p < .01, *** p < .005 (two-sided). Data are from Wave 1 (April/May 2019). Cell entries are means with robust standard errors in parentheses.

D-5 Table D5: Eﬀect of demographics on attrition in India study

(a) Online sample

Variable (1) (2) (3) (4) (5) (6) (7) (8) (9) (10) Female 0.003 (0.019) Low Caste 0.039 (0.017) Age 30–44 -0.100*** (0.019) Age 45–59 -0.151*** (0.037) Age 60+ 0.117*** (0.027) College -0.161*** (0.019) Muslim -0.018 (0.028) Hindu -0.059*** (0.019) Political interest -0.031* (0.008) Political knowledge -0.077*** (0.007) BJP support -0.061*** (0.017) BJP oppose -0.015 (0.019) Constant 0.581*** 0.566*** 0.614*** 0.704*** 0.584*** 0.624*** 0.693*** 0.801*** 0.607*** 0.586*** (0.010) (0.011) (0.012) (0.016) (0.009) (0.016) (0.031) (0.021) (0.011) (0.010) N (respondents) 3273 3273 3273 3273 3273 3273 3188 3273 3273 3273

(b) Face-to-face sample

Variable (1) (2) (3) (4) (5) (6) (7) (8) (9) (10) Female -0.000 (0.014) Low Caste -0.002 (0.015) Age 30–44 -0.003 (0.021) Age 45–59 -0.059 (0.023) Age 60+ -0.047 (0.027) College -0.045 (0.026) Muslim 0.071*** (0.016) Hindu -0.068*** (0.016) Political interest -0.008 (0.004) Political knowledge 0.008 (0.006) BJP support -0.045** (0.013) BJP oppose 0.045* (0.016) Constant 0.220*** 0.221*** 0.247*** 0.223*** 0.201*** 0.270*** 0.239*** 0.204*** 0.241*** 0.208*** (0.008) (0.013) (0.015) (0.007) (0.008) (0.014) (0.015) (0.014) (0.010) (0.008) N (respondents) 3744 3744 2481 3744 3744 3744 3461 3744 3744 3744

* p < .05, ** p < .01, *** p < .005 (two-sided). Data are from Wave 2 (May 2019). OLS models with robust standard errors. Dependent variable is attrition (failure to complete wave 2). Signiﬁcance tests were adjusted using the Holm correction for family-wise error rate. D-6 Table D6: Eﬀect of media literacy intervention on attrition by fact-checking condition

Online sample Face-to-face sample Media literacy intervention -0.054 0.005 (0.030) (0.019) Fact-check condition 0.011 0.024 (0.029) (0.019) Media literacy × fact-check condition 0.005 -0.003 (0.042) (0.027) Constant 0.599*** 0.206*** (0.021) (0.013) N (respondents) 2207 3744

* p < .05, ** p < .01, *** p < .005 (two-sided). Data are from Wave 2 (May 2019). Cell entries are OLS coeﬃcients with robust standard errors in parentheses. The dependent variable is attrition (failure to complete wave 2).

D-7 D.4 Eﬀect of intervention on binary outcome To provide intuition about the eﬀects, Figure 2 in the main text presents results in an analysis where the outcome has been transformed into a binary variable. Under this coding scheme, the dependent variables for perceived false and mainstream news accuracy take the value of 0 for ratings of “Not at all accurate” or “Not very accurate” and 1 for ratings of “Somewhat accurate” or “Very accurate.” Table D7 shows the model associated with these results.

Table D7: Eﬀect of India media literacy intervention on perceived accuracy by news type (binary)

(a) Intent to treat eﬀects (ITT)

Online sample Face-to-face sample False news Mainstream news False news Mainstream news Media literacy intervention -0.051*** -0.026* -0.003 0.005 (0.012) (0.012) (0.009) (0.009) Headline ﬁxed eﬀects XX N (headlines) 17031 17163 13712 13969 N (respondents) 3160 3160 3140 3140

(b) Average treatment eﬀects on the treated (ATT)

Online sample Face-to-face sample False news Mainstream news False news Mainstream news Media literacy intervention -0.191*** -0.095* -0.015 0.024 (0.043) (0.043) (0.042) (0.043) Headline ﬁxed eﬀects XX N (headlines) 17031 17163 13712 13969 N (respondents) 3160 3160 3140 3140

* p < .05, ** p < .01, *** p < .005 (two-sided). Data from Wave 1 (April/May 2019). Cell entries are OLS or two-stage least squares coeﬃcients with robust standard errors in parentheses (clustered by respondent). Dependent variables for perceived false and mainstream news accuracy take the value of 0 for ratings of “Not at all accurate” or “Not very accurate” and 1 for ratings of “Somewhat accurate” or “Very accurate.”

D-8 D.5 Eﬀect of intervention with ordered probit

Table D8: Eﬀect of India media literacy intervention on perceived accuracy by news type (ordered probit)

Online sample Face-to-face sample False news Mainstream news False news Mainstream news Media literacy intervention -0.126*** -0.075*** -0.006 0.001 (0.026) (0.026) (0.023) (0.025) Headline ﬁxed eﬀects XXXX N (headlines) 17031 17163 13712 13969 N (respondents) 3177 3182 3267 3314

* p < .05, ** p < .01, *** p < .005 (two-sided). Data from Wave 1 (April/May 2019). Cell entries are ordered probit coeﬃcients with robust standard errors in parentheses, clustered by respondent for false and mainstream news accuracy. Dependent variables for perceived false and mainstream news accuracy are measured on a 1–4 scale where 1 represents “Not at all accurate” and 4 represents “Very accurate.”

D-9 D.6 Heterogeneous effects by congeniality One additional concern is that the media literacy intervention will only make individuals skeptical of uncongenial news content that they would already tend to disbelieve (RQ2). In Figure D1 and Table D9, we show that there is insufficient evidence to conclude that the congeniality of the headline moderated the effect of the digital literacy intervention.

Figure D1: Eﬀect of India media literacy intervention on perceived accuracy of false news headline accuracy by partisan congeniality

(a) Online (b) Face-to-face

Very accurate Very accurate

Somewhat accurate Somewhat accurate

Not very accurate Not very accurate

Not at all accurate Not at all accurate Uncongenial Congenial Uncongenial Congenial

Control Media literacy intervention Control Media literacy intervention

Average respondent ratings of false news headlines. Headlines were balanced by partisan congeniality and presented in random order. Congeniality results are presented for BJP supporters and BJP opponents. Error bars are 95% conﬁdence intervals for the mean.

D-10 Table D9: Eﬀect of tips on perceived accuracy by partisan congeniality

Online sample Face-to-face sample False Mainstream False Mainstream Media literacy intervention -0.113*** -0.066* -0.009 0.018 (0.029) (0.028) (0.028) (0.027) Congenial headline 0.426*** 0.332*** 0.451*** 0.256*** (0.032) (0.029) (0.037) (0.033) Media literacy intervention × congenial headline -0.064 -0.026 0.016 -0.057 (0.041) (0.038) (0.048) (0.043)

Headline ﬁxed eﬀects XXXX N (headlines) 17031 17163 13712 13969 N (respondents) 3177 3182 3267 3314

* p < .05, ** p < .01, *** p < .005 (two-sided). Data are from Wave 1 (April/May 2019). Cell entries are OLS coef- ﬁcients with robust standard errors in parentheses (clustered by respondent for false and mainstream news accuracy). Dependent variables for perceived false and mainstream news accuracy are measured on a 1–4 scale where 1 represents “Not at all accurate” and 4 represents “Very accurate.” Headlines are coded as congenial if either respondent is a BJP supporter and the headline makes claims in favor of the BJP’s interests or if the respondent is a BJP opponent and the headline makes claims against the BJP’s interests.

D-11 D.7 Robustness to inclusion of additional wave 2 false headlines In wave 2 of the India surveys, additional headlines were included on the survey that were part of a separate fact-checking experiment (see Section A.2.5 in the Online Appendix). Here we show that our Wave 2 estimates are robust to the inclusion or exclusion of these headlines.

D-12 Table D10: Eﬀects of India media literacy intervention on perceived accuracy by news type, comparison of Wave 2 (W2) and Wave 2 with extra factchecked questions (W2+FC)

(a) Intent to treat eﬀects (ITT)

Online sample Face-to-face sample False news Mainstream news Mainstream−false False news Mainstream news Mainstream−false W2 W2+FC W2 W2+FC W2 W2+FC W2 W2+FC W2 W2+FC W2 W2+FC Media literacy intervention 0.009 0.007 -0.062 -0.062 -0.064 -0.070 -0.022 -0.013 -0.013 -0.013 0.019 -0.007 (0.042) (0.041) (0.039) (0.039) (0.041) (0.040) (0.031) (0.029) (0.032) (0.032) (0.045) (0.042) Constant 0.499*** 0.507*** 0.742*** 0.773*** (0.029) (0.029) (0.032) (0.029)

Headline fixed effects XXXXXXXX N (headlines) 6938 11497 6844 6844 8434 13881 8421 8421 N (respondents) 1304 1308 1298 1298 1289 1291 2455 2578 2412 2412 2202 2295 D-13 (b) Average treatment effects on the treated (ATT)

Online sample Face-to-face sample False news Mainstream news Mainstream−false False news Mainstream news Mainstream−false W2 W2+FC W2 W2+FC W2 W2+FC W2 W2+FC W2 W2+FC W2 W2+FC Media literacy intervention 0.028 0.021 -0.192 -0.192 -0.189 -0.206 -0.093 -0.056 -0.057 -0.057 0.077 -0.030 (0.126) (0.126) (0.121) (0.121) (0.121) (0.120) (0.129) (0.121) (0.138) (0.138) (0.186) (0.173) Constant 0.499*** 0.507*** 0.742*** 0.773*** (0.029) (0.029) (0.032) (0.029)

Headline ﬁxed eﬀects XXXXXXXX N (headlines) 6938 11497 6844 6844 8434 13881 8421 8421 N (respondents) 1304 1308 1298 1298 1289 1291 2455 2578 2412 2412 2202 2295

* p < .05, ** p < .01, *** p < .005 (two-sided). Data from Wave 2 (May 2019). Cell entries are OLS coefficients with robust standard errors in parentheses (clustered by respondent for false and mainstream news accuracy). Dependent variables for perceived false and mainstream news accuracy are measured on a 1–4 scale where 1 represents “Not at all accurate” and 4 represents “Very accurate.” The dependent variable for the difference in perceived false versus mainstream news accuracy is calculated at the respondent level as the mean difference in perceived accuracy between all false and all mainstream news headlined viewed. D.8 Heterogeneous effects by headline plausibility We conducted an exploratory analysis to determine the relationship between the baseline plausibility of various news headlines and headline-specific treatment effects. Specifically, we assessed the perceived baseline accuracy of each headline using the average score given to that headline by the control group. In Table D11, we then calculate the effect of the media literacy intervention for each headline. These numbers are plotted together in Figure D2.

Figure D2: Eﬀect of media literacy intervention by baseline headline accuracy (India online sample)

-.05

-.1 Treatment effect literacy) (media -.15

2.2 2.4 2.6 2.8 Perceived baseline accuracy

False Mainstream

Plot depicts mean accuracy ratings in the control condition and estimated media literacy intervention treatment eﬀects (ITT) for each headline in wave 1 of the India online study.

D-14 Table D11: Eﬀect of India media literacy intervention on perceived accuracy (question-level)

(a) False headlines

Online sample Face-to-face sample Anti-BJP 2 Anti-BJP 1 Pro-BJP 2 Nationalist 2 Nationalist 1 Pro-BJP 1 Anti-BJP 2 Pro-BJP 2 Anti-BJP 1 Nationalist 1 Nationalist 2 Pro-BJP 1 Media literacy intervention -0.070 -0.089* -0.099* -0.134*** -0.210*** -0.157*** -0.032 0.059 -0.006 -0.019 -0.044 0.001 (0.040) (0.040) (0.041) (0.041) (0.042) (0.040) (0.052) (0.055) (0.056) (0.048) (0.046) (0.044) Constant 2.128*** 2.185*** 2.257*** 2.486*** 2.661*** 2.702*** 2.279*** 2.322*** 2.474*** 3.182*** 3.192*** 3.306*** (0.028) (0.028) (0.029) (0.029) (0.030) (0.029) (0.037) (0.039) (0.040) (0.034) (0.032) (0.031) N (respondents) 2852 2924 2789 2845 2836 2785 2289 2260 2056 2320 2392 2395 D-15 (b) Mainstream headlines

Online sample Face-to-face sample Anti-BJP 2 Anti-BJP 1 Pro-BJP 2 Nationalist 2 Pro-BJP 1 Nationalist 1 Anti-BJP 2 Pro-BJP 2 Anti-BJP 1 Nationalist 2 Nationalist 1 Pro-BJP 1 Media literacy intervention -0.102** -0.001 -0.096* -0.045 -0.078* -0.107** 0.080 -0.062 -0.034 0.043 -0.004 -0.010 (0.038) (0.039) (0.040) (0.039) (0.040) (0.039) (0.053) (0.055) (0.048) (0.042) (0.043) (0.044) Constant 2.548*** 2.615*** 2.719*** 2.770*** 2.848*** 2.960*** 2.706*** 2.793*** 2.942*** 3.212*** 3.274*** 3.315*** (0.027) (0.028) (0.028) (0.027) (0.028) (0.027) (0.038) (0.039) (0.034) (0.030) (0.030) (0.031) N (respondents) 2681 2949 2875 2959 2752 2947 2006 2095 2468 2760 2434 2206

* p < .05, ** p < .01, *** p < .005 (two-sided). Data are from Wave 1 (April/May 2019). Cell entries are OLS coeﬃcients with robust standard errors in parentheses. Dependent variables for perceived false and mainstream news accuracy are measured on a 1–4 scale where 1 represents “Not at all accurate” and 4 represents “Very accurate.” Headline details and coding are provided in the “Survey instrument” document that will be included in the replication archive for this study. D.9 Eﬀect of media literacy intervention by WhatsApp use and sample

Table D12: Eﬀect of media literacy intervention by WhatsApp use and sample (India)

Mainstream - false difference (1) (2) Media literacy intervention -0.006 -0.005 (0.030) (0.071) WhatsApp use 0.158*** 0.226*** (0.028) (0.050) Media literacy intervention × WhatsApp use 0.076 0.073 (0.039) (0.076) Face-to-face sample 0.068 (0.052) Media literacy intervention × face-to-face sample -0.002 (0.078) WhatsApp use × face-to-face sample -0.128 (0.079) Media literacy intervention × WhatsApp use × face-to-face sample 0.023 (0.117) Constant 0.218*** 0.157*** (0.021) (0.047) Tips effect: WhatsApp users (both) 0.071** (0.025) Tips effect: WhatsApp users (online) 0.068* (0.027) Tips effect: WhatsApp users (FTF) 0.090 (0.083) N 6300 6300

* p < .05, ** p < .01, *** p < .005 (two-sided). Data are from Wave 1 (April/May 2019). Cell entries are OLS coeﬃcients with robust standard errors in parentheses. The dependent variables is the diﬀerence in the perceived accuracy of mainstream and false news headlines measured on a 1–4 scale where 1 represents “Not at all accurate” and 4 represents “Very accurate.”

D-16 Figure D3: Eﬀect of media literacy intervention by WhatsApp use and sample

0 Online Face-to-face Online Face-to-face

Not WhatsApp user WhatsApp user

Control Media literacy intervention

Average respondent-level diﬀerence between mainstream and false news headlines in the India online and face-to-face studies by WhatsApp usage and assignment to the media literacy intervention condition or the control condition.

D-17 D.10 Pooled analysis of India samples

Table D13: Eﬀect of India media literacy intervention on perceived accuracy by sample

(a) Intent to treat eﬀects (ITT)

False news Mainstream news Mainstream−false Media literacy intervention -0.125*** -0.071** 0.063* (0.026) (0.025) (0.025) Face-to-face sample 0.394*** 0.303*** -0.124*** (0.025) (0.025) (0.028) Media literacy × face-to-face 0.118*** 0.072* -0.057 (0.035) (0.035) (0.039) Constant 0.361*** (0.017) Headline ﬁxed eﬀects XX N (headlines) 30743 31132 N (respondents) 6300 6300 6300

(b) Average treatment eﬀects on the treated (ATT)

False news Mainstream news Mainstream−false Media literacy intervention -0.466*** -0.259** 0.221* (0.097) (0.095) (0.088) Face-to-face sample 0.394*** 0.303*** -0.124*** (0.025) (0.025) (0.028) Media literacy × face-to-face 0.428*** 0.266 -0.194 (0.149) (0.148) (0.163) Constant 0.361*** (0.017) Headline ﬁxed eﬀects XX N (headlines) 30743 31132 N (respondents) 6300 6300 6300

* p < .05, ** p < .01, *** p < .005 (two-sided). Data from Wave 1 (April/May 2019). Cell entries are OLS or two-stage least squares coefficients with robust standard errors in parentheses (clustered by respondent for false and mainstream news accuracy). Dependent variables for perceived false and mainstream news accuracy are measured on a 1–4 scale where 1 represents “Not at all accurate” and 4 represents “Very accurate.” The dependent variable for the difference in perceived false versus mainstream news accuracy is calculated at the respondent level as the mean difference in perceived accuracy between all false and all mainstream news headlined viewed.

D-18 E Benchmarking eﬀect sizes

It is useful to benchmark our ﬁndings against relevant interventions in prior studies. The comparison set is relatively small, however, and concentrated in the U.S. and Europe. First, identifying false information is a relatively new concern in the media literacy scholarship — prior media literacy interventions often addressed topics and outcomes that are quite diﬀerent from ours (Jeong, Cho and Hwang 2012). In addition, many recent studies in this vein are correlational and are also excluded (Amazeen and Bucy 2019; Kahne and Bowyer 2017; Craft, Ashley and Maksl 2017; Vraga and Tully 2019). We summarize the experimental studies that are most comparable to ours in Table E1 (Clayton et al. 2019; Lutzke et al. 2019; Roozenbeek and van der Linden 2019):1

Table E1: Selected related interventions and outcomes

Study Country Mode/sample N Intervention Eﬀect size India face-to-face study India Face-to-face representative 3,140 Facebook/WhatsApp news tips d = .00 Lutzke et al. (2019) U.S. Online convenience 2,750 News literacy tips (reinforcement) d = .07† Clayton et al. (2019) U.S. Online non-representative 2,994 Misinformation warning d = .08 Lutzke et al. (2019) U.S. Online convenience 2,750 News literacy tips d = .10† India online study India Online convenience 3,160 Facebook/WhatsApp news tips d = .11 Roozenbeck et al. (2019) Multiple Online convenience 1,770 Fake news creation game d = .16‡ U.S. study U.S. Online representative 4,907 Facebook news tips d = .20

† Recalculated by the authors from the full Lutzke et al. (2019) replication dataset to avoid potential post-treatment sample restrictions. ‡ Sample and eﬀect size for polarizing headlines.

1We exclude Tully, Vraga and Bode (2019) because it conditions sample inclusion on recall of experimental treatments, which can induce post-treatment bias Montgomery, Nyhan and Torres (2018).

E-1 F Pooled analysis (U.S. and India samples)

Figure 3 in the main text includes a pooled model for all three studies (including ﬁxed eﬀects for each sample). The full results for this model are shown in Table F1.

Table F1: Pooled estimates

False news Mainstream news Mainstream − false Media literacy intervention -0.1029*** -0.0412*** 0.0833*** (0.0145) (0.0126) (0.0151) Constant 0.3506*** (0.0145) Headline ﬁxed eﬀects XX N (headlines) 40556 50755 N (respondents) 11351 11403 11207

* p < .05, ** p < .01, *** p < .005 (two-sided). OLS models with robust standard errors, clustered by respondent for false and mainstream news accuracy. Dependent variables for perceived false and mainstream news accuracy are measured on a 1–4 scale where 1 represents “Not at all accurate” and 4 represents “Very accurate.” Models include headline and sample ﬁxed eﬀects.

F-1 G Estimating the net eﬀect of the media literacy intervention

One normative concern is that the media literacy intervention could lead individuals to distrust both mainstream and false news sources. Given that people mostly encounter news from reputable sources, indiscriminately making people skeptical of all news content could have a net negative effect on the accuracy of people’s beliefs about the news. In this section, we evaluate this concern using additional information we have about respondents in the U.S. sample. First, we use pre-treatment web consumption data to estimate the probabilities of encountering news articles from different types of news sources (i.e., fake, hyperpartisan, and mainstream news outlets). Our goal is to estimate whether people will be able to correctly assess the accuracy of news headlines from different sources. We make the simplifying assumption that headlines from reputable sources such as The Wall Street Journal or Politico are credible and should generally be regarded as accurate while headlines from fake or hyperpartisan news sources are not. Let bi ∈ {0,1} represent the binary accuracy belief of a respondent i where 1 indicates that they believe the article is “somewhat accurate” or “very accurate” and 0 otherwise. Further, let S ∈ { f ,h,m,u} represent the source of the article where f represents outlets known to distribute “fake” content, h represent hyperpartisan news sources, m represent “high-prominence” mainstream news sources, and u represent “low-prominence” mainstream sources. (We provide additional details on how we operationalize these categories below.) Finally, let P(S) be the baseline probability that respondents encounter a story from each type of news source and Ti ∈ {0,1} represent treatment status where a 1 indicates being assigned to receive the media literacy intervention. We would like assess the joint distribution of accuracy beliefs and news types across treatment conditions. Are people more likely to believe stories from high-quality sources and disbelieve stores from low-quality sources depending on whether or not they receive the media literacy intervention? We also wish to estimate the effect of the intervention on the probability of people disbelieving stories from reputable sources (which we treat as false negatives per the assumption above) and believ- ing stories from fake and hyperpartisan sources (assumed false positives). A complete schematic is shown in Table G2a. We answer these questions by calculating the joint distribution of b and S. First, we calculate the marginal distribution of news content quality P(S) based on the web tracking data from respondents in the U.S. sample who are members of the YouGov Pulse panel.1 We calculate the proportion of news visits to the following sets of domains:

• We began with a list of “mainstream” news domains we have previously used in surveys to measure self-reported news consumption. We then filtered out domains with fewer than 5 million total monthly unique visits in comScore web traffic data collected in 2015. If not already in our list, we added domains affiliated with a major broadcast news network, national fact-checking organizations, and sources used in our survey instrument. We then divided this list into “high-prominence” and “low-prominence” subsets, with the former consisting of well-known legacy media outlets that are known for their professional commitment to original news-gathering processes (newspapers, broadcast news networks, and cable news networks). The “high-prominence” sites were: cnbc.com, nydailynews.com, cbsnews.com, abcnews.go.com, nbcnews.com, msn.com (affiliated with MSNBC), washingtonpost.com, foxnews.com, cnn.com, latimes.com, chicagotribune.com, nypost.com, nytimes.com, and

1This data was collected from Nov. 12–Dec. 26, 2018 but varied by respondent. We included visits up to the day before each respondent’s wave 1 survey responses in the post-election period.

G-1 wsj.com. The “low-prominence” sites were: businessinsider.com, teenvogue.com, vice.com, thedailybeast.com, politico.com, buzzfeed.com, realclearpolitics.com, fivethirtyeight.com, theatlantic.com, snopes.com, time.com, usnews.com, politifact.com, huffingtonpost.com, aol.com, and factcheck.org. • For our “hyperpartisan” list, we selected previously identified “hard news” domains that are relatively far from the ideological center as determined by the estimated “alignment scores” produced by Bakshy, Messing and Adamic (2015) using Facebook sharing data.2 We also ensured that this list, the list of untrustworthy websites that frequently share false or misleading information (which we call “false” below), and the mainstream outlet lists were mutually exclusive. The final hyperpartisan list contained 170 unique domains that included highly ideological websites like thenation.com, breitbart.com, dailykos.com, and theblaze.com, promi- nent hyperpartisan sites such as westernjournalism.com, and obscure websites such as patri- otupdate.com. • Finally, our “fake” list was determined by combining the “black,” “orange,” and “red” lists compiled by Grinberg et al. (2019) for a total of 490 domains. Using these domain lists, we estimate that p(S = f ) = 0.0307, p(S = h) = 0.1675, p(S = m) = 0.6357, and p(S = u) = 0.1661. In other words, when we consider all the URLs viewed by our respondents to any site in these lists, the probability that a given URL is 3% that it is from a false news site, 17% that it is from a hyperpartisan site, 64% that it is from a high-prominence mainstream news site, and 17% that it is from a low-prominence mainstream news site. Next, we leverage results from the U.S. study to calculate the conditional distribution of accuracy beliefs. So, for instance, the respondents in the control condition rated 65.1% of headlines from mainstream sources as “somewhat accurate” or “very accurate” and they rated 67.4% of such headlines as accurate in the treatment condition. These numbers correspond to p(b = 1|S = m,T = 0) and p(b = 1|S = m,T = 1), respectively (where p(b = 0|S = m,T = 1) = 1− p(b = 1|S = m,T = 1)). The full conditional distributions are shown in Table G1.

Table G1: Binary accuracy beliefs for headlines by news type and treatment condition

False Hyperpartisan High-prominence Low-prominence Control 0.318 0.309 0.651 0.479 Treatment 0.244 0.248 0.674 0.417

Cell entries represent the proportion of headlines rated as “somewhat accurate” or “very accurate” in wave 1 of the U.S. survey. Treatment corresponds to being assigned to read the digital media literacy news tips.

We can then calculate the joint distributions p(b,S|T = 0) and p(b,S|T = 1) using the axioms of probability from P(S)p(b|S,T ).3 Table G2a provides a guide for how to assess headline accuracy

2Specifically, we selected those outlets whose absolute value was greater than 0.7 (they can range from -1 to 1 on a scale measuring the extent to which Facebook shares for a media outlet come from self-identified liberals versus conservatives). 3We assume that the baseline probability of being exposed to news content from different source types is unrelated to treatment assignment.

G-2 rating performance across conditions. Our results, which we present in Tables G2b and G2c, are largely encouraging. For the control group, we expect that 62.9% of beliefs about the accuracy of a headline will be correct (stories from mainstream outlets deemed accurate or false/hyperpartisan stories judged not accurate), 6.1% will be false positives (belief in false/hyperpartisan stories), and 30.8% will be false negatives (lack of belief in mainstream stories). In the treatment condition, by contrast, we expect that 64.6% of beliefs about the accuracy of headlines will be correct, 4.9% of beliefs will be false positives, and 30.4% will be false negatives. These results indicate that, overall, the media literacy intervention helps people evaluate the accuracy of claims in headlines.

Table G2: Assessing the eﬀect of news tips intervention on aggregate accuracy judgments

(a) Schematic for assessing belief accuracy

Fake Hyperpartisan Mainstream Unfamiliar Belief = 0 Correct beliefs Correct beliefs False negative False negative Belief = 1 False positive False positive Correct beliefs Correct beliefs

(b) Control: Joint distribution of accuracy beliefs and news sources

Fake Hyperpartisan Mainstream Unfamiliar Belief = 0 0.021 0.116 0.222 0.087 Belief = 1 0.010 0.052 0.413 0.080

The false positive rate is calculated as p(b = 1,S = { f ,h}) = 0.010 + 0.052 ≈ 0.061. The false negative rate is p(b = 0,S = {m,u}) = 0.222 + 0.087 ≈ 0.308. Numbers may not sum to one or reﬂect simple addition due to rounding.

Fake Hyperpartisan Mainstream Unfamiliar Belief = 0 0.023 0.126 0.207 0.097 Belief = 1 0.007 0.042 0.428 0.069

The false positive rate is calculated as p(b = 1,S = { f ,h}) = 0.007 + 0.042 ≈ 0.049. The false negative rate is p(b = 0,S = {m,u}) = 0.207 + 0.097 ≈ 0.304. Numbers may not sum to one or reﬂect simple addition due to rounding.

G-3 H U.S. and India study news headlines

H.1 U.S. news headline stimuli Respondents evaluated 16 total articles: 4 mainstream news articles that were congenial to Democrats (2 from low-prominence sources and 2 from high-prominence sources), 4 mainstream news articles that were congenial to Republicans (2 from low-prominence sources and 2 from high-prominence sources), 2 pro-Democrat false news articles, 2 pro-Republican false news articles, 2 pro-Democrat hyperpartisan sources, and 2 pro-Republican hyperpartisan sources.

Pro-Democrat hyperpartisan news [Pro-D hyper 1] Donald Trump caught privately wishing he’d sided more thoroughly with white supremacists. https://www.palmerreport.com/analysis/white-supremacists- trump-siding/12478/ [Pro-D hyper 2] Franklin Graham: Attempted rape not a crime. Kavanaugh ‘respected’ his victim by not ﬁnishing. https://www.dailykos.com/stories/2018/9/19/1797143/-Graham- Attempted-rape-not-a-crime-Kavanaugh-respected-his-victim-by-not- finishing

Pro-Democrat false news [Pro-D false 1] VP Mike Pence Busted Stealing Campaign Funds To Pay His Mortgage Like A Thief. http://bipartisanreport.com/2018/09/03/vp-mike-pence-busted- stealing-campaign-funds-to-pay-his-mortgage-like-a-thief/ [Pro-D false 2] Vice President Pence now being investigated for campaign fraud his ties to Russia and Manafort. dctribune.org/2018/08/23/vice-president-pence-now-being- investigated-for-campaign-fraud-his-ties-to-russia-and-manafort/

Pro-Republican hyperpartisan news [Pro-R hyper 1] Soros Money Behind ‘Black Political Power’ Outfit Supporting Andrew Gillum in Florida. https://www.breitbart.com/big-government/2018/09/20/soros-money- behind-black-political-power-outfit-supporting-andrew-gillum-in-florida/ [Pro-R hyper 2] Kavanaugh Accuser Christine Blasey Exposed For Ties To Big Pharma Abortion Pill Maker... Effort To Derail Kavanaugh Is Plot To Protect Abortion Industry Profits. https:// www.infowars.com/kavanaugh-accuser-christine-blasey-exposed-for-ties- to-big-pharma-abortion-pill-maker-effort-to-derail-kavanaugh-is-plot- to-protect-abortion-industry-profits/

Pro-Republican false news [Pro-R false 1] Special Agent David Raynor was due to testify against Hillary Clinton when he died. http://www.neonnettle.com/features/1398-fbi-agent-who-exposed-hillary- clinton-s-cover-up-found-dead

H-1 [Pro-R false 2] Lisa Page Squeals: DNC Server Was Not Hacked By Russia. https://yournewswire. com/lisa-page-squeals-dnc-server-not-hacked-russia/

Mainstream news that is congenial to Democrats (low-prominence source) [Pro-D Mainstream 1] A Series Of Suspicious Money Transfers Followed The Trump Tower Meet- ing. https://www.buzzfeednews.com/article/anthonycormier/trump-tower- meeting-suspicious-transactions-agalarov [Pro-D Mainstream 2] A Border Patrol Agent Has Been Called a ‘Serial Killer’ by Police After Mur- dering 4 Women. https://www.teenvogue.com/story/border-patrol-agent- arrested-murder-4-women-serial-killer

Mainstream news that is congenial to Democrats (high-prominence source) [Pro-D Mainstream 3] Detention of Migrant Children Has Skyrocketed to Highest Levels Ever. https://www.nytimes.com/2018/09/12/us/migrant-children-detention.html [Pro-D Mainstream 4] ‘And now it’s the tallest’: Trump, in otherwise sombre 9/11 interview, couldn’t help touting one of his buildings. https://www.washingtonpost.com/gdpr- consent/?destination=%2fnews%2fmorning-mix%2fwp%2f2018%2f09%2f11%2fand- now-its-the-tallest-trump-in-otherwise-somber-9-11-interview-couldnt- help-touting-one-of-his-buildings%2f%3f

Mainstream news that is congenial to Republicans (low-prominence source) [Pro-R Mainstream 1] Google Workers Discussed Tweaking Search Function to Counter Travel Ban. http://uk.businessinsider.com/google-employees-search-protest-travel- ban-2018-9 [Pro-R Mainstream 2] Feds said alleged Russian spy Maria Butina used sex for inﬂuence. Now, they’re walking that back. https://news.vice.com/en_us/article/wjyqe4/feds- said-alleged-russian-spy-maria-butina-used-sex-for-influence-now- theyre-walking-that-back

Mainstream news that is congenial to Republicans (high-prominence source) [Pro-R Mainstream 3] Small business optimism surges to highest level ever, topping previous record under Reagan. https://www.cnbc.com/2018/09/11/small-business-optimism- surges-to-highest-ever.html [Pro-R Mainstream 4] Economy adds more jobs than expected in August, and wage growth hits post- recession high. https://www.cnbc.com/2018/09/07/us-nonfarm-payrolls-aug- 2018.html

H-2 Figure H1: U.S. news headline stimuli: Democrat-congenial headlines

False news Hyperpartisan news

Mainstream news (low-prominence) Mainstream news (high-prominence)

H-3 Figure H2: U.S. news headline stimuli: Republican-congenial headlines

False news Hyperpartisan news

Mainstream news (low-prominence) Mainstream news (high-prominence)

H-4 H.2 India news headline stimuli - Wave 1 Respondents evaluated all articles on a four-point accuracy scale: Not at all accurate (1) Not very accurate (2) Somewhat accurate (3) Very accurate (4) Not sure [IRB/MTurk]

Mainstream news that is congenial to BJP supporters [Pro-BJP 1] A recent article was published with the headline “PM Narendra Modi arrives in Varanasi, lays foundation stone of Kashi Vishwanath Temple Corridor.” To the best of your knowledge, how accurate is the claim that Modi laid the temple corridor’s foundation stone? https://www.indiatoday.in/india/story/pm-narendra-modi-kashi-vishwanath- temple-corridor-foundation-stone-1473055-2019-03-08

[Pro-BJP 2] A recent article was published with the headline “Rahul Gandhi greeted with ‘Modi Modi’ slogans as he interacts with students in Pune.” To the best of your knowledge, how accurate is the claim that Rahul Gandhi was greeted with ‘Modi Modi’ slogans? https://zeenews.india.com/video/india/rahul-gandhi-greeted-with-modi- modi-slogans-as-he-interacts-with-students-in-pune-2193049.html

Mainstream news that is congenial to BJP opponents [Anti-BJP 1] A recent article was published with the headline “India’s job crisis is worse than people thought — and its government tried to squelch the data” To the best of your knowledge, how accurate is the claim that the government tried to suppress data on jobs in India? https://www.washingtonpost.com/world/2019/02/01/indias-job-crisis- is-worse-than-people-thought-its-government-tried-squelch-data

[Anti-BJP 2] A recent article was published with the headline “Study ﬁnds that India would have seen 11% more communal riots had Congress lost closely-fought MLA seats between 1960 and 2000” To the best of your knowledge, how accurate is the claim that a study was conducted and found that there would have been more communal riots had Congress lost closely-fought elections? https://www.indiaspend.com/had-congress-lost-closely-fought-mla-seats- 1960-2000-india-wouldve-seen-11-more-communal-riots-study-10951/

Mainstream news that is congenial to Nationalists [Nat 1] A recent article was published with the headline “IAF ﬁghters chase out 2 Pakistani Air Force jets trying to violate Indian airspace again.” To the best of your knowledge, how accurate is the claim that IAF ﬁghters turned away Pakistan Air Force jets?

H-5 https://zeenews.india.com/india/iaf-chases-out-2-pakistani-jets-trying- to-violate-indian-airspace-again-2184190.html

[Nat 2] A recent article was published with the headline “After Pulwama, India to stop share of water ﬂowing to Pakistan.” To the best of your knowledge, how accurate is the claim that India will stop water from ﬂowing to Pakistan? https://www.indiatoday.in/fyi/story/indus-water-treaty-nitin-gadkari- pulwama-attack-1462432-2019-02-22

Pro-BJP false news [Pro-BJP 3] A recent article was published with the headline “PM Modi visits Kumbh, ﬁrst head of state to visit sacred ceremony.” To the best of your knowledge, how accurate is the claim that Modi was the ﬁrst head of state to visit Kumbh? (Note: This a false headline adapted from a fact-check published by AltNews on February 25, 2019.) https://www.altnews.in/pm-modi-is-not-the-first-indian-prime- minister-to-visit-kumbh-as-claimed-by-bjp-it-cell-head/

[Pro-BJP 4] A recent article was published with the headline “Congress workers chant ‘Pakistan Zindabad’ slogan in Rajasthan’s Rajsamand” To the best of your knowledge, how accurate is the claim that Congress workers chanted “Pakistan Zindabad”? https://www.altnews.in/multiple-videos-falsely-claim-congress-workers- thrashed-by-police-for-chanting-pakistan-zindabad/

Anti-BJP false news [Anti-BJP 3] A recent article was published with the headline “Supreme Court has given a big ver- dict, Ordered PM Modi to be charged in the Rafale scam case, BJP is shaken.” To the best of your knowledge, how accurate is the claim that the Supreme Court decided to ﬁle a case against Modi? https://www.altnews.in/fake-news-supreme-court-orders-case-to-be- registered-against-pm-modi-over-rafale-deal/

[Anti-BJP 4] A recent article was published with the headline “Britain’s former Prime Minister Tony Blair says India’s future is safe in Rahul Gandhi’s hands.” To the best of your knowledge, how accurate is the claim that Blair said India’s future is safe in Gandhi’s hands? https://www.boomlive.in/did-former-british-pm-say-indias-future-is- safe-in-rahul-gandhis-hands-a-factcheck/

Nationalist false news [Nat 3] A recent article was published with the headline “Saudi prince says he believes Kashmir is ‘Hindu land’.” To the best of your knowledge, how accurate is the claim that a Saudi prince believes

H-6 Kashmir is Hindu land? https://www.altnews.in/viral-video-false-claim-of-saudi-prince-giving- opinions-on-kashmir/

[Nat 4] A recent article was published with the headline “Modi government starts scheme to collect funds for battle casualties and purchasing weapons, mission to make India a super power.” To the best of your knowledge, how accurate is the claim that the government has started a scheme to collect funds to buy weapons and make India a super power? https://www.altnews.in/old-message-of-army-battle-casualties-fund- viral-again-not-meant-for-weapons-purchase/

H.3 India news headline stimuli - Wave 2 Respondents evaluated all articles on a four-point accuracy scale: Not at all accurate (1) Not very accurate (2) Somewhat accurate (3) Very accurate (4) Not sure [IRB/MTurk]

Mainstream news that is congenial to BJP supporters A recent article was published with the headline “India’s mom-and-pop merchants lean toward Modi’s BJP in election.” To the best of your knowledge, how accurate is the claim that most mom- and-pop merchants support the BJP? https://asia.nikkei.com/Politics/India-election/India-s-mom-and-pop- merchants-lean-toward-Modi-s-BJP-in-election

A recent article was published with the headline “Congress spokeswoman Priyanka Chaturvedi quits in middle of Indian election, joins BJP ally.” To the best of your knowledge, how accurate is the claim that Priyanka Chaturvedi quit Congress and joined a BJP ally? https://www.reuters.com/article/india-election-congress-idUSKCN1RV0N3

Mainstream news that is congenial to BJP opponents A recent article was published with the headline “Congress party launches new theme song written by superstar Bollywood composer Javed Akhtar as vote approaches.” To the best of your knowledge, how accurate is the claim that Congress has launched a new theme song by Javed Akhtar? https://www.straitstimes.com/asia/south-asia/indias-opposition-congress- party-enlists-bollywood-star-power-as-vote-approaches

H-7 A recent article was published with the headline “Another BJP ally leaves alliance with BJP; Om Prakash Rajbhar quits Yogi government in UP.” To the best of your knowledge, how accurate is the claim that another BJP ally left the BJP government’s alliance in UP? https://www.nationalheraldindia.com/national/bjp-ally-sbsp-quits- yogi-adityanath-government-in-up

Mainstream news that is congenial to nationalists A recent article was published with the headline “India shelling kills two on Pakistan side of the India-Pakistan border in Kashmir.” To the best of your knowledge, how accurate is the claim that India shelling killed two people on the Pakistan side of the border in Kashmir? https://www.aljazeera.com/news/2019/05/india-shelling-kills-people- pakistan-administered-kashmir-190506083017472.html

A recent article was published with the headline “Report ﬁnds that cow vigilantes killed 44 people in the last three years.” To the best of your knowledge, how accurate is the claim that 44 people have been killed by cow vigilantes in the past three years? https://www.bloomberg.com/news/articles/2019-02-20/cow-vigilantes- in-india-killed-at-least-44-people-report-finds

Pro-BJP false news A recent article was published with the headline “Prime Minister Nardendra Modi is giving talk time of 400 rupees in the celebration of the smart city scheme.” To the best of your knowledge, how accurate is the claim that PM Modi is giving 400 rupees of talk time? https://www.factcrescendo.com/fact-check-isnt-this-spam/

A recent article was published with the headline “Congress paid the same elderly woman Rs 5000 for photoshoots with diﬀerent leaders.” To the best of your knowledge, how accurate is the claim that an elderly woman was paid to be in photos with Congress leaders? https://www.altnews.in/false-claim-congress-leaders-photographed- with-the-same-elderly-woman-in-multiple-pictures/

Anti-BJP false news A recent article was published with the headline “BJP candidate’s trousers fall down while giving speech.” To the best of your knowledge, how accurate is the claim that a BJP candidate’s trousers fell down while giving a speech? https://www.altnews.in/did-a-bjp-leader-have-a-clothing-malfunction- while-campaigning-old-video-false-claim/

H-8 A recent article was published with the headline “10 lakh of people welcome Rahul Gandhi at public meeting in Jalore.” To the best of your knowledge, how accurate is the claim that Rahul Gandhi was met by 10 lakh of people in Jalore? https://www.factcrescendo.com/fact-check-is-this-image-of-rahul-gandhis- jalore-rally

Nationalist false news A recent article was published with the headline “Sadhvi Pragya Thakur, Hindu activist who helped plan 2008 anti-Muslim bombing, acquitted of terrorism charges” To the best of your knowledge, how accurate is the claim that Sadhvi Pragya Thakur was acquitted for her role in an anti-Muslim bombing? https://www.altnews.in/false-claim-sadhvi-pragya-has-been-acquitted- of-terror-charges/

A recent article was published with the headline “Martyred Jawans’ bodies kept in garbage boxes after Naxal attack in Gadchiroli.” To the best of your knowledge, how accurate is the claim that the bodies of Jawans killed in a Naxal attack were kept in garbage boxes? https://www.factcrescendo.com/fact-check-were-martyr-jawans-bodies- kept-in-garbage-boxes/

False news with distributed fact-checks [FTF] A recent article was published with the headline “BJP President of Madhya Pradesh, Nandkumar Singh Chauhan claims that some women in Uttar Pradesh give birth to a child every week, 52 children per year.” To the best of your knowledge, how accurate is the claim that the BJP President of Madhya Pradesh believes that some women in Uttar Pradesh give birth every week? https://www.factcrescendo.com/fact-check-did-mp-bjp-chief-say-up- women-give-birth-to-kid-every-week/

A recent article was published with the headline “Tax authorities raid house of Uttar Pradesh minister Swami Prasad Maurya, ﬁnd thousands of crores of black money” To the best of your knowledge, how accurate is the claim that the minister Swami Prasad Maurya was raided by the tax authorities ? https://www.factcrescendo.com/fake-image-claims-20000-crores-found/

A recent article was published with the headline “Yogi Adityanath ﬁlmed giving cash to people for voting for the BJP.” To the best of your knowledge, how accurate is the claim that Yogi Adityanath paid people to vote for the BJP? https://www.boomlive.in/was-yogi-adityanath-enticing-potential-voters- with-cash-a-factcheck/

H-9 A recent article was published with the headline “Kanhaiya Kumar holds campaign rally with photo of the terrorist Afzal Guru.” To the best of your knowledge, how accurate is the claim that Kanhaiya Kumar used a photo of Afzal Guru in his campaign? https://www.altnews.in/kanhaiya-kumars-campaign-vehicle-photoshopped- with-image-of-afzal-guru/

[IRB/MTurk] A recent article was published with the headline “Delhi police stop ambulance for Rahul Gandhi’s rally, leads to girl’s death.” To the best of your knowledge, how accurate is the claim that the Delhi police stopped an ambulance for Rahul Gandhi’s rally and led to a girl’s death? https://www.altnews.in/hindi/ambulance-stuck-due-to-rahul-gandhis- rally-no-old-video-of-road-block-for-malaysian-delegate/

“In surprise, BJP President Amit Shah calls BJP’s own candidate Meenakshi Lekhi a liar, urges crowd not to vote for her.” To the best of your knowledge, how accurate is the claim that Amit Shah called a BJP candidate a liar and encouraged people not to vote for her? https://www.altnews.in/hindi/did-amit-shah-exhort-crowd-to-not-vote- for-meenakshi-lekhi-ajay-maken-tweets-clipped-misleading-video/

A recent article was published with the headline “Congress workers burn effigy of Modi in Kar- nakata, set own lunghis on fire.” To the best of your knowledge, how accurate is the claim that Congress workers burned an effigy of Modi and set their own lunghis on fire? https://www.factcrescendo.com/fake-video-claims-to-show-indias-opposition- congress-party-workers-burning-an-effigy-of-modi/

A recent article was published with the headline “Indian Muslims burn Indian ﬂag in protest against Prime Minister Modi.” To the best of your knowledge, how accurate is the claim that Indian Mus- lims burned the Indian ﬂag when protesting against Modi? https://www.factcrescendo.com/fact-check-are-these-people-burning- our-national-flag/

H-10 References

Allcott, Hunt, Matthew Gentzkow and Chuan Yu. 2019. “Trends in the diﬀusion of misinformation on social media.” Research & Politics 6(2):2053168019848554.

Amazeen, Michelle A and Erik P Bucy. 2019. “Conferring Resistance to Digital Disinformation: The Inoculating Inﬂuence of Procedural News Knowledge.” Journal of Broadcasting & Elec- tronic Media 63(3):415–432.

Bakshy, Eytan, Solomon Messing and Lada A. Adamic. 2015. “Exposure to ideologically diverse news and opinion on Facebook.” Science 348(6239):1130–1132.

Clayton, Katherine, Spencer Blair, Jonathan A. Busam, Samuel Forstner, John Glance, Guy Green, Anna Kawata, Akhila Kovvuri, Jonathan Martin, Evan Morgan, Morgan Sandhu, Rachel Sang, Rachel Scholz-Bright, Austin T. Welch, Andrew G. Wolﬀ, Amanda Zhou and Brendan Nyhan. 2019. “Real Solutions for Fake News? Measuring the Eﬀectiveness of General Warnings and Fact-Check Tags in Reducing Belief in False Stories on Social Media.” Political Behavior .

Craft, Stephanie, Seth Ashley and Adam Maksl. 2017. “News media literacy and conspiracy theory endorsement.” Communication and the Public 2(4):388–401.

Franco, Annie, Neil Malhotra, Gabor Simonovits and L.J. Zigerell. 2017. “Developing standards for post-hoc weighting in population-based survey experiments.” Journal of Experimental Political Science 4(2):161–172.

Grinberg, Nir, Kenneth Joseph, Lisa Friedland, Briony Swire-Thompson and David Lazer. 2019. “Fake news on Twitter during the 2016 US presidential election.” Science 363(6425):374–378.

Guess, Andrew, Brendan Nyhan and Jason Reiﬂer. 2020. “Exposure to untrustworthy websites in the 2016 U.S. election.” Nature Human Behaviour .

Jeong, Se-Hoon, Hyunyi Cho and Yoori Hwang. 2012. “Media literacy interventions: A meta- analytic review.” Journal of Communication 62(3):454–472.

Kahne, Joseph and Benjamin Bowyer. 2017. “Educating for democracy in a partisan age: Con- fronting the challenges of motivated reasoning and misinformation.” American Educational Re- search Journal 54(1):3–34.

Lutzke, Lauren, Caitlin Drummond, Paul Slovic and Joseph Árvai. 2019. “Priming critical thinking: Simple interventions limit the inﬂuence of fake news about climate change on Facebook.” Global Environmental Change 58:101964.

Marbach, Moritz and Dominik Hangartner. 2020. “Proﬁling Compliers and Non-compliers for Instrumental Variable Analysis.” Political Analysis .

Miratrix, Luke W., Jasjeet S. Sekhon, Alexander G. Theodoridis and Luis F. Campos. 2018. “Worth weighting? How to think about and use weights in survey experiments.” Political Analysis 26(3):275–291.

H-11 Montgomery, Jacob M., Brendan Nyhan and Michelle Torres. 2018. “How conditioning on post- treatment variables can ruin your experiment and what to do about it.” American Journal of Political Science 62(3):760–775.

Pennycook, Gordon, Tyrone D. Cannon and David G. Rand. 2018. “Prior exposure increases perceived accuracy of fake news.” Journal of Experimental Psychology: General 147(12):1865– 1880.

Roozenbeek, Jon and Sander van der Linden. 2019. “Fake news game confers psychological resistance against online misinformation.” Palgrave Communications 5(1):12.

Tully, Melissa, Emily K Vraga and Leticia Bode. 2019. “Designing and Testing News Literacy Messages for Social Media.” Mass Communication and Society pp. 1–25.

Vraga, Emily K and Melissa Tully. 2019. “News literacy, social media behaviors, and skepticism toward information on social media.” Information, Communication & Society pp. 1–17.

H-12