Downloaded by guest on September 25, 2021 a media Guess M. partisan Andrew online of consequences The eae frsac ntemnmlefcstaiin which tradition, effects minimal PNAS the echoes in affairs of research state internet’s This the of society. about in decades accounts divisions popular fostering in and role issues these of sions social polarized fragmented, (15). a elusive in remains media age of media con- reasons, impact these the For on settings. sensus controlled news in individual clips as TV or such articles treatments curated of relatively (13, impact the on short-term exposure focused have media designs of research Experimental measures 14). traditional of and news (10–12), unreliability of designs the self-selection forced-stimuli of to artificiality due the bias (9), sources address Previous to effects. sought media have of study studies the in challenges well-known ing frequently news. of those (6–8). sources among partisan especially to sources impact, exposed causal congenial ques- consumers its raises about news consumption politically tions information online people’s for on which evidence preference This to extent Schol- a the behaviors. exhibit political debated their have people and environment ars ways information the information political understanding the seek for to implications profound changes (3– have flourish structural reality—to These of of boundaries 5). accounts rumors, the biased also and set perspectives—but of misinformation, to range wider gatekeepers eroded a allowing have traditional debate, dynamics of These power 2). (1, the movements perspec- social underrepresented and for field tives playing access, force the democratizing level to a to is helps barriers internet that the removing whether In debated have information. some of availability the T media to exposure news. of partisan consequences long-term the experimentation studying powerful for a approach of illustrates techniques combination science social The computational and limited, dynamics. sub- are of cumulative possibility media the tle, about tradi- partisan questions minimal-effects raise online findings the our of although with consequences Consistent later. direct y tion, mainstream 1 the to in trust up in Still, media decrease meaningful affect. pro- and treatments or lasting both opinions a that duce suggest on waves impact survey we direct later however, from a results comparisons, of recent evidence multiple of little for knowledge find adjusting and After but visits immediate 8 website events. cause least in can at increases news for partisan short-lived to persisted exposure consumption Greater news wk. in changes campaign. right-leaning resulting election midterm or US on (HuffPost) 2018 Data the left-leaning during a slant News) either encounter- (Fox of with likelihood social news the and ing boosting settings patterns, default following browser media their randomized change to a polar- (N incentivized from survey the represen- panel results nationally online in a report tative in play embedded we experiment media Here field longitudinal extreme society? for (received of ideologically 2021 17, ization do February Levi Margaret role Member Board What Editorial by accepted and NC, 2020) Durham, 29, University, June Duke review Bail, Andrew Christopher by Edited and Germany; Berlin, 10117 c eateto oiia cec n nentoa eain,Uiest fSuhr aiona o nee,C 90089; CA Angeles, Los California, Southern of University Relations, International and Science Political of Department eateto oiis rneo nvriy rneo,N 08544; NJ Princeton, University, Princeton Politics, of Department sarsl,teei ag a ewe cdmcdiscus- academic between gap large a is there result, a As overcom- requires news slanted of consequences the Assessing asi hc hsi curn sb rmtclyincreasing dramatically by is occurring fundamental is the this of which One in society. ways transforming is internet he 01Vl 1 o 4e2013464118 14 No. 118 Vol. 2021 | politics ≈ 9mlinwbvst yrsodnsidct that indicate respondents by visits web million 19 | polarization a,b,1,2 e eateto omncto,Uiest fIlni tUbn-hmag,Ubn,I 61801 IL Urbana, Urbana-Champaign, at Illinois of University Communication, of Department | al Barber Pablo , opttoa oilscience social computational ,3)i hc atcpnswere participants which in 1,037) = a ´ c,1 io Munzert Simon , b colo ulcadItrainlAfis rneo nvriy rneo,N 08544; NJ Princeton, University, Princeton Affairs, International and Public of School d,1 doi:10.1073/pnas.2013464118/-/DCSupplemental at online information supporting contains article This 2 1 BY-NC-ND) (CC 4.0 NoDerivatives License distributed under is article access open This ulse ac 9 2021. 29, March Published Editorial the by invited editor guest a is C.A.B. Submission. Direct PNAS Board. a is article This interest.y competing no declare authors The paper.y the research, performed wrote research, and designed data, J.Y. analyzed and S.M., P.B., A.M.G., contributions: Author focuses 2 Study assessing polarization. outcomes political of affects range media a partisan covers how 1 Study analysis. for studies impact long-term the about media. hypotheses partisan of online oppor- range participants of the a track provided test months to to for ability tunity data behavioral sustained our and and powerful, survey intervention with The the inadvertently. brows- of if web nature day-to-day even their experience, news during a slant encountered ing as partisan subjects served a apps that thus with or likelihood framed intervention websites the The news boosting (24). from nudge, sites news media get social they than for that say media Americans more to social (23), continue expression of is public importance and treatment (22) of the agenda-setting choice people’s Despite Our valid: increased websites. ecologically news naturalistically partisan but to exposure subtly that experiment downstream have disproportionately that could habits In consequences. nudges) consumption experience. (or information online changes affect people’s small and view, structure features, this design of that subtle lens settings frictions, the default (21): through architecture environments choice information peo- that partisan synthesis people’s change strong internet-age an views without can propose those We news (17–20). especially implies attachments cable strongly behavior, research especially voting recent ple’s and more predispositions newspapers time, existing same that the reinforce At primarily (16). media that argues owo orsodnemyb drse.Eal [email protected] Email: addressed. be may correspondence work.y whom this To to equally contributed J.Y. and S.M., P.B., A.M.G., euneo niepria ei a ea rso ftrust of erosion con- an real news. be mainstream the may in that media suggest partisan online span of time media sequence longer competing a same from the by data At it persuaded time, campaign. election that be contentious a to confirm during people accounts results for The difficult techniques. computa- is through science and effects social experimentation these real-world tional demonstrating of exploring by combination online for ground a approach of breaks nudge-like nature study method- a multifaceted This to effects. due the information. elusive media of and is sources difficulties proposition slanted ological this providing for evidence by Yet as pol- to such related role behaviors itics, major and a attitudes plays people’s influencing internet in the that suggests wisdom Popular Significance u rrgsee yohsswr iie notoseparate two into divided were hypotheses preregistered Our field online longitudinal large, a designed we end, that To n ugwnYn ( Yang JungHwan and , y https://doi.org/10.1073/pnas.2013464118 . y raieCmosAttribution-NonCommercial- Commons Creative d aaSineLb eteSchool, Hertie Lab, Science Data ) . y e,1 https://www.pnas.org/lookup/suppl/ | f8 of 1

POLITICAL SCIENCES mainly on information and credibility—how our partisan media selective exposure are related to a belief that the media is treatments affect media trust, political knowledge, and the accu- biased (32). Perceptions of bias have a rich history in studies racy of election predictions. The first set of hypotheses in study 1 of exposure to media content (33–35). The well-known hostile relates to the effects of partisan media exposure on people’s atti- media phenomenon was demonstrated in scenarios modeled on tudes and opinions. Research on persuasion and opinion change the broadcast era, with ostensibly neutral or balanced stimuli. suggests that individuals take cues from the media—especially Related dynamics have also been shown in response to partisan those that highlight copartisan elite figures—when evaluating news shows (36). In both sets of studies, however, outcomes focus candidates and policy options (25, 26). We follow recent research on subjects’ attitudes toward the specific clips, shows, or hosts in looking for shifts in policy opinions across a range of issues shown as treatments rather than the media writ large. Build- (27). In addition, we test for effects on a range of outcomes based ing on evidence of partisan criticism of the media overall and on findings in prior research, including agenda-setting (9), which the effects of partisan media on trust toward specific sources has been powerfully demonstrated in one of the few longitudi- or figures, it is reasonable to ask whether partisan media can nal field experiments on social media (23); affective polarization reduce the general public’s overall level of trust in the main- (28); and perceived polarization (29). For these and other out- stream press. This has not been the focus of prior experimental comes, a minimal-effects interpretation of the evidence to date research in the United States. Finally, we assess the basic ability would lead to an expectation of negligible impact. At the same of media to enhance people’s recall of current news events (37), time, recent formulations of this tradition argue that salient a key question for understanding whether people are receptive issues may be an exception (ref. 15, p. 725). For this reason, we to information from partisan media outlets, even ones they may additionally preregistered a hypothesis about attitudes toward not trust. immigration, which was the topic of contentious debate during the 2018 US midterm election campaign, during which this study Design. To test these hypotheses, we worked with an online was fielded. polling firm to field a nationally representative panel survey of Second, we test hypotheses related to online and offline polit- Americans, which interviewed the same respondents eight times ical behavior. A potential immediate consequence of increased from July 2018 to October 2019 (Fig. 1). These respondents were exposure to partisan political coverage is changes to one’s infor- recruited from YouGov’s Pulse panel, which unobtrusively col- mation diet, including possible lasting effects on media con- lects anonymous web visit and mobile app use data from those sumption habits. This expectation is consistent with viewing our who consent to install passive metering software. To maintain intervention as a subtle nudge altering informational default realism within the high-choice online environment of our sub- options (21). Expanding beyond online media, it is possible that jects, we opted for an encouragement design (38). In the third exposure to information could motivate people to behave dif- wave of the survey, we randomly incentivized subsets of our ferently on social media, either directly—by sharing links to sample to change the default homepage on their primary web viewed articles and posts—or indirectly, by altering the amount browsers for one month. One-third of the sample was asked of posting activity (3). to set their homepage to Fox News (foxnews.com) and one- Third, we study the effect of partisan media on trust in the third to HuffPost (huffingtonpost.com), while the last third was media generally (study 2). The long-term decline in trust and assigned to a pure control group. The treatment encourage- confidence in the mainstream press has been linked to numer- ment was aimed at browsers on desktop computers or laptops. ous potential causes, most plausibly an increase in attacks from We additionally asked subjects in each treatment group to fol- political elites and elected officials (30). Since this behavior is low the source’s associated Facebook page and subscribe to not symmetric across political parties in the United States, pat- affiliated newsletters in order to maximize the strength of the terns of media trust have become polarized (31). The rise of intervention (39). We collected pretreatment survey measures in blogs and social media has further enabled critiques of main- the first three waves of the panel and posttreatment outcomes stream coverage; consistent with this, tendencies toward partisan beginning in wave 4. The size of our final experimental sample

Fig. 1. Overview of study design. Subjects in wave 3 who were randomly assigned to the Fox News or HuffPost encouragement groups were offered $8 in YouGov incentives to participate in the treatment.

2 of 8 | PNAS Guess et al. https://doi.org/10.1073/pnas.2013464118 The consequences of online partisan media Downloaded by guest on September 25, 2021 Downloaded by guest on September 25, 2021 oprsn atcpnsi h rtetetpro pn less (M spent site period News pretreatment Fox the the in s on participants 50 day comparison, nearly per of (M increase min site mean 2 treatments HuffPost a the corresponding significant: on day the was per week of first intensity the the each in from that pages on spent reveals time average on site Focusing on site. other day the per to visits site between that visited on treatment pages (M News additional Fox four and the three to assigned those and ramn iie prxmtl n diinlpg ntesite our the (M on HuffPost postencouragement, page the day d additional to per one 7 assigned approximately subjects first visited that treatment the reveal data In tracking 2). web (Fig. media partisan intended online to as exposure real-world increased they that Results. news on research out- with of range associated impact. a typically covering those and beyond higher-frequency content comes news studying partisan by to (18) exposure about behavior questions political answer and to experiments media field for research potential demonstrat- research for the previous ing usernames of off their builds from design share lists This follow purposes. to and agreed posts who visits, Twitter web include respondents to data addition trace compliance In digital treatment research. our measure previous of than to precisely measures as more posttreatment well as and consumption pretreatment news both allows period, create 2-mo a to over us visits million 19 than more comprising outcomes (d survey small most is for in effect (ITT) detail well- intent-to-treat we detectable reasonably as for allow estimates: variables powered dependent our of measures (N panelists. Pulse YouGov from are data Behavioral intervals. confidence lines) (thin 95% and 2. Fig. h osqecso niepria media partisan online of consequences The al. et Guess ,3) ihsto rtetetcvrae,adrepeated and covariates, pretreatment of set rich a 1,037), = 3 = ekyaverage Weekly . 60; ecaatrz h rtsaeo u ramnst verify to treatments our of stage first the characterize We < p 1 = .Tewbdt olce rmorpanelists, our from collected data web The 0.10). < .03; .Nihrtetetdtcal increased detectably treatment Neither 0.001). p nme fvst )t (Left to 1) + visits of log(number < oprdt h oto group, control the to compared 0.001) 48; = IAppendix SI p < 126; = n oethan more and 0.001) h minimum the , onw.o n (Right and foxnews.com ) p < .By 0.001). oatido hs h a o iie hma l ro to prior all at them visited not quarter had a who among those websites of partisan third the a to to visits new in treatment induced HuffPost. show baseline, News to visits to also their Fox compared increased We group the News HuffPost the Fox in of Fox to 50% those and visit visits of their to increased 57% induced group was HuffPost: sample or our News of proportion substantial more already in compliers who of those characteristics in the detail among explore news We it. partisan consume to merely not exposure did the treatments of increase our compliers that between between indicates or either This sample treatments. HuffPost the two of or rest News the and Fox compliers to do ideology visits the to in pretreatment of differences encouraged rest strong or no the if are than only there interest Reassuringly, political and sample. con- on if higher who score HuffPost data to or so—tend visit News data. web Fox trace share sumed to digital agreeing have those study—participants we among 29% whom measure—approximately for according similar preregistered esti- is a treatment the each to for reassuringly, intervention compliers although of the share powerful, to mated 3.66 more due and inherently day was HuffPost per esti- to News an Fox visits (p with to additional week, visits 0.34 eighth of additional of the impact in average The websites. measurable mated news-related still on is week treatment per min 34 than * o esgopad7%i h ufotgop,atog ecudntvrf the verify not could we although sign-ups. group), actual HuffPost to the led the in clicks in (76% which high 78% to was extent and measure group this by News Compliance Fox group. treatment either to assigned eas eoddcik nteealnwlte usrpinlnssont respondents to shown links subscription newsletter email the on clicks recorded also We okn eodaeae,w n esuigeiec hta that evidence reassuring find we averages, beyond Looking < ufigops.o ytetetasgmn,wt 8 tiklines) (thick 68% with assignment, treatment by huffingtonpost.com ) .Teerslssgetta h o estreatment News Fox the that suggest results These 0.01). eto 2.2. section Appendix, SI httetreatments the that 2.2 , section Appendix, SI https://doi.org/10.1073/pnas.2013464118 ∗ h opir four of compliers The PNAS | f8 of 3

POLITICAL SCIENCES Fig. 3. Summary of effects (with 95% confidence intervals) on news visits/shares (study 1) and event knowledge (study 2). For visits, coefficients represent effect of treatment assignment on log(number of news visits + 1) over given time period postintervention. Each visit type (hard news, conservative news, and liberal news) excludes visits to both foxnews.com and huffingtonpost.com. Shares data show log(number of tweets + 1) containing liberal (HuffPost treatment) or conservative (Fox News treatment) links. Models are fully saturated OLS regressions with indicator variables for browser using preregistered LASSO covariate selection procedure. For event knowledge, standardized coefficients represent CACE estimates from two-stage least squares models in which treatment receipt, measured using web visit data, is instrumented with treatment assignment.

the experiment and that about a third of each treatment group sharing links on Twitter), although these effects are less pre- increased consumption of their assigned websites by a factor of cisely estimated.† Results remain similar even after we tested at least 10. revised hypotheses by directly comparing the two treatment The exogenous increase in partisan news exposure induced by groups with each other (SI Appendix, section 6). These find- the randomized encouragement led to additional changes in sub- ings are consistent with recent research evaluating the persuasive jects’ news diets in the weeks after the intervention. Although impact of advertising and campaign contact, which suggest neg- we could not detect an increase in visits to liberal news sites ligible effects especially in settings that map onto the political other than HuffPost as a result of that treatment, Fox News divide (43, 44). They also align with a recent field experiment boosted visits to other conservative news sites in the first week on the effects of Facebook news subscriptions (39), which simi- (M = 0.32; p < 0.01), an effect that barely decayed after the larly finds no changes in opinions (however, the study does find sixth week (Fig. 3). A possible explanation for the divergent par- that counterattitudinal news reduces negative attitudes toward tisan effects on news consumption is that HuffPost effectively the out-party). acted as a substitute for other news sources, while Fox, given At the same time, we uncover findings that suggest an under- its centrality in conservative media, drove traffic to other sites appreciated pathway of influence. First, we produce evidence in the ecosystem (40). Regardless of the difference, both treat- that Fox News online caused among our subjects a decrease in ments immediately increased subjects’ ability to recognize recent overall trust and confidence in the mainstream media within the events in the news and to distinguish them from made-up events first several weeks of boosted exposure (complier average causal (Fig. 3, Right). This finding is notable given popular accounts effect [CACE], −0.56 SD; p < 0.05), although this does not sur- that Fox News, in particular, causes people to become less vive any of our multiple comparisons adjustments (corrected, informed (41, 42). p = 0.32). However, looking beyond our original panel data to Turning to less proximate outcomes, we uncover results that waves that we subsequently collected, we find a lasting effect not in many ways provide support for a minimalist view of media only for Fox News (CACE, −0.54 SD; p < 0.05) but for Huff- impact. Especially after correcting for multiple comparisons Post as well (CACE, −0.70 SD; p < 0.01) approximately 5 mo within studies (SI Appendix, section 7), we do not find measurable after the intervention (in wave 7), with little decay. Remark- impact on many attitudes and behaviors (Fig. 4). We are able to ably, these changes appear fairly durable even after a year (in rule out modest effects (Cohen’s d < 0.10) of either treatment wave 8) with similar magnitudes, although the effect of Fox News on agenda-setting, elite approval, and an index of issue positions. (CACE, −0.48 SD; p < 0.10) falls short of significance at the We also cannot reject the null hypothesis that exposure to either treatment shifted people’s feelings toward the parties or that they affected perceptions of polarization, self-reported vote turnout †In SI Appendix, we conduct an analysis of the minimum detectable effect for each intentions, or social media activity (following news sources and treatment–outcome combination expressed in standardized (Cohen’s d) terms.

4 of 8 | PNAS Guess et al. https://doi.org/10.1073/pnas.2013464118 The consequences of online partisan media Downloaded by guest on September 25, 2021 Downloaded by guest on September 25, 2021 h osqecso niepria media partisan online of consequences The al. allowed et Guess waves was survey intervention multiple our although costs: bundle, without a come necessarily This not achieves (45). design does validity our internal approach data, sacrificing online without trace an validity digital in individ- ecological linked on encouragement with internet naturalistic survey the a panel and embedding media By social for uals. of framework methodological effect a the demonstrate studying we article, this In research. Discussion prior long-term of subtle, attention the a eluded of has mea- indicative that survey effect waves—as other media later our these on on effects significant sures of with lack results—along analyses additional continuing preregister these a specifically interpret we not 5, did wave Appendix, beyond (SI we sample Although the in S15). trust Fig. media conser- of and levels liberals’ average between vatives’ difference the of 40% to equivalent SD; 5 ee tosdd ncnrs oHfPs (CACE, HuffPost to contrast in (two-sided) level 95% assignment treatment with instrumented is data, visit web using measured LASSO. receipt, via treatment selected which are in covariates models squares and least two-stage from estimates CACE 4. Fig. p < umr fefcs(ih9%cndneitras niseiprac,attds n efrpre eair tnadzdcefiinsrepresent coefficients Standardized behavior. self-reported and attitudes, importance, issue on intervals) confidence 95% (with effects of Summary .Sbtniey h ieo hs fet slarge, is effects these of size the Substantively, 0.05). −0. 54 clsit fteps 5y hseiec scnitn iha with consistent of out is arising evidence effects (16). this War minimal World y, media’s Second 75 on the research past of the tradition polit- of and shifts societal, direct technological, ical of seismic evidence by the strong and Despite studied find influence. agenda-setting, frequently not behavior, do outcomes polarization—we voting of affective as kinds How- the scientists—such events. of social political basic online most recent a of their for on awareness work ever, alter increase to can seems to it way level: people this in Encouraging environments information par- source. right-leaning or news left-leaning increased tisan a intervention: either strong to exposure relatively real-world a of effects cumulative effects subgroup consumption). media for pretreatment test and ideology, to influence. powered of well (see not pathways generally effects, different main are of estimates for we precise fairly test enables sample and our While prespecify to us h ogtdnlntr fordt nbe st td the study to us enables data our of nature longitudinal The IAppendix SI o nlsso eeoeeu fet yparty, by effects heterogeneous of analyses for https://doi.org/10.1073/pnas.2013464118 PNAS | f8 of 5

POLITICAL SCIENCES In assessing the generalizability of our findings, we consider suggesting that a behavioral science perspective can usefully be how the experimental participants who complied with the treat- applied to people’s interactions with internet media (52). The ment differ from the population. In a best-case scenario for intervention can be interpreted as boosting incidental exposure strong effects, we could have had compliers who are infrequent to partisan content in a way that might be comparable to seeing news consumers, highly inattentive, and with weak political pref- posts from weak ties on social media (e.g., retweets or reshares) erences, or in the other extreme, compliers could have been par- that are more likely to be political and counterattitudinal in tisan, frequent consumers of political news with more crystallized nature. Taken together, our findings point to the conclusion that attitudes. As shown in SI Appendix, Fig. S12, the compliers in this despite today’s extremely high-choice media environment (53), study are comparable to the overall sample in many important frictions and “architectures of serendipity” (50) can still, at least respects, including pretreatment visits to Fox News/HuffPost, in principle, allow for engagement with different perspectives in ideology, and age, although they are notably more interested in the news. politics (more than 80% of compliers have very high political ‡ interest compared to an already-high 70% in the full sample). Materials and Methods Thus, while our treatments did not simply increase the parti- Sampling and Participants. In partnership with the online survey firm san news consumption of people who already visit such websites, YouGov, we initially recruited a total of 1,551 respondents from the Pulse they did successfully encourage people who tended to have a panel, a subset of YouGov’s traditional survey panels in which members opt higher interest in politics to begin with. For this reason, it may in to install passive metering software on their desktop and mobile devices. be reasonable to think of our findings on attitudes and behav- Respondents agreed to join a “Politics and Media” study with multiple ior as closer to the lower bound of possible effects as compared survey waves. Their participation was rewarded using YouGov’s proprietary to research focusing on compliers with a different, less polit- points system and included a bonus for completing all waves in order to dis- ically engaged profile (20). Further, our study was conducted incentivize attrition. Participation was voluntary, and respondents were able to opt out from the passive metering part of the study at any time. Respon- during the Trump presidency, when a polarized and high-volume dents were sampled according to YouGov’s demographic/political targets political media environment plausibly pretreated these engaged and then reweighted in order to obtain a sample that is representative of citizens, potentially attenuating the effects of our randomized the US population (54). encouragements (46). Understanding the limited effects of expo- Specifically, respondents were weighted to a sampling frame constructed sure to partisan media on relatively engaged partisans is itself of from the full 2016 American Community Survey 1-y sample. The sample cases substantive interest, given the outsize visibility and political influ- were weighted to the sampling frame using propensity scores. The weights ence of this group (8). At the same time, it is precisely people were then poststratified on 2016 presidential vote choice and a four-way with these characteristics—interested in politics but not frequent stratification of gender, age (four categories), race (four categories), and partisan news consumers—who may have less crystallized views education (four categories) to produce the final weights. about the media in general. Our suggestive results on the long-term consequences of Digital Trace Data. The Pulse panel uses passive metering technology devel- oped by Reality Mine which relies on a combination of proxy connections increased exposure to online partisan news for overall trust and browser plug-ins to collect real-time data on web visits and mobile in the media point to a possible resolution of a related puz- app use. After preprocessing, we are left with 19,105,773 URL visits (lap- zle in research on consumption of online news and its effects: top/desktop and mobile/tablet combined) covering the period September controlled experiments have often found that exposure to par- 1 to October 28, 2018. Respondents provided informed consent and were tisan media content causes polarization, although whether this given the ability to pause or halt data sharing at any time; see below for is driven by the inadvertent audience (11) or already polarized details on consent and privacy protections and refs. 5, 8 for extensive val- partisans (27, 47) remains the subject of debate. At the same idation. We also scraped homepages and article text from Fox News and time, fine-grained evidence on people’s online news diets reveals HuffPost stories published in 2018. that the actual consumption of partisan content is limited and News visits were identified based on the list of top 500 domains shared on Facebook and compiled in ref. 3. We classify domains as liberal or conserva- that exposure to cross-cutting information is relatively common tive based on their audience ideology score (negative for liberal and positive on the web and on social media (3, 8, 48, 49). If echo cham- for conservative), which was computed by ref. 3 based on the self-reported bers are not as common as popular discourse suggests, then can political identity of Facebook users that shared those URLs on Facebook. online partisan media be to blame for polarization (50)? Our In addition to web data, we collected public Twitter data linked to findings on lowered overall trust in mainstream media suggest an respondents who agreed to share their profile information with the answer: even if people are regularly exposed to content (partisan researchers via an approved authentication app. In addition to survey or not) that challenges them, they may more heavily discount it respondents’ posted tweets, we periodically collected the friends (fol- as their trust and confidence in news decreases. In this way, parti- lowees) and followers of participants’ Twitter accounts during the study san media can help drive polarization—although not necessarily period. by influencing opinions directly, as the null effects on our index of issue positions indicates. This conclusion has mixed norma- Survey Waves and Treatment. Fig. 1 provides an overview of the data collec- tion process. We conducted multiple survey waves: a baseline survey (wave tive implications: it may be more difficult than ever for people to 1, July 3 to 22, 2018; N = 1, 551), a survey with additional pretreatment be swayed by potentially inflammatory partisan content encoun- covariates (wave 2, August 28 to September 10, 2018; N = 1, 339), another tered on the internet, but this may come at the cost of reduced survey with pretreatment covariates in which the treatments were admin- trust in crucial informational intermediaries that work to sus- istered (wave 3, October 5 to 29, 2018; N = 1, 098), and a posttreatment tain agreement on a shared set of facts and norms. This is an survey that contained our outcome measures and was fielded immediately important topic for continued research. prior to the midterm elections (wave 4, October 30 to November 5, 2018; In a direct test of the power of defaults to structure the infor- N = 1, 197). An additional wave was designed to test for persistence of mation environment, our primary nudge urged participants to effects (wave 5, December 20, 2018, to January 7, 2019; N = 1, 195), and set their browser homepages—an option they may not have been three more waves (not part of our preanalysis plan) were fielded through October 2019. aware existed (51). As illustrated by our tracking data, this sim- In the first two waves of the survey, we asked a battery of questions ple change had lasting impact on downstream web consumption, about news media consumption, attitudes about domestic and foreign pol- icy issues, turnout and vote choice, and presidential approval (wave 1) as well as affective polarization, social distance, news reception, policy atti- ‡As we note in SI Appendix, it is possible that the compliers additionally differ from the tudes, and projected US midterm election outcomes (wave 2). Question full sample in unobserved ways. Beyond this, it is important to recall that our sample is wordings and response options for all survey items in all waves used in the itself a subsample of respondents willing to share private web data with researchers. study are available in SI Appendix.

6 of 8 | PNAS Guess et al. https://doi.org/10.1073/pnas.2013464118 The consequences of online partisan media Downloaded by guest on September 25, 2021 Downloaded by guest on September 25, 2021 h osqecso niepria media partisan online of consequences The al. et Guess ueo ramn aeu,wihw osrc sn e ii aai the in data In visit encouragement. web the using receive construct subjects we after which mea- week take-up, credible a treatment requires of This sure (56). framework instrumental-variables an use and covariates. shrinkage select absolute to (least procedure LASSO operator) a selection use randomization we blocked regressions, our all enable For given procedure. effects methods treatment Both of (55). estimation unbiased regressions saturated and difference-in-means in 4. available and are Analysis. 3 attrition, the waves and subset in wordings we part question Appendix SI analyses, took all to size including primary who specifically sample surveys, our respondents trust our those For reduces (including This only News). trust media to Fox and sample and use, HuffPost media in effects social the political and about inten- trade, beliefs vote knowledge, domestic of perceptions, and threat included about political resentment, approval and racial attitudes problem, presidential tion/choice, 3 as polarization, important wave affective such most than perceptions, variables, longer issues, outcome was foreign of wave battery This a election). the turnout before and ther- ques- midterms, feeling the few presidential after a problem, preference intention. included important domestic control about also most Congress attitudes mometer, 3 issues, incen- covariates: standard Wave foreign pretreatment to other and surveys). a addition to with the related (in so tions completing $8 doing of for to for category equivalent how tives compensated reward sixth for were points instructions a and YouGov tailored encouragement and clear, the Edge, received follow Explorer, Respondents Internet others). for Safari, Firefox, (Chrome, the during data web-tracking empirical polit- period. on current pretreatment based the not in also was HuffPost but and sites environment News news Fox ical these of significance of the and Selection on pages newsletters. based only Facebook email related corresponding another for sites’ and up the sign (foxnews.com), follow to outlet a respondents news to asked asked group). right-leaning (control homepage encouragement was a no browser received to third third their it another change change (huffingtonpost.com), to to outlet asked one was news encouragement: sample left-leaning randomized the a of administered third we experiment, the For onohrrsacesi tiigt etteecalne ihrobust with privacy challenges obtained, is these consent meet informed to that We striving ensure (57). to in designed methods procedures research researchers science typ- other social those join beyond traditional challenges with ethical associated and ically privacy poses responses survey to be Ethics. potentially and could Consent compliance preregistered which we in which ways variable, possible measured. this many for the rules given coding and definition our § .S yna,D .Kinder, news R. D. online Iyengar, and online Americans’ S. chambers, on evidence 9. echo New moderation: bubbles, in everything Filter (Almost) Guess, Rao, M. A. M. J. 8. Goel, S. offline. Flaxman, and S. online segregation 7. Ideological US Shapiro, 2016 M. the J. in Gentzkow, websites news untrustworthy M. to Fake Exposure 6. Reifler, Lazer, J. Nyhan, D. B. Guess, Swire-Thompson, M. A. B. and 5. Friedland, news L. diverse Joseph, ideologically to K. Exposure Grinberg, Adamic, N. A. L. 4. Messing, S. Bakshy, E. 3. Tufekci, Z. 2. Morozov, E. 1. encouragements. See oetmt h ramn fetaogcmles rteCC,w also we CACE, the or compliers, among effect treatment the estimate To day (the 5 November to 30 October from field the in was 4 Wave type browser pretreatment detected by blocked was Randomization 2018. 5, October on starting 3 wave in deployed was treatment The Uiest fCiaoPes hcg,1987). Chicago, Press, Chicago of (University diets. media consumption. (2011). 1799–1839 126, election. presidential election. US 2016 the during (2019). Twitter on Facebook. on opinion 2017). Press, University 2012). IAppendix SI eetmt vrg ramn fet sn ohweighted both using effects treatment average estimate We a.Hm Behav. Hum. Nat. . wte n erGs h oe n rglt fNtokdProtest Networked of Fragility and Power The Gas: Tear and Twitter 011/js159(2021). 10.1111/ajps.12589 Sci., Polit. J. Am. h e euin h akSd fItre Freedom Internet of Side Dark The Delusion: Net The ul pn Q. Opin. Publ. o eal nhww hs hs ie n h ultx fthe of text full the and sites these chose we how on details for olcigpsiebhvoa aata a elinked be can that data behavioral passive Collecting Science esta atr:Tlvso n mrcnOpinion American and Television Matters: that News 7–8 (2020). 472–480 4, 80 1013 (2015). 1130–1132 348, S) 9–2 (2016). 298–320 (S1), N = ,07 diinldtisaotthe about details Additional 037. 1, § h norgmn also encouragement The IAppendix SI Science 374–378 363, (PublicAffairs, especify we .J Econ. J. Q. (Yale 6 .F aased .Brlo,H Gaudet, of H. foundations Berelson, changing B. The Lazarsfeld, effects? F. P. minimal of media 16. era political new online A Iyengar, of S. test Bennett, experimental W. An 15. measure: news for of Measure measurement Guess, better M. through A. research 14. effects media Improving Prior, M. 13. 2 .D eeitsKsnr .A am .J eisy .Ymmt,Prudn the Persuading Yamamoto, T. Berinsky, J. A. Baum, A. M. heterogeneous Benedictis-Kessner, De J. of 12. estimation Experimental Johnson, M. Arceneaux, Kuklinski, K. 11. H. J. Gaines, J. B. 10. epu n osrciefebc ntemanuscript. their the for on reviewers feedback anonymous constructive five and and editor feed- helpful the helpful Brandice thank for especially Gilardi and We Fabrizio back. Politics; and in Anselm Wessels, and Workshop Munger, Bernhard Kayser, Kevin Media Bias Mark Kalmoe, Hager, on Media Nathan Gentzkow, Symposium of Matthew Michigan Canes-Wrone, Economics the Fourth Uni- and Stanford the Berlin and Los at Urbana-Champaign, California, participants at of Illinois versity; School, University of Lab, University Mas- Hertie the Media the the Angeles, University, Technology Leiden of University, Konstanz, Institute Rutgers of sachusetts University at assistance. Frankfurt, research Califor- attendees of excellent Southern University for workshop Schulz of Will thank University and the We Ronen Jona at thank Studies We Princeton and International nia. the Sciences for Social by and Center supported Humanities the the additionally in Research was on experiment Committee refer- University The Initiative, 143. Science 92 Social ence Computational Foundation Volkswagen the repository: GitHub ACKNOWLEDGMENTS. following the at at available is available plan are article https://github.com/NetDem-USC/homepage-experiment this in results Availability. Data Pulse YouGov the of time.” part any as which at in tracking, participate panel, web to of any agreed out previously answer opt have to to may free you not voluntary are you choose and Furthermore, may questions. You survey all consumption. or short media a online of of vol- tracking completion is participation involves “Your them, Participation informing agreed untary. statement additionally consent respondents collection separate study, a to specific to agree this they charac- joining that are sociodemographic Before and part, they teristics. behavior in internet any panel, says, their at Pulse about that information removed the policy of be join privacy can clear subjects software a When the given temporarily. that paused with told shared the or not also of are time are nature data that They the and about parties. anonymous, told kept third is are it panel that Pulse collected, the data join who People consent. replication as released being are counts aggregates with these (as data. Only level visits). respondent of the shares to individual- or only aggregated using are data paper that survey the variables for the level analyses from primary location the separate performing themselves strictly by URLs a and in the securely pages, stored these data visit of information. content such the contain collect page). could profile not possi- Facebook did contain own is that we one’s it visiting queries While by search information, example, made (for identifying or details websites for personal visited ask subjects some not that do ble we While researchers. designated a via Illinois of University agreement. the University IRB by California Princeton Southern authorized of of and University IRBs the (UP-17-00513) the and 10041) by or best and approved 10014, emerging 8327, was stored (protocols and protocol study guidance strict This not a (IRB) (58). followed Board is practices we Review study, Institutional information this by in identifiable informed analyzed data personally the For and reported. protected, is ¶ rtpae n oGvprom niiilsrbo esnlyietfiginformation identifying personally of scrub data. initial the an delivering performs before YouGov and place, first euetascin n asod r o olce rsae ihrsacesi the in researchers with shared or collected not are passwords and transactions Secure eod hrn e iiswt eerhr sdn ihflyinformed fully with done is researchers with visits web sharing Second, the to anonymous are respondents research, survey any with as First, pHsMn naPeieta Campaign Presidential a in 1944). Mind His up communication. political exposure. exposure. (2019). nm:Etmtn h esaieefcso atsnmdawt h preference- the with design. media assignment partisan and of choice effects persuasive incorporating the Estimating enemy: Politics, American in Studies (Chicago 2013). Choice of Age an self-selection. to related (2011). effects treatment oi.Anal. Polit. Polit. J. 9–0 (2009). 893–908 71, aafie n cit eesr orpiaethe replicate to necessary scripts and files Data https://osf.io/zj65h. 97 (2015). 59–75 23, hsrsac a eeosyfne yagatfrom grant a by funded generously was research This .Commun. J. hnigMnso hnigCanl? atsnNw in News Partisan Channels?: Changing or Minds Changing ¶ ehnl hsb epn h a web raw the keeping by this handle We 0–3 (2008). 707–731 58, h epesCoc:HwteVtrMakes Voter the How Choice: People’s The https://doi.org/10.1073/pnas.2013464118 Clmi nvriyPes e York, New Press, University (Columbia m oi.Si Rev. Sci. Polit. Am. m .Plt Sci. Polit. J. Am. nvriyo hcg Press, Chicago of University 5) Preanalysis (59). PNAS 902–916 113, 724–736 55, | f8 of 7

POLITICAL SCIENCES 17. S. DellaVigna, E. Kaplan, The Fox News effect: Media bias and voting. Q. J. Econ. 122, 40. Y. Benkler, R. Faris, H. Roberts, Network Propaganda: Manipulation, Disinformation, 1187–1234 (2007). and Radicalization in American Politics (Oxford University Press, 2018). 18. A. S. Gerber, D. Karlan, D. Bergan, Does the media matter? A field experiment mea- 41. U. Sehgal, Jon Stewart’s rare, unexpectedly serious interview with Fox News suring the effect of newspapers on voting behavior and political opinions. Am. Econ. (2011). https://www.theatlantic.com/culture/archive/2011/06/jon-stewart-makes-rare- J. Appl. Econ. 1, 35–52 (2009). appearance-fox-new-call-its-viewers-misinformed/351904/. Accessed 9 March 2021. 19. D. J. Hopkins, J. M. Ladd, The consequences of broader media choice: Evidence from 42. M. B. Kelley, Study: Watching only Fox News makes you less informed than the expansion of Fox News. Qu. J. Polit. Sci. 9, 115–135 (2014). watching no news at all (2012). https://www.businessinsider.com/study-watching- 20. G. J. Martin, A. Yurukoglu, Bias in cable news: Persuasion and polarization. Am. Econ. fox-news-makes-you-less-informed-than-watching-no-news-at-all-2012-5. Accessed 9 Rev. 107, 2565–2599 (2017). March 2021. 21. R. H. Thaler, C. R. Sunstein, Nudge: Improving Decisions about Health, Wealth, and 43. J. L. Kalla, D. E. Broockman, The minimal persuasive effects of campaign contact in Happiness (Penguin, 2009). general elections: Evidence from 49 field experiments. Am. Polit. Sci. Rev. 112, 148– 22. P. Barbera´ et al., Who leads? Who follows? Measuring issue attention and agenda 166 (2018). setting by legislators and the mass public using social media data. Am. Polit. Sci. Rev. 44. A. Coppock, S. J. Hill, L. Vavreck, The small effects of political advertising are small 113, 883–901 (2019). regardless of context, message, sender, or receiver: Evidence from 59 real-time 23. G. King, B. Schneer, A. White, How the news media activate public expression and randomized experiments. Sci. Adv. 6, eabc4046 (2020). influence national agendas. Science 358, 776–780 (2017). 45. S. Munzert, P. Barbera,´ A. Guess, J. Yang, Do online voter guides empower cit- 24. A. Mitchell “Americans still prefer watching to reading the news – and mostly izens? Evidence from a field experiment with digital trace data. Publ. Opin. Q., still through television” (Pew Research Center, December 3, 2018). https://www. 10.1093/poq/nfaa037 (2021). journalism.org/wp-content/uploads/sites/8/2018/12/PJ 2018.12.03 read-watch-listen 46. J. N. Druckman, T. J. Leeper, Learning more from political communication experi- FINAL1.pdf. Accessed June 4, 2020. ments: Pretreatment and its effects. Am. J. Polit. Sci. 56, 875–896 (2012). 25. J. R. Zaller, The Nature and Origins of Mass Opinion (Cambridge University Press, New 47. M. Levendusky, How Partisan Media Polarize America (University of Chicago Press, York, 1992). 2013). 26. G. S. Lenz, Follow the Leader? How Voters Respond to Politicians’ Policies and 48. G. Eady, J. Nagler, A. Guess, J. Zilinsky, J. A. Tucker, How many people live in political Performance (University of Chicago Press, 2013). bubbles on social media? Evidence from linked survey and Twitter data. Sage Open 27. C. A. Bail et al., Exposure to opposing views on social media can increase political 9, 2158244019832705 (2019). polarization. Proc. Natl. Acad. Sci. U.S.A. 115, 9216–9221 (2018). 49. M. Scharkow, F. Mangold, S. Stier, J. Breuer, How social network sites and other online 28. J. N. Druckman, S. Gubitz, A. M. Lloyd, M. S. Levendusky, How incivility on partisan intermediaries increase exposure to news. Proc. Natl. Acad. Sci. U.S.A. 117, 2761–2763 media (de) polarizes the electorate. J. Polit. 81, 291–295 (2019). (2020). 29. M. S. Levendusky, N. Malhotra, (Mis) perceptions of partisan polarization in the 50. C. R. Sunstein, #Republic: Divided Democracy in the Age of Social Media (Princeton American public. Publ. Opin. Q. 80 (S1), 378–391 (2016). University Press, 2017). 30. J. Ladd, Why Americans Hate the Media and How It Matters (Princeton University 51. E. Hargittai, M. Micheli, “Internet skills and why they matter” in Society and the Press, 2011). Internet: How Networks of Information and Communication are Changing Our Lives, 31. M. Jurkowitz, A. Mitchell, E. Shearer, M. Walker, “U.S. media polarization M. Graham, W. H. Dutton, Eds. (Oxford University Press, 2019), p. 109. and the 2020 election: A nation divided” (Pew Research Center, January 24, 52. P. Lorenz-Spreen, S. Lewandowsky, C. R. Sunstein, R. Hertwig, How behavioural sci- 2020). https://www.journalism.org/2020/01/24/u-s-media-polarization-and-the-2020- ences can promote truth, autonomy and democratic discourse online. Nat. Hum. election-a-nation-divided/. Accessed June 9, 2020. Behav. 4, 1102–1109 (2020). 32. M. Barnidge et al., Politically motivated selective exposure and perceived media bias. 53. M. Prior, Post-Broadcast Democracy: How Media Choice Increases Inequality Commun. Res. 47, 0093650217713066 (2017). in Political Involvement and Polarizes Elections (Cambridge University Press, 33. R. P. Vallone, L. Ross, M. R. Lepper, The hostile media phenomenon: Biased perception 2007). and perceptions of media bias in coverage of the Beirut massacre. J. Pers. Soc. Psychol. 54. D. Rivers, Sampling for web surveys. http://citeseerx.ist.psu.edu/viewdoc/download? 49, 577 (1985). doi=10.1.1.468.9645&rep=rep1&type=pdf. Accessed 12 March 2021. 34. A. C. Gunther, K. Schmitt, Mapping boundaries of the hostile media effect. J. 55. W. Lin, Agnostic notes on regression adjustments to experimental data: Reexamining Commun. 54, 55–70 (2004). freedman’s critique. Ann. Appl. Stat. 7, 295, 318 (2013). 35. M. R. Perryman, Biased gatekeepers? Partisan perceptions of media attention in the 56. A. S. Gerber, D. P. Green, Field Experiments: Design, Analysis, and Interpretation 2016 U.S. presidential election. Journal. Stud. 20, 1–18 (2019). (W. W. Norton, 2012). 36. K. Arceneaux, M. Johnson, C. Murphy, Polarized political communication, opposi- 57. S. Stier, J. Breuer, P. Siegers, K. Thorson, Integrating survey data and digital trace tional media hostility, and selective exposure. J. Polit. 74, 174–186 (2012). data: Key issues in developing an emerging field. Soc. Sci. Comput. Rev. 38, 503–516 37. V. Price, J. Zaller, Who gets the news? Alternative measures of news reception and (2019). their implications for research. Publ. Opin. Q. 57, 133–164 (1993). 58. E. Menchen-Trevino, “Digital trace data and social research: A proactive research 38. B. Albertson, A. Lawrence, After the credits roll: The long-term effects of educational ethics” in The Oxford Handbook of Networked Communication, B. F. Welles, television on public knowledge and attitudes. Am. Polit. Res. 37, 275–300 (2009). S. Gonzalez-Bail´ on,´ Eds. (Oxford University Press, 2018), pp. 1–20. 39. R. Levy, Social media, news consumption, and polarization: Evidence from a field 59. A. Guess et al., Data from “The consequences of online partisan media.” GitHub. experiment. Am. Econ. Rev. 111, 831–870 (2021). https://github.com/NetDem-USC/homepage-experiment. Deposited 9 March 2021.

8 of 8 | PNAS Guess et al. https://doi.org/10.1073/pnas.2013464118 The consequences of online partisan media Downloaded by guest on September 25, 2021