Downloaded by guest on October 1, 2021 iscnsaentol au ttmnsbtas eif about beliefs also partisan but that statements value is 13, only research (11, not attitude crowds shape political of can of wisdom bias finding the key of A studies 14). esti- been experimental frequently nonpartisan have in the that in used estimates) present distance (e.g., not tasks are mation that biases cognitive by true to the toward response belief in 15). mean (13, less the answer answers pulling their individuals thus information, that revise social observation accurate the more by are the explained who improve be observe 14). can (13, can can similar effect more exchange This become group individuals information as a even members, accuracy in group other individuals of when value beliefs that the about shown research concerns experimental has However, raised (12). accuracy has democracy belief deliberative that of undermine argument to an expected 11), (10, are 11). groupthink 10, herd- of (3, and dynamics aggregate the ing other, their in each influence out that can individuals cancel such When within and independent, uncorrelated individuals statistically are beliefs, be errors accurate states must produce which groups theory, to those crowds groups of for wisdom that the in premise common of (7–9). aggregation biases the group responses—amplify with survey contrast independent that processes—in of argument influence theories the social by normative challenged 6), remain (5, democracy deliberative principles optimistic democratic offer theories for statistical individ- support these are While members inaccurate. group “wise”— ually be when can (2–4) accurate—even groups crowds” factually large of of i.e., beliefs “wisdom aggregated the the found on has research yet (1), deliberation 60626; IL Evanston, a Becker crowds Joshua partisan of wisdom The www.pnas.org/cgi/doi/10.1073/pnas.1817195116 A democracy deliberative intelligence collective generalize help results domains. accu- Our political to increased polarization. research intelligence reduced only collective also not but networks In racy homogeneous network. in social information social homogeneous that exchange found politically we esti- theories, a polarization the to in observing contrast questions after peers and factual of before bias mates answered partisan elicit individuals to known which web-based two wisdom in conducted the we bias, experiments whether partisan test to To robust is accuracy. crowds belief of reduce parti- and amplify bias to formation, san expected belief are accurate networks informa- homogeneous promote and politically to social Democrats expected both to are containing response Republicans networks diminished bipartisan a While extreme with more tion. with also only but not associated beliefs, are concern biases primary partisan A bias. that political is partisan by motivated accurate—when are less beliefs extreme—and imply more noncon- polarization become will on political groups of that accuracy theories belief matters, factual increase troversial that group can found the has improve exchange crowds” on of can information “wisdom based the processing on are research information While democracy beliefs. social deliberative 11, April that Richeson of A. premise Jennifer favor Member Board in Editorial by Theories accepted and 2018) MA, 5, Cambridge, October Technology, review of for Institute (received Massachusetts 2019 Rand, G. David by Edited and 20052; DC Washington, University, Washington nebr colfrCmuiain nvriyo enyvna hldlha A19104; PA Philadelphia, Pennsylvania, of University Communication, for School Annenberg hl uhrslsaepoiig oiia eif r shaped are beliefs political promising, are results such While a from derives democracy deliberative against argument One ipytoinrn fbscpltclfcst eetfrom benefit to facts political basic of ignorant too simply ao ocr o eortcterssi htctzn are citizens that is theorists democratic for concern major a,b,c,1 c otwsenIsiueo ope ytm,Nrhetr nvriy vntn L60626; IL Evanston, University, Northwestern Systems, Complex on Institute Northwestern | polarization ta Porter Ethan , | networks d n ao Centola Damon and , | h idmo crowds of wisdom the e colo niern,Uiest fPnslai,Piaepi,P 19104 PA Philadelphia, Pennsylvania, of University Engineering, of School a,e | 6 6 a encnendpiaiywt tiuedifferences attitude with primarily concerned accuracy been belief (9, has polarization 26) than political information on 16, rather research social previous extremity However, to 26). belief (25, response with topics, correlated partisan is for that instead tation and processing 26). 25, information (9, expected biases social partisan the amplify of reverse party—will effects political learning political one only of of networks—containing theories members homogeneous However, and that 24). accuracy maintain (20, belief polarization improve biases to partisan found reduce been has Republicans, influence and Democrats social hetero- both politically containing In networks (23). of geneous exposure information supportive can by less accuracy improved that be be suggests would evidence experimental beliefs loyalties, corrected party when and support soning to (18). world politicians the or such parties about different attitudes, beliefs each for their political preferences with their adjust affect consistent will can be people preference to that beliefs This their (22). adjust other will means people which that consistency, the cognitive from for results preference reasoning psychological Motivated is beliefs (21). One factual reasoning 20). in bias (17, motivated partisan accuracy of emergence their the for for respondents incentive explanation survey financial when a even offered persist are biases Such (16–19). facts aadpsto:Tedt eotdi hspprhv endpstdi h Harvard the in (https://github.com/ deposited GitHub been have on paper joshua-a-becker/wisdom-of-partisan-crowds and this in (https://doi.org/10.7910/DVN/OE6UMR) reported Dataverse data The deposition: Data the under Published 1 Editorial the by invited editor guest a is D.G.R. Submission. Direct Board. PNAS data- a is the article This constructed interesty of D.C. conflict no and declare platform authors J.B. The data-collection the paper; constructed E.P. J.B. the 2.y and and Experiment wrote 1; for J.B. Experiment E.P. data; for collected and platform J.B. collection J.B. research; designed data; E.P. analyzed and J.B. contributions: Author ulse nieMy1,2019. 13, May online Published 1073/pnas.1817195116/-/DCSupplemental. y at online information supporting contains article This owo orsodnesol eadesd mi:[email protected]. Email: addressed. be should correspondence whom To o fcod srbs opria is oilinfluence polarization Social decreases also bias. ties. but partisan network between-group accuracy wis- to without increases the robust that only is find not crowds we However, of party. members dom political with own only their communicate of people net- when heterogeneous amplify bias to politically partisan expected are in networks social emerge homogeneous to risk works, this expected While not bias. political is partisan by that accurate—when motivated imply less are in extreme—and beliefs polarization more accuracy political become of will belief groups theories increase but can cases, many exchange found has information crowds” improve of can “wisdom that the processing on on Research information beliefs. based social group are that democracy premise deliberative the of theories Normative Significance h iko ooeeu ewrsdrvsfo h expec- the from derives networks homogeneous of risk The rea- motivated by shaped are beliefs inaccurate when Even y b elg colo aaeet otwsenUniversity, Northwestern Management, of School Kellogg PNAS NSlicense.y PNAS | a 8 2019 28, May d colo ei n ulcAfis George Affairs, Public and Media of School ). y | o.116 vol. www.pnas.org/lookup/suppl/doi:10. | o 22 no. | 10717–10722 y

POLITICAL SCIENCES and has not directly examined the effect of social influence on which questions were presented (SI Appendix). For each ques- belief accuracy. To understand the potential effects of partisan tion set, we collected data for 3 networked groups and 1 control bias on the wisdom of crowds, we first study a formal model group for each political party (i.e., 4 independent groups for each of belief formation to generate hypotheses relating polarization party). In total, we collected data for 12 networks and 4 con- theories to political belief accuracy. This model is formally iden- trol groups for each party (1,120 subjects in total). SI Appendix, tical to that used in previous research on the wisdom of crowds Fig. S1 illustrates our experimental design. (13, 27), but parameterized to account for a possible correlation The experimental questions have true answers with values between belief extremity and adjustment to social information. ranging from 4.9 to 224,600,000. To compare across questions, Echoing previous experimental findings (20), this model shows we follow similar studies (11) and log-transform all responses that opposing biases cancel out in in politically diverse biparti- and true values before analysis using the natural logarithm. This san networks, leaving the average belief unchanged even when allows for comparison across conditions because log(A) − log(B) bias is correlated with response to social information. However, approximates percentage of difference, and thus calculated in politically homogeneous “echo chamber” networks, a correla- errors for each response are approximately equal to percentage tion between bias and adjustment causes group beliefs to become of error. This also accounts for the observation that estimates of more extreme and less accurate (SI Appendix, Fig. S4), consistent this type are frequently distributed log-normally (11, 28). We find with political theories of polarization (26) (see SI Appendix for that alternative normalization procedures produce comparable detailed model results). results (SI Appendix). To test whether the wisdom of crowds is robust to partisan Because responses by individuals within a social network are bias, we conducted two web-based experiments examining social not independent, we measure all outcomes at the trial level. To influence in homogeneous social networks. Contrary to predic- produce this metric, we first calculate the mean (logged) belief tions based on the “law of group polarization” (26) we find of the 35 responses given for a single round of a single question that homogeneous social networks are not sufficient to amplify in a single trial. We then measure group error for each round partisan biases. Instead, we find that beliefs become more accu- of each question as the absolute value of the arithmetic differ- rate and less polarized. These results suggest that prior models ence between the mean (logged) belief and the (logged) true of the wisdom of crowds generalize to factual belief forma- value. We then measure the change in error for each question tion on partisan political topics even in politically homogeneous of each trial as the arithmetic difference between the error of networks. the mean at Round 1 and the error of the mean at Round 3. This method produces four measurements of change in error for Experimental Design each trial, i.e., one for each question. We then calculate the aver- Following a preregistered experimental design, our first exper- age of this value over all four questions completed by each trial iment asked subjects recruited from Amazon Mechanical Turk to measure average change in error for each trial. We thus pro- to answer four fact-based questions (e.g., “What was the unem- duce 24 independent observations of the effect of social influence ployment rate in the last month of Barack Obama’s presidential on group accuracy when beliefs are motivated by partisan bias, administration?”). Subjects were compensated for their partic- including 12 independent observations of Republican networks ipation according to the accuracy of their final responses. The and 12 independent observations of Democrat networks. In addi- four questions used in this experiment (Materials and Methods) tion, we produce 8 independent control observations, including were selected because they showed the greatest levels of partisan 4 independent observations of Republican control groups and 4 bias among 25 pretested questions. independent observations of Democrat control groups. Subjects were randomly assigned to either a social condition or We replicated this entire design in a second experiment, with a control condition. For each question, subjects first provided an modifications intended to increase the effect of partisan bias independent answer (“Round 1”). In the social condition, sub- on responses to social information. We describe this replication jects were then shown the average belief of four other subjects below after presenting the results from Experiment 1. connected to them in a social network and were prompted to pro- vide a second, revised answer (“Round 2”). Subjects in the social Results (Experiment 1) condition were then shown the average revised answer of their We find no evidence that social influence in homogeneous net- network neighbors and were prompted to provide a third and works either reduces accuracy or increases polarization on fac- final answer (“Round 3”). In the control condition, subjects were tual beliefs. Instead, we find that social influence increased accu- prompted to provide their answer three times, but with no social racy for both Republicans and Democrats and also decreased information. Besides the absence or presence of social informa- polarization despite the absence of between-group ties. We begin tion, subject experience was identical in both social and control our analysis by confirming that in Experiment 1, subjects’ inde- conditions. Subjects in both conditions were provided 60 s to pro- pendent beliefs demonstrated partisan bias, as expected based vide their answer each round, for a total of 3 min per question. on previous research (5, 17, 20). In Round 1 (before social As soon as subjects provided their response, they were advanced influence), responses provided by Democrats were significantly to the next round, even if there was time remaining. different from responses provided by Republicans for all ques- Each trial contained 35 subjects. For each trial in the social tions (Fig. 1 and SI Appendix; P < 0.001 for all questions except condition, all subjects participated simultaneously. Subjects in race in California, for which P < 0.05). the social condition were connected to each other in random net- To illustrate the change in beliefs for each question, Fig. 1 works in which each subject observed the average response of shows the truth-centered mean of normalized beliefs (so that a four other subjects and was observed by those same four sub- negative value indicates an underestimate, and a positive value jects, forming a single connected network of 35 subjects. To indicates an overestimate) in social conditions at each round of test whether the wisdom of crowds is robust to partisan bias in both experiments. The value for each data point is obtained by politically homogeneous networks, each trial in each condition calculating the arithmetic difference between the mean belief consisted of either only Republicans or only Democrats. Sub- and the true value at each round for each question and then jects in the social condition interacted anonymously and were not averaging this value across all 12 social network trials for each informed that they were observing the responses by people who political party. In every case, the average estimate became closer shared their partisan preferences. to the true value after social influence. We controlled for question order effects by using four ques- To test whether this change could be explained by random fluc- tion sets, each of which was identical except for the order in tuation, we calculate the error for each round of each question

10718 | www.pnas.org/cgi/doi/10.1073/pnas.1817195116 Becker et al. Downloaded by guest on October 1, 2021 Downloaded by guest on October 1, 2021 rusdcesdb ny1% usatal mle change control smaller in substantially mean a the 15%, only of by by error explained decreased The be groups effects. cannot learning networks social individual the homogeneous in in that than error (P greater decrease significantly condition 8 did was control the condition error social and that (P the find trials condition in We network control 2. the social in Fig. 24 slightly in the shown for trials, error control aver- in the calculate change in condition. we age control above, improvement answers. described the compared procedure their in we the improvement revise Following case, with to the condition is social subjects the this for whether opportunity test the To to due is 1 Round from 35% by that decreased 3. find mean Round we the to of Democrats, error and average Republicans the (P both trials Across Democrat 1. 12 (P the iment trials of Republican at one 12 error every the than as of of well lower one error significantly every the was for influ- that 3 1 find social Round Round we at after condition, belief value the social mean the true the average, in the those on to For whether, closer ence. within determines became error mean analysis absolute group four This in all across change trial. value average each this average measure error and to in 3 change questions Round the to abso- calculate 1 the then Round (i.e., We from mean truth). truth-centered from the distance of lute value absolute the as overestimated subjects (e.g., overestimates the and and sign, under- immigration). the relative reverses show 2 normaliza- to Experiment the questions in unemployment) For (immigration, process Democrats. answer tion true by negative responses a indicates with blue responses indicates the and Red line). to question. Republicans, (dashed due one by point shows groups panel social per Each than trials size. variation sample control smaller random 4 more or show line) groups (solid Control point per trials social 1. Fig. ekre al. et Becker Truth-Centered Mean Truth-Centered Mean Truth-Centered Mean n osblt sta mrvmn ntesca condition social the in improvement that is possibility One omlzd rt-etrdma tec on,aeae cos12 across averaged round, each at mean truth-centered Normalized, California (Exp. 1) California (Exp. Soldiers (Exp. 2) Soldiers (Exp. Election (Exp. 1) Election (Exp. < .3,idctn htterdcinin reduction the that indicating 0.03), Unemployment (Exp. 2) Unemployment (Exp. Immigration (Exp. 2) Immigration (Exp. Taxes (Exp.1) < .5,btta h change the that but 0.15), < Unemployment (Exp. 1) Unemployment (Exp. .0)i Exper- in 0.001) y Military (Exp.2) xsi inverted is axis < .0)as 0.001) istetnec oudr roeetmt—a ntesame the the in overestimate—was capture estimation or that fully under- is not to observation tendency notable did A bias—the 1 bias. partisan Experiment of that effects is possibility One 2) (Experiment Priming Partisan to in Robustness decrease 33% networks. a social in with individuals compared by error individuals isolated of in error than groups (P control networks in social smaller (P significantly groups was control individual in improvement slightly average (P decreased that also networks net- error find ual social social we 24 in 1, all decreased across Experiment error averaging For all then across trials. and averaging work then trial, question, a given individu- in This a all questions for average). across group the error a averaging of within first als error by the measured to is quantity opposed (as error individual group among similarity increased members. to leads networks social neous ok a infiatygetrta hnei oto groups control in change than greater (P significantly was works (P groups control Round to (P for 1 networks Round social from in significantly 3 questions. decreased and four SD before that all 1 find across We Experiment averaging in exchange, groups responses information networked of To after 24 SD the directions. the of measure opposite each first by in we possibility, moved would this extremists dispersed—e.g., which investigate widely and accurate, more moderates more become if became beliefs individual mean if group occur the as even bias. partisan of effects infor- the may social counteract groups—can pro- help that partisan answer homogeneous while suggest in their results processing—even Thus, revise mation these networks. accuracy, to belief social opportunity improve in the individuals decrease viding 35% the than .Errbr ipa tnad9%cndneitra rudtemean. the around interval of confidence 95% round standard each display at bars Error line) 3. (dashed conditions (Right control experiment. 8 the and line) (solid ditions 2. Fig. eas ietyts h feto oiliflec naverage on influence social of effect the test directly also We nte osblt sta niiul eaels accurate less became individuals that is possibility Another < .0) ugsigta nomto xhnei homoge- in exchange information that suggesting 0.001), (Left Error At Each Round Each Error At Change Cumulative Round Each Error At omlzderro h en vrgdars 4sca con- social 24 across averaged mean, the of error Normalized ) < uuaiecag nerrfo on oRound to 1 Round from error in change Cumulative ) PNAS .0) iha7 eraei h average the in decrease 7% a with 0.001), Social Control Social .5.W n httecag nnet- in change the that find We 0.25). = < Control | .0)btddntsgicnl change significantly not did but 0.001) a 8 2019 28, May Experiment 1 Experiment 2 | Change Round 1 to 3 Change Round 1 to 3 < o.116 vol. .0) hl individ- While 0.001). Cumulative Change | o 22 no. < .1,the 0.11), | 10719

POLITICAL SCIENCES direction for both Republicans and Democrats. However, nearly Replication Results. As with Experiment 1, we begin our replica- all of the 25 pilot questions generated bias in the same direc- tion analysis by ensuring that subjects showed partisan bias, find- tion. We also find this pattern in previous research on partisan ing significant differences between Republicans and Democrats factual beliefs (17), suggesting that same-direction bias is a com- for all four questions (P < 0.001). For the question on unem- mon feature of partisan beliefs. While this same-direction bias ployment, which was reused from Experiment 1 and reframed to runs counter to intuitive expectations about partisan polariza- emphasize change, we now observe a meaningful split between tion, it is consistent with previous research on estimation bias, the two parties: A majority (54%) of Democrats stated that which shows that people have a general tendency to under- unemployment decreased under President Obama, while a or overestimate for any given question (28). The belief dif- majority (67%) of Republicans stated the opposite. Nonetheless, ferences between Democrats and Republicans may be under- the overall numeric bias was still in the same direction: The mean stood as an additional partisan bias added on top of a general answer for both parties was an overestimate. As this example estimation bias. shows, divergent beliefs between Democrats and Republicans Nonetheless, a limitation of Experiment 1 is that questions can nonetheless generate numeric estimation bias in the same were chosen based on the numeric magnitude of bias in pretest- direction. ing and not on the controversial nature of the questions. More- Figs. 1 and 2 show outcomes of the replication. We again find over, the experimental interface was politically neutral and did that social influence increased the accuracy of mean beliefs for not communicate to subjects in the social condition that they both Democrats (P < 0.03) and Republicans (P < 0.001). Across were in homogeneous partisan networks, factors which may have all trials, we found that the error of the mean decreased by 31% prevented subjects from perceiving the questions as partisan in for subjects in the social condition, approximately the same effect nature. We therefore replicated our initial experiment with sev- size observed in Experiment 1. In contrast, we saw a 4% increase eral changes designed to increase the effect of partisan bias on in error for the control condition, although this change was not response to social information. statistically significant (P > 0.46). The two conditions were sig- nificantly different (P < 0.002), indicating that the benefits of Replication Methods. Instead of choosing questions based on social information cannot be explained by individual learning numeric polarization in pretesting, we selected questions based effects. on their connection to controversial policy topics. For example, Similar to Experiment 1, we found that SD decreased signifi- we asked participants about the number of illegal immigrants cantly in the social condition (P < 0.001), but increased slightly in the United States at a time when illegal immigration was in the control condition (P > 0.19) and the two conditions were at the center of national debate (when disagreement over “the significantly different (P < 0.001). This result shows that subjects wall” with Mexico led to a US government shutdown in Jan- became more similar over time as a result of social information, uary 2019). We also framed questions to emphasize change (i.e., indicating that social learning effects are robust to explicit parti- we requested numeric estimates for the magnitude and direc- san primes. In addition to learning at the group level, we found a tion of change) to allow for more partisan expressiveness. We 34% decrease in individual error for subjects in the social con- reused one question from Experiment 1, asking about unemploy- ditions (P < 0.001) and a nominal 3% increase in individual ment, because that question taps into a strong policy controversy error for control subjects (P > 0.74). The two conditions were (the economy) and showed the greatest partisan bias in the significantly different (P < 0.001), showing that social learning first experiment. By reusing this question with an emphasis on is robust to partisan priming for both group-level improvement directional change, we expected to observe demonstration of a and individual improvement. split-direction partisan bias. Exact wording of all four questions is provided in Materials and Methods. Polarization and the Wisdom of Crowds In addition to selecting more controversial questions, we also Results from both experiments show that the wisdom of crowds modified the experimental interface to include partisan primes in networks is robust to political partisan bias. We find that an that have been shown in prior research (20) to enhance the increase of in-group belief similarity generates improvements at effects of partisan bias on social information processing. First, both the group level and the individual level. One risk, however, we required all subjects to confirm their political party before is that this increase of in-group similarity is accompanied by a entering the experimental interface, to prime them to the polit- decrease in between-group similarity, generating increased belief ical nature of the study. Second, we included an image of an polarization even as groups become more accurate. To mea- elephant and a donkey (i.e., symbols for the Democratic and sure belief polarization, we conduct a paired analysis for each Republican parties) on the experimental interface (SI Appendix, experiment, matching the 12 Republican networks with the 12 Fig. S3). Third, for subjects in the social condition, we indicated Democrat networks (based on trial number, per our preregis- the party membership of other subjects in the study when provid- tered analysis) and calculating their similarity at each round (SI ing social information. Finally, subjects upon recruitment were Appendix). invited to participate in the “Politics Challenge,” and the URL We measured polarization using two outcomes. Fig. 3 (Left) to the web platform included the phrase Politics Challenge. shows the average distance (absolute value of the arithmetic dif- Questions in this second experiment allowed negative answers, ference; SI Appendix) between the mean normalized belief for for which the logarithm is not defined, and so we normalize Republicans and the mean normalized belief for Democrats at results by dividing by the true answer, which also represents per- each round of the experiment. Among subjects in the social centage of difference. However, this method leaves our analysis condition, the average distance between the mean belief of extremely sensitive to large values as might occur through typo- Democrats and the mean belief of Republicans decreased graphic error. While these extreme values do not change our sta- by 37% for Experiment 1 (P < 0.01) and 46% for Exper- tistical analysis, the inclusion of all responses yields implausible iment 2 (P < 0.02). In contrast, the distance between the effect sizes. (For example, we find that error in the social condi- mean Republican and Democrat belief nominally increased tion decreased by 3.6 ×107% while error in the control groups for the control condition in both experiments, although the increased by 5.3 ×104%.) We therefore present results in the effects were not statistically significant (P < 0.13 for Experi- main text and figures after manually removing extremely large ment 1, and P > 0.87 for Experiment 2). Overall, the change values, a process which impacts fewer than 1% of responses. An in polarization was significantly different between the control analysis that includes all submitted responses is provided in SI and social conditions (P < 0.01 for Experiment 1, P < 0.08 Appendix. for Experiment 2).

10720 | www.pnas.org/cgi/doi/10.1073/pnas.1817195116 Becker et al. Downloaded by guest on October 1, 2021 Downloaded by guest on October 1, 2021 ekre al. et Becker ok.Hmgnossca ewrs uha hs estudy, we net- those social as segregated such democ- politically networks, In in deliberative social partisan even (13). that Homogeneous possible of works. crowds suggest be effects of results may the racy wisdom our about concerns chambers, the pre- growing echo on as of beliefs research context individuals political the prior factual accurate by on that influence dicted explanation greatest the the results exert with our and social (26), consistent polarization bias homogeneous of are partisan theories politically to by contrast in In shaped networks. embedded were influence, beliefs were social fact-based that of individuals objective, result fact a to the as despite response accurate more mean became the questions that observed We Discussion (P groups control in greater than significantly experiment). was networks distance social pairwise change, in (5% average nominal reli- 2 in a Experiment change decrease showing change, in not (6% information, increase did 1 nal social Experiment value of in this absence decrease the that decrease (P in ( show 37% 2 ably 1 groups Experiment a Experiment control for with decrease for for 48% experiments, distance a in both and pairwise embed- this similar across average more For Republicans trials in became forum. and networks 24 public social Democrats a all homogeneous that in in chance found ded Repub- by a we selected meet and metric, Democrat could randomly ran- the a who reflecting between a a as lican belief of understood of in be distance that belief can expected and outcome and the This Democrat between pairing Republican. distance distance selected cross-party average expected domly two-person the the possible measures reflects every metric Republicans This between individual between Democrats. distance and pairwise average the randomly a between difference Republican. and expected Democrat normalized the selected the measures and which Democrats responses, of (Right belief Republicans. of mean belief (Left mean normalized line). the (dashed groups in control and ference line) (solid networks social both 3. Fig. shows (Right) 3 Fig. polarization, of measure second a As onsidct oaiaina ahrudo h xeietfor experiment the of round each at polarization indicate Points Between Groups Between Groups Social Social Control Control Experiment 2 Experiment 1 vrg ariedsac fnormalized of distance pairwise Average ) P P Between Individuals Between Individuals > < P .2 u nomi- a but 0.12) .0) Outcomes 0.001). .5.Overall, 0.25). = < .1freach for 0.01 Control Control < Social Social 0.001) Dif- ) ilawy mlf xsigbae.Rte,w n hti the in that find we Rather, biases. existing amplify networks homogeneous always hypoth- politically the will in reject information we social results, net- that experimental esis homogeneous our politically on in Based forma- works. topics belief studying controversial by bias about partisan tion to robust is crowds of propor- dom the controversial advance and to beliefs motivated are accurate views. who different hold individuals to to of tion motivated findings individuals are our of proportion who the motivated of on generally depend robustness would not settings empirical the are people accuracy, Under some toward hold. an that to polarization assumption but group of of the con- position), law wisdom the the network meet for the required might central ditions disrupt controversy by a to motivated hold motivated population likely entire person they be single (unless not A crowds future would settings. for controversy accuracy empirical direction by toward across motivations important vary individual an how can examine emerge, to is to work crowds of dom instead provocative be to motivated accuracy, accurate. are from of people away example, pro- people of for may drive direction if, that contexts the incentives empirical not perverse but some duce not (17) Nonetheless, are sizes change. effect incen- they belief the financial when impact that could even suggesting tives (23), accurate accuracy that more for found compensated has become beliefs can incen- political on financial subjects research without prior formed However, typically tive. but answers, neces- are nonsense was attitudes entering which political from accuracy, subjects for discourage to subjects sary paid We formation belief broadly. political more of dynamics whether the reflects i.e., experiment validity, second our ecological A is sug- replicate. generalizability to (30), about likely concern are samples results representative experimental our nationally gesting typically on MTurk well as polariza- such that replicates initial samples find research nonrepresentative underestimated political on have of research liberal, analyses may skew however, to Generally, sample expected tion. tend our be also would so MTurkers than and weaker population. be general may informa- the results social in our on so weight less and placing tion, search, more relied web have on thus may effectively popu- experiment our general in the Subjects than (29). lation sam- sophisticated digitally representative more are who nationally and subjects younger attracts a (MTurk) Turk not Mechanical that Amazon is ple; is population concern One subject research. our our of generalizability the address group simply of influence law the social by that predicted such polarization. as bias, fail, partisan to correlation existing crowds enhances a cause of can that centrality wisdom group find network the social we on and extremity Appendix) influence belief (SI between networks disproportional simulation could social exert In platforms) in beliefs. to web-based positions extremists or (13); central media allow crowds broadcast obtain of through wisdom to (e.g., the ability undermine the gen- to in and found centralization been Network has belief networks. eral sustain disproportion- and social such generate in of factors can polarization presence structural which individuals), the to central (i.e., polariza- due ately be centralization of may network coexistence learning as The despite social learning. exist and social can tion differ- for Polarization some potential and parties. as the well political polarization as between remains decreased ences error some observe accuracy, remains we increased opinion public Although yet bias, political polarized. partisan partisan mitigate can increase exchange to sufficient own their polarization. on not are h rmr olo hsrsac a ots hte h wis- the whether test to was research this of goal primary The wis- the for necessary appear incentives accuracy Because to important is it study, our of limitations the considering In Information tension: a presents however, finding, This PNAS | a 8 2019 28, May | o.116 vol. | o 22 no. | 10721

POLITICAL SCIENCES networks studied here, information exchange increases belief proportion of GDP). What was tax revenue as a percentage of the economy accuracy and reduces polarization. While the wisdom of crowds in 2010? (Give a number from 0 to 100.) may not hold in all possible empirical settings, our results open the question of when—if ever, and in what circumstances—the Questions for Experiment 2. (i) For every dollar the federal government wisdom of partisan crowds will fail. spent in fiscal year 2016, about how much went to the Department of Defense (US Military)? Answer with a number between 0 and 100. (ii) In Materials and Methods 2007, it was estimated that 6.9 million unauthorized immigrants from Mex- ico lived in the United States. How much did this number change by 2016, Subjects provided informed consent before entering the experimental before President Trump was elected? Enter a positive number if you think interface. Experiment 1 was run on a custom platform and approved by it increased and a negative number if you think it decreased. Express your University of Pennsylvania IRB, Experiment 2 was run on the open source Empirica.ly platform and approved by Northwestern University IRB (31). answer as a percentage of change. (iii) How much did the unemployment Replication data and code are available at the Harvard Dataverse at https:// rate in the United States change from the beginning to the end of Demo- doi.org/10.7910/DVN/OE6UMR (32) and on GitHub at https://github.com/ cratic President Barack Obama’s term in office? Enter a positive number if joshua-a-becker/wisdom-of-partisan-crowds (33). you think it increased and a negative number if you think it decreased. Express your answer as a percentage of change. (iv) About how many US Questions for Experiment 1. (i) In the 2004 election, individuals gave $269.8 soldiers were killed in Iraq between the invasion in 2003 and the withdrawal million to Republican candidate George W. Bush. How much did they give of troops in December 2011? to Democratic candidate John Kerry? (Answer in millions of dollars—e.g., 1 for $1 million.) (ii) According to 2010 estimates, what percentage of people ACKNOWLEDGMENTS. This project was made possible by the 2017 Summer Institute in Computational Social Science (SICSS) at Princeton. in the state of California identify as Black/African-American, Hispanic, or We thank Matt Salganik, Chris Bail, participants at SICSS, Michael X. Asian? (Give a number from 0 to 100.) (iii) What was the US unemployment Delli-Carpini, Steven Klein, Thomas J. Wood, and members of the DiMeNet rate at the end of Barack Obama’s presidential administration—i.e., what research group for helpful comments. We are grateful to Joseph “Mac” percentage of people were unemployed in December 2016? (Give a number Abruzzo for help with recruitment and to Abdullah Almaatouq for support from 0 to 100.) (iv) In 1980, tax revenue was 18.5% of the economy (as a with the Empirica.ly platform.

1. Somin I (2010) Deliberative democracy and political ignorance. Crit Rev 22:253–279. 20. Guilbeault D, Becker J, Centola D (2018) Social learning and partisan bias 2. Pennycook G, Rand DG (2019) Fighting misinformation on social media using crowd- in the interpretation of climate trends. Proc Natl Acad Sci USA 115:9714– sourced judgments of news source quality. Proc Natl Acad Sci USA 116:2521–2526. 9719. 3. Nofer M, Hinz O (2014) Are crowds on the internet wiser than experts? The case of a 21. Kunda Z (1990) The case for motivated reasoning. Psychol Bull 108:480–498. stock prediction community. J Business Econ 84:303–338. 22. Gawronski B (2012) Back to the future of dissonance theory: Cognitive consistency as 4. Wolf M, et al. (2015) Collective intelligence meets medical decision-making: The a core motive. Soc Cogn 30:652–668. collective outperforms the best radiologist. PLoS One 10:e0134269. 23. Wood T, Porter E (2018) The elusive backfire effect: Mass attitudes’ steadfast factual 5. Bartels LM (1996) Uninformed votes: Information effects in presidential elections. Am adherence. Polit Behav 41:135–163. J Polit Sci 40:194–230. 24. Shi F, Teplitskiy M, Duede E, Evans JA (2019) The wisdom of polarized crowds. Nat 6. Galton F (1907) Vox populi (the wisdom of crowds). Nature 75:450–451. Hum Behav 3:329–336. 7. Chambers S (2003) Deliberative democratic theory. Annu Rev Polit Sci 6:307–326. 25. Schkade D, Sunstein CR, Kahneman D (2000) Deliberating about dollars: The severity 8. Moscovici S, Zavalloni M (1969) The group as a polarizer of attitudes. J Pers Soc Psychol shift. Columbia Law Rev 100:1139–1175. 12:125–135. 26. Sunstein CR (2009) Going to Extremes: How like Minds Unite and Divide (Oxford Univ 9. Sunstein CR (2002) The law of group polarization. J Polit Philos 10:175–195. Press, New York). 10. Surowiecki J (2004) The Wisdom of Crowds (Anchor, New York). 27. DeGroot MH (1974) Reaching a consensus. J Am Stat Assoc 69:118–121. 11. Lorenz J, Rauhut H, Schweitzer F, Helbing D (2011) How social influence can 28. Kao AB, et al. (2018) Counteracting estimation bias and social influence to improve undermine the wisdom of crowd effect. Proc Natl Acad Sci USA 108:9020–9025. the wisdom of crowds. J R Soc Interf 15:20180130. 12. Solomon M (2006) Groupthink versus the wisdom of crowds: The social epistemology 29. Munger K, Luca M, Nagler J, Tucker J (2019) Age matters: Sampling strategies for of deliberation and dissent. South J Philos 44:28–42. studying digital media effects. Available at https://osf.io/sq5ub/. Accessed April 23, 13. Becker M, Brackbill D, Centola D (2017) Network dynamics of social influence in the 2019. wisdom of crowds. Proc Natl Acad Sci USA 114:E5070–E5076. 30. Coppock A, Leeper TJ, Mullinix KJ (2018) The generalizability of heterogeneous 14. Jayles B, et al. (2017) How social information can improve estimation accuracy in treatment effect estimates across samples. Proc Natl Acad Sci USA 115:12441– human groups. Proc Natl Acad Sci USA 114:12620–12625. 12446. 15. Madirolas G, de Polavieja GG (2015) Improving collective estimations using resistance 31. Paton N, Almaatouq A (2018) Empirica: Open-source, real-time, synchronous, virtual to social influence. PLoS Comput Biol 11:e1004594. lab framework. Available at https://doi.org/10.5281/zenodo.1488413. Accessed April 16. Bartels LM (2002) Beyond the running tally: Partisan bias in political perceptions. Polit 23, 2019. Behav 24:117–150. 32. Becker J, Porter E (2019) Data from “Replication code for the wisdom of parti- 17. Bullock JG, Gerber AS, Hill SJ, Huber GA (2015) Partisan bias in factual beliefs about san crowds.” Harvard Dataverse. Available at https://doi.org/10.7910/DVN/OE6UMR. politics. Q J Polit Sci 10:519–578. Deposited April 24, 2019. 18. Lodge M, Taber CS (2013) The Rationalizing Voter (Cambridge Univ Press, New York). 33. Becker J, Porter E (2019) Data from “Replication materials for the wisdom 19. Taber CS, Lodge M (2006) Motivated skepticism in the evaluation of political beliefs. of partisan crowds.” GitHub. Available at https://github.com/joshua-a-becker/ Am J Polit Sci 50:755–769. wisdom-of-partisan-crowds. Deposited April 26, 2019.

10722 | www.pnas.org/cgi/doi/10.1073/pnas.1817195116 Becker et al. Downloaded by guest on October 1, 2021