Available online at www.sciencedirect.com ScienceDirect Thinking clearly about causal inferences of politically motivated reasoning: why paradigmatic study designs often undermine causal inference 1 2 1,3 Ben M Tappin , Gordon Pennycook and David G Rand A common inference in behavioral science is that people’s Accordingly, politically motivated reasoning is considered motivation to reach a politically congenial conclusion causally a subset of the phenomenon of ‘directional’ motivated affects their reasoning—known as politically motivated reasoning, in which the person’s goal is to a reach a 4 reasoning. Often these inferences are made on the basis of particular conclusion, political or otherwise [6] . Like data from randomized experiments that use one of two its directional superset, politically motivated reasoning is paradigmatic designs: Outcome Switching, in which identical typically contrasted with a motivation for accuracy when methods are described as reaching politically congenial versus reasoning; thus, to engage in politically motivated rea- uncongenial conclusions; or Party Cues, in which identical soning is to forsake or otherwise diminish the motivation information is described as being endorsed by politically to be accurate. congenial versus uncongenial sources. Here we argue that these designs often undermine causal inferences of politically Research emphasizes various incentives that motivate motivated reasoning because treatment assignment violates people to reach politically congenial conclusions when the excludability assumption. Specifically, assignment to reasoning: From psychological incentives—such as treatment alters variables alongside political motivation that defending their existing political beliefs to avoid cogni- affect reasoning outcomes, rendering the designs confounded. tive dissonance—to material incentives, such as signaling We conclude that distinguishing politically motivated reasoning coalitional allegiance to safeguard their standing in social from these confounds is important both for scientific relationships that confer material benefits [4 ,5,8 ,9,10]. understanding and for developing effective interventions; and Despite these differences in emphasis, however, the core we highlight those designs better placed to causally identify assumption is the same: Politically motivated reasoners politically motivated reasoning. are motivated to reach one political conclusion over another, and this causes reasoning to be performed in Addresses such a way that is (i) different, and (ii) worse for accuracy, 1 Sloan School of Management, Massachusetts Institute of Technology, than if the motivation were absent. United States 2 Hill/Levene School of Business, University of Regina, Canada 3 Department of Brain and Cognitive Sciences, Massachusetts Institute Causal inferences of politically motivated of Technology, United States reasoning Causal inferences of politically motivated reasoning are Corresponding author: Tappin, Ben M ([email protected]) widespread in behavioral science. These inferences are most often made on the basis of data from randomized Current Opinion in Behavioral Sciences 2020, 34:81–87 experiments. The outcome variables used in these This review comes from a themed issue on Emotion, motivation, experiments are diverse, but, generally speaking, include personality and social sciences *political ideologies* either (i) people’s evaluations or endorsements of new Edited by John Jost, Eran Halperin, and Kristin Laurin information—for example, the extent to which they eval- uate new information as high or low quality—or (ii) belief updating—the extent to which the information changes their relevant beliefs. Both of these outcome variables https://doi.org/10.1016/j.cobeha.2020.01.003 plausibly capture processes that could be described as 2352-1546/ã 2020 Elsevier Ltd. All rights reserved. ‘reasoning’. In this paper, however, we focus on experiment designs that use the former type of outcome variable—for two reasons. First, this outcome variable appears to be the more commonly studied. For example, in a recent meta- The concept of politically motivated reasoning is likely analysis, Ditto et al. [11 ] identify 51 experiments span- familiar to most behavioral scientists. While precise defi- ning 40 years of research using this outcome variable. nition of the concept has proved elusive [1], the common Though we lack a precise estimate of the number of working conception is that it is a mode of reasoning in which the person’s goal is to reach a particular, politically 4 congenial conclusion when reasoning [1–3,4 ,5]. But see Ditto [7] for an alternative conception. www.sciencedirect.com Current Opinion in Behavioral Sciences 2020, 34:81–87 82 Emotion, motivation, personality and social sciences *political ideologies* counterpart experiments that use belief updating as the group in which they receive no party endorsement. The outcome, our impression is that it is less common. Second, key result is that subjects are most likely to endorse and perhaps explaining the previous point, the evalua- (oppose) the policy if their party (the opposition party) tions outcome variable is argued to provide clearer evi- endorses it. To illustrate, in one study US subjects were dence of politically motivated reasoning than the belief asked for their opinion about a new welfare policy [14]. updating outcome variable [4 ,8 ,11 ] because of the The randomization of party cue consisted in subjects difficulty in establishing a clear benchmark for what being informed either that the Democratic Party favored politically unmotivated belief updating should look like the policy and Republicans opposed, or that the Repub- [4 ,12]. lican Party favored the policy and Democrats opposed. The key result was that self-identified liberals were more The recent meta-analysis of Ditto et al. [11 ] highlights likely to support the policy in the former treatment, and two paradigmatic study designs that use the evaluations vice versa for self-identified conservatives. outcome variable. We refer to these study designs as Outcome Switching and Party Cues designs. These designs There exists much debate over whether such results are paradigmatic insofar as they appear repeatedly in the provide evidence of politically motivated reasoning, or, research literature [11 ], and, as described above, are instead, show that people use party endorsements as argued to provide some of the clearest evidence of politi- ‘cognitive shortcuts’ to help them form appropriate policy cally motivated reasoning [4 ,8 ]. We describe the typical opinions [1,10,15–20]. However, a number of recent structure of these designs below. studies suggest that politically motivated reasoning is the more plausible of these two causal mechanisms, on Outcome switching the basis of several kinds of results. In particular, party In this design, subjects are randomly assigned to receive cue effects are larger among those who know about and one of two pieces of information; where the substantive engage with politics [20] (but see e.g. [19,21]); among detail of the information is held constant across condi- those with a combination of strong party attachment and tions, but its implication for subjects’ political identities high cognitive ability [15]; and in contexts where elite or preferences is varied between conditions. The key result partisan polarization is more salient [16,17,20]. These is that subjects’ evaluation of the information differs by results appear difficult to reconcile with a cognitive condition, and, in particular, that this difference is corre- shortcut mechanism. lated with their political identities or preferences. Spe- cifically, people evaluate the information less favorably Why paradigmatic designs undermine causal when it is discordant with their political identities or inference preferences than when it is concordant with their political In this section, we outline why the paradigmatic study identities or preferences. designs often undermine causal inferences of politically motivated reasoning. In particular, we organize various To illustrate, in one study U.S. subjects were asked to arguments made elsewhere about why these designs are evaluate the validity of a test of ‘open-minded and often confounded, and we trace these arguments with reflective’ thinking [13]. Before providing their evalua- respect to a single analytic framework. tions, subjects completed the test themselves and were randomly assigned to one of two treatments (or control) in The treatment in the paradigmatic designs consists in which they were provided information about the test. In randomly assigning subjects to receive different informa- treatment A, subjects were told that people who believe tion, as per the aforementioned examples, and recording that climate change is happening tend to score higher on reasoning outcomes in the form of their beliefs, attitudes, the test than people who are skeptical that climate change or opinions. Accordingly, causal inferences of politically is happening; implying the former are more open- motivated reasoning that are often made on the basis of minded. In treatment B, they were told the reverse: that these designs assume that the information treatment people who are skeptical that climate change is happen- affects people’s reasoning only insofar as it affects their ing tend to score higher in the test, implying they are more political motivation. That is, the effect of the information open-minded. Subjects who identified on the