American Political Science Review Vol. 104, No. 2 May 2010 doi:10.1017/S0003055410000092 Are Survey Experiments Externally Valid? JASON BARABAS and JENNIFER JERIT Florida State University esearchers use survey experiments to establish causal effects in descriptively representative samples, but concerns remain regarding the strength of the stimuli and the lack of realism in R experimental settings. We explore these issues by comparing three national survey experiments on Medicare and immigration with contemporaneous natural experiments on the same topics. The survey experiments reveal that providing information increases political knowledge and alters attitudes. In contrast, two real-world government announcements had no discernable effects, except among people who were exposed to the same facts publicized in the mass media. Even among this exposed subsample, treatment effects were smaller and sometimes pointed in the opposite direction. Methodologically, our results suggest the need for caution when extrapolating from survey experiments. Substantively, we find that many citizens are able to recall factual information appearing in the news but may not adjust their beliefs and opinions in response to this information. ocial science researchers seek to establish causal comparing the effects of two naturally occurring po- relationships that are generalizable—that is, they litical events to contemporaneous survey experiments Stry to maximize internal and external validity. Sur- that delivered similar information to diverse samples vey experiments are becoming more popular among of the American public. In the first study, we examine scholars because they seem to possess both proper- the effects of news coverage of the 2007 Medicare trust ties. The random assignment of respondents to treat- fund warning and compare it to a survey experiment ment and control conditions reveals whether one factor that provided the same key facts about Medicare’s fi- causes another, whereas the use of a representative nancial status. In the second study, we analyze media sample allows generalization to the larger population. coverage of a new citizenship test launched by the U.S. However, even in nationally representative survey Citizenship and Immigration Services (USCIS) in the experiments, external validity may still be a concern if fall of 2008. Once again, we conducted a concurrent the treatments do not resemble the relevant phenom- survey experiment that delivered similar information ena in question or if the experimental setting exagger- to an adult national sample. Taken together, the two ates the effect of the stimulus. This article investigates studies offer a rare opportunity to compare the treat- a question that recent studies have raised (e.g., Gaines, ment effects from survey experiments with the effects Kuklinski, and Quirk 2007; Kinder 2007) but that has of real-world political events. not been examined empirically: are the causal findings This research is timely, if only because survey ex- of survey experiments reliable predictors of how opin- periments have overturned much of the conventional ion changes in the wake of actual political events? wisdom on the nature of public opinion (e.g., Gibson For researchers using survey experiments, the im- 1998; Kinder and Sanders 1996; Krosnick and Schu- plicit assumption is that a significant treatment effect man 1988; Prior and Lupia 2008; Sniderman and Pi- says something about the direction, if not the rough azza 1993; Zaller and Feldman 1992). Indeed, one magnitude, of effects that might be expected to occur scholar recently proclaimed, “survey experiments that in the real world (Gaines, Kuklinski, and Quirk 2007, integrate representative samples with the experimental 5). Our study investigates this assumption. We do so by control of questions represent the most valuable tool for gaining access to the processes that underlie opin- ion formation” (Lavine 2002, 242). With the expansion Jason Barabas is Associate Professor, Department of Political Sci- of Internet surveys and multi-investigator studies such ence, Florida State University, 531 Bellamy Building, Tallahassee, FL 32306 ([email protected]). as Time-sharing Experiments in the Social Sciences, Jennifer Jerit is Associate Professor, Department of Political Sci- researchers from a variety of subfields are now using ence, Florida State University, 531 Bellamy Building, Tallahassee, survey experiments (Druckman et al. 2006). Thus, the FL 32306 ([email protected]). number of scholars affected by this issue is large and The authors appreciate the helpful comments they received from growing.1 Scott Allard, Charles Barrilleaux, Bill Berry, John Bullock, Brian Gaines, Cindy Kam, Gary King, Jon Krosnick, Jim Kuklinski, Skip We draw two lessons from our study—one is pri- Lupia, Cherie Maestas, Rose McDermott, Betsy Sinclair, Mark marily methodological, the other is more substantive. Souva, John Transue, Mathieu Turgeon, Gabor Toka, Paul Quirk, Methodologically, we show that survey experiments Will Shadish, Joe Young, and participants in colloquia at Florida generate effects that are observable among particu- State University, the University of North Carolina at Chapel Hill, lar subgroups, not necessarily the entire population. Texas A&M University, and Vanderbilt University. Earlier ver- sions of this article were presented at the Visions in Methodol- Insofar as researchers keep this point in mind, we be- ogy Conference at Ohio State University, and at annual meetings lieve that survey experiments can be a valuable tool of the American Political Science Association, the Canadian Po- litical Science Association, and the International Society of Politi- cal Psychology. Ben Gaskins provided valuable research assistance. 1 Indeed, scholars in other fields are engaged in an analogous line of Data, replication code, and an online Appendix are available at inquiry (e.g., Benz and Meier 2008; Cook, Shadish, and Wong 2008; http://polisci.fsu.edu/people/faculty/index.htm. Levitt and List 2007). 226 American Political Science Review Vol. 104, No. 2 for studying public opinion. Substantively, we find that findings may be “the product of an unrealistically pow- people in the natural experiments do not integrate new erful manipulation, one that rarely occurs in natural information and adjust their political beliefs to the de- settings.” For example, in a critique of the framing lit- gree that they do in survey experiments. Thus, scholars erature, Sniderman and Theriault (2004) argue that it might come to different conclusions about the nature is unrealistic to examine the effects of a single frame of public opinion depending on the manner in which when citizens typically experience competing frames they study it. (also see Chong and Druckman 2007). To the extent that treatments in survey experiments are overly strong or atypical, the observed effects may not generalize VARIETIES OF EXPERIMENTS beyond the particular study at hand. AND VALIDITY A second and related issue is the manner in which Years ago, scholars rarely undertook social science treatments are received (i.e., the experimental setting). experiments (Kinder and Palfrey 1993). Today, exper- Kinder (2007, 157) is concerned that imental research is conducted in many different ways (McDermott 2002): laboratory experiments, carried experimental results can always be questioned on their out in controlled settings with students or members generalizability, and framing effects are no exception. The of the local community as subjects (e.g., Grosser and major worry in this respect is that framing experiments— Schram 2006; Iyengar and Kinder 1987; Kam, Wilking, like experiments in mass communication generally— typically obliterate the distinction between the supply of and Zechmeister 2007; Mutz and Reeves 2005); information, on the one hand, and its consumption, on field experiments, in which randomized treatments are the other. That is, experiments are normally carried out in delivered outside the lab (e.g., Arceneaux and Kolodny such a way that virtually everyone receives the message. 2009; Gerber and Green 2000); and natural experi- The typical experiment thereby avoids a major obstacle ments, where analysts take advantage of variation in standing in the way of communication effects, namely, an real-world phenomena (e.g., Huber and Arceneaux inattentive audience, lost in the affairs of private life. By 2007; Lassen 2005; Mondak 1995).2 Increasingly, public ensuring that frames reach their intended audiences, ex- opinion researchers employ survey experiments that periments may exaggerate their power. deliver randomized treatments in telephone or Inter- net polls. This approach is common in the area of citizen Likewise, in their framework, Gaines, Kuklinski, and competence, where researchers use survey experi- Quirk (2007, 16) introduce an “inflation parameter” in ments to examine whether people are capable of learn- recognition of the fact that “the artificially clean envi- ing (e.g., Kuklinski et al. 2000) and to analyze how the ronment of the survey question makes treatment easier presentation of information alters beliefs or opinions to receive than in real life.” Unlike the natural world, (e.g., Berinsky 2007; Kuklinski et al. 2001; Lupia et al. which contains competing messages and other distrac- n.d.; Sniderman and Theriault 2004; Turgeon 2009). tions that make exposure to the treatment probabilistic, Because of their randomized treatments, experi- exposure is essentially forced in a survey experiment.