Cambridge Handbook of Experimental Political Science

Laboratory experiments, survey experiments, and field experiments occupy a central and growing place in the discipline of political science. The Cambridge Handbook of Experimental Political Science is the first text to provide a comprehensive overview of how experimental research is transforming the field. Some chapters explain and define core concepts in experimental design and analysis. Other chapters provide an intellectual history of the experimental movement. Throughout the book, leading scholars review groundbreaking research and explain, in personal terms, the growing influence of experimental political science. The Cambridge Handbook of Experimental Political Science provides a collection of insights that can be found nowhere else. Its topics are of interest not just to researchers who are conducting experiments today, but also to researchers who believe that experiments can help them make new and important discoveries in political science and beyond.

James N. Druckman is Payson S. Wild Professor of Political Science at Northwestern Univer- sity. He has published articles in journals such as the American Political Science Review, American Journal of Political Science, and Journal of Politics. He is currently the editor of Public Opinion Quarterly. Professor Druckman’s research focuses on political preference formation and communication, and his recent work examines how citizens make political, economic, and social decisions in various contexts.

Donald P. Green is A. Whitney Griswold Professor of Political Science at Yale University. He is the author of four books and several dozen articles on a wide array of topics, including partisanship, campaign ﬁnance, voting, and prejudice. Since 1998, his work has focused on the design, implementation, and analysis of ﬁeld experiments.

James H. Kuklinski is Matthew T. McClure Professor of Political Science at the University of Illinois at Urbana–Champaign. His interests include citizen decision making and the formulation of experimental designs that generate more accurate inferences about the external world than the traditional random assignment experiment.

Arthur Lupia is Hal R. Varian Professor of Political Science at the University of Michigan. He studies politics with a focus on how people make decisions when they lack information. Professor Lupia cofounded TESS (Time-Sharing Experiments for the Social Sciences) and served as Principal Investigator of the American National Election Studies.

Cambridge Handbook of Experimental Political Science

Edited by JAMES N. DRUCKMAN Northwestern University

DONALD P. GREEN Yale University

JAMES H. KUKLINSKI University of Illinois at Urbana–Champaign

ARTHUR LUPIA University of Michigan cambridge university press Cambridge, New York, Melbourne, Madrid, Cape Town, Singapore, Sao˜ Paulo, Delhi, Tokyo, Mexico City

Cambridge University Press 32 Avenue of the Americas, New York, ny 10013-2473, usa www.cambridge.org Information on this title: www.cambridge.org/9780521174558

C Cambridge University Press 2011

This publication is in copyright. Subject to statutory exception and to the provisions of relevant collective licensing agreements, no reproduction of any part may take place without the written permission of Cambridge University Press.

First published 2011

Printed in the United States of America

A catalog record for this publication is available from the British Library.

Library of Congress Cataloging in Publication data

Cambridge Handbook of Experimental Political Science / [edited by] James N. Druckman, Donald P. Green, James H. Kuklinski, Arthur Lupia. p. cm. Includes bibliographical references and index. isbn 978-0-521-19212-5 (hardback) – isbn 978-0-521-17455-8 (paperback) 1. Political science – Methodology. 2. Political science – Research. 3. Political science – Experiments. I. Druckman, James N., 1971– II. Title. ja71.c325 2011 320.072 –dc22 2010044869 isbn 978-0-521-19212-5 Hardback isbn 978-0-521-17455-8 Paperback

Cambridge University Press has no responsibility for the persistence or accuracy of urls for external or third-party Internet Web sites referred to in this publication and does not guarantee that any content on such Web sites is, or will remain, accurate or appropriate. Contents

List of Tables page viii List of Figures ix Contributors xi Acknowledgments xiii

INTRODUCTION

1 Experimentation in Political Science 3 James N. Druckman, Donald P. Green, James H. Kuklinski, and Arthur Lupia

PART I: DESIGNING EXPERIMENTS

2 Experiments: An Introduction to Core Concepts 15 James N. Druckman, Donald P. Green, James H. Kuklinski, and Arthur Lupia 3 Internal and External Validity 27 Rose McDermott 4 Students as Experimental Participants: A Defense of the “Narrow Data Base” 41 James N. Druckman and Cindy D. Kam 5 Economics versus Psychology Experiments: Stylization, Incentives, and Deception 58 Eric S. Dickson

PART II: THE DEVELOPMENT OF EXPERIMENTS IN POLITICAL SCIENCE

6 Laboratory Experiments in Political Science 73 Shanto Iyengar

v vi Contents

7 Experiments and Game Theory’s Value to Political Science 89 John H. Aldrich and Arthur Lupia 8 The Logic and Design of the Survey Experiment: An Autobiography of a Methodological Innovation 102 Paul M. Sniderman 9 Field Experiments in Political Science 115 Alan S. Gerber

PART III: DECISION MAKING

10 Attitude Change Experiments in Political Science 141 Allyson L. Holbrook 11 Conscious and Unconscious Information Processing with Implications for Experimental Political Science 155 Milton Lodge, Charles Taber, and Brad Verhulst 12 Political Knowledge 171 Cheryl Boudreau and Arthur Lupia

PART IV: VOTE CHOICE, CANDIDATE EVALUATIONS, AND TURNOUT

13 Candidate Impressions and Evaluations 187 Kathleen M. McGraw 14 Media and Politics 201 Thomas E. Nelson, Sarah M. Bryner, and Dustin M. Carnahan 15 Candidate Advertisements 214 Shana Kushner Gadarian and Richard R. Lau 16 Voter Mobilization 228 Melissa R. Michelson and David W. Nickerson

PART V: INTERPERSONAL RELATIONS

17 Trust and Social Exchange 243 Rick K. Wilson and Catherine C. Eckel 18 An Experimental Approach to Citizen Deliberation 258 Christopher F. Karpowitz and Tali Mendelberg 19 Social Networks and Political Context 273 David W. Nickerson

PART VI: IDENTITY, ETHNICITY, AND POLITICS

20 Candidate Gender and Experimental Political Science 289 Kathleen Dolan and Kira Sanbonmatsu 21 Racial Identity and Experimental Methodology 299 Darren Davis Contents vii

22 The Determinants and Political Consequences of Prejudice 306 Vincent L. Hutchings and Spencer Piston 23 Politics from the Perspective of Minority Populations 320 Dennis Chong and Jane Junn

PART VII: INSTITUTIONS AND BEHAVIOR

24 Experimental Contributions to Collective Action Theory 339 Eric Coleman and Elinor Ostrom 25 Legislative Voting and Cycling 353 Gary Miller 26 Electoral Systems and Strategic Voting (Laboratory Election Experiments) 369 Rebecca B. Morton and Kenneth C. Williams 27 Experimental Research on Democracy and Development 384 Ana L. De La O and Leonard Wantchekon

PART VIII: ELITE BARGAINING

28 Coalition Experiments 399 Daniel Diermeier 29 Negotiation and Mediation 413 Daniel Druckman 30 The Experiment and Foreign Policy Decision Making 430 Margaret G. Hermann and Binnur Ozkececi-Taner

PART IX: ADVANCED EXPERIMENTAL METHODS

31 Treatment Effects 445 Brian J. Gaines and James H. Kuklinski 32 Making Effects Manifest in Randomized Experiments 459 Jake Bowers 33 Design and Analysis of Experiments in Multilevel Populations 481 Betsy Sinclair 34 Analyzing the Downstream Effects of Randomized Experiments 494 Rachel Milstein Sondheimer 35 Mediation Analysis Is Harder than It Looks 508 John G. Bullock and Shang E. Ha

AFTERWORD

36 Campbell’s Ghost 525 Donald R. Kinder

Name Index 531 Subject Index 548 List of Tables

4.1 Comparison of Students versus Nonstudent General Population page 52 9.1 Approximate Cost of Adding One Vote to Candidate Vote Margin 118 9.2 Voter Mobilization Experiments Prior to 1998 New Haven Experiment 119 11.1 Schematic of Racial Implicit Association Test Using Pleasant and Unpleasant Words and European American and African American Stereotype Words 162 15.1 Methodological Consequences of Differences between Observational and Experimental Studies of Candidate Advertisements 217 25.1 Testing the Uncovered Set with Previous Majority Rule Experiments 361 26.1 Payoff Schedule 375 26.2 Payoff Schedule 377 28.1 Potential Coalitions and Their Respective Payoffs 406 32.1 One-by-One Balance Tests for Covariates Adjusted for Blocking in the Blocked Thirty-Two-City Study 466 32.2 One-by-One Balance Tests for Covariates in the Blocked Eight-City Study 467 32.3 One-by-One Balance Tests Adjusted for Covariates, by Post-Stratiﬁcation in the Blocked Thirty-Two-City Study 471 33.1 Intent-to-Treat Effects 489 34.1 Classiﬁcation of Target Population in Downstream Analysis of Educational Intervention 499

viii List of Figures

1.1 Experimental Articles in the American Political Science Review page 5 4.1 Sampling Distribution of bT, Single Treatment Effect 47 4.2 Sampling Distribution of bT, Heterogeneous Treatment Effects 49 4.3 Sampling Distributions of bT and bTZ, Heterogeneous Treatment Effects 49 6.1 Race of Suspect Manipulation 79 6.2 Facial Similarity Manipulation 80 9.1 Graphical Representation of Treatment Effects with Noncompliance 127 10.1 Pre-Test–Post-Test Control Group Design 147 10.2 Pre-Test–Post-Test Multiple Experimental Condition Design 147 10.3 Post-Test–Only Control Group Design 148 10.4 Post-Test–Only Multiple Experimental Group Design 148 11.1 Spreading Activation in a Sequential Priming Paradigm for Short and Long Stimulus Onset Asynchrony 160 21.1 Example of Experimental Design for Racial Identity 304 24.1 A Prisoner’s Dilemma Game 340 24.2 Contributions in a Public Goods (PGs) Game 343 24.3 A Common Pool Resource (CPR) Game 347 25.1 Outcomes of Majority Rule Experiments without a Core 355 25.2 Majority Rule with Issue-by-Issue Voting 356 25.3 Effect of Backward and Forward Agendas 357 25.4 Effect of Monopoly Agenda Setting 359 25.5 Sample Majority Rule Trajectory for Configuration 1 362 25.6 (a) Uncovered Set and Outcomes for Configuration 1. (b) Uncovered Set and Outcomes for Configuration 2 363 25.7 Senatorial Ideal Points and Proposed Amendments for Civil Rights Act of 1964 365 26.1 Median Voter Theorem 370 32.1 Efficiency of Paired and Unpaired Designs in Simulated Turnout Data 464

ix x List of Figures

32.2 Graphical Assessment of Balance on Distributions of Baseline Turnout for the Thirty-Two-City Experiment Data 466 32.3 Poststratication Adjusted Conﬁdence Intervals for the Difference in Turnout between Treated and Control Cities in the Thirty-Two-City Turnout Experiment 470 32.4 Covariance Adjustment in a Simple Random Experiment 472 32.5 Covariance Adjustment in a Blocked Random Experiment 473 32.6 Covariance Adjusted Conﬁdence Intervals for the Difference in Turnout between Treated and Control Cities in the Thirty-Two-City Turnout Experiment Data 476 33.1 Multilevel Experiment Design 488 36.1 Number of Articles Featuring Experiments Published in American Political Science Review, 1906–2009 526 Contributors

John H. Aldrich Ana L. De La O Duke University Yale University

Cheryl Boudreau Eric S. Dickson University of California, Davis New York University

Jake Bowers Daniel Diermeier University of Illinois at Urbana–Champaign Northwestern University

Sarah M. Bryner Kathleen Dolan The Ohio State University University of Wisconsin, Milwaukee

John G. Bullock Daniel Druckman Yale University George Mason University

Dustin M. Carnahan James N. Druckman The Ohio State University Northwestern University

Dennis Chong Catherine C. Eckel Northwestern University University of Texas at Dallas

Eric Coleman Shana Kushner Gadarian Florida State University Syracuse University

Darren Davis Brian J. Gaines University of Notre Dame University of Illinois at Urbana–Champaign

xi xii Contributors

Alan S. Gerber Tali Mendelberg Yale University Princeton University

Donald P. Green Melissa R. Michelson Yale University Menlo College

Shang E. Ha Gary Miller Brooklyn College, City University of New Washington University in St. Louis York Rebecca B. Morton Margaret G. Hermann New York University Syracuse University Thomas E. Nelson Allyson L. Holbrook The Ohio State University University of Illinois at Chicago David W. Nickerson University of Notre Dame Vincent L. Hutchings University of Michigan Elinor Ostrom Indiana University Shanto Iyengar Stanford University Binnur Ozkececi-Taner Hamline University Jane Junn University of Southern California Spencer Piston University of Michigan Cindy D. Kam Kira Sanbonmatsu Vanderbilt University Rutgers University Christopher F. Karpowitz Betsy Sinclair Brigham Young University The University of Chicago Donald R. Kinder Paul M. Sniderman University of Michigan Stanford University James H. Kuklinski Rachel Milstein Sondheimer University of Illinois at Urbana–Champaign United States Military Academy Richard R. Lau Charles Taber Rutgers University Stony Brook University Milton Lodge Brad Verhulst Stony Brook University Stony Brook University Arthur Lupia Leonard Wantchekon University of Michigan New York University

Rose McDermott Kenneth C. Williams Brown University Michigan State University

Kathleen M. McGraw Rick K. Wilson The Ohio State University Rice University Acknowledgments

This volume has its origins in the Ameri- Our ﬁrst task in developing the Handbook can Political Science Review’s special 2006 cen- was to generate a list of topics and possible tennial issue celebrating the evolution of the authors; we were overwhelmed by the posi- study of politics. For that issue, we proposed tive responses to our invitations to contribute. a paper that traced the history of experiments Although we leave it to the reader to assess within political science. The journal’s editor, the value of the book, we can say that the Lee Sigelman, responded to our proposal for experience of assembling this volume could the issue with a mix of skepticism – for exam- not have been more enjoyable and instruc- ple, asking about the prominence of experi- tive, thanks to the authors. ments in the discipline – and encouragement. Nearly all of the authors attended a con- We moved forward and eventually published ference held at Northwestern University (in an article in the special issue, and there is no Evanston, IL, USA) on May 28 and 29, doubt that it was much better than it would 2009. We were extremely fortunate to have have been absent Lee’s constant construc- an exceptionally able group of discussants tive guidance. Indeed, Lee, who himself con- take the lead in presenting and comment- ducted some remarkably innovative experi- ing on the chapters; we deeply appreciate ments, pushed us to think about what makes the time and insights they provided. The political science experiments unique relative discussants included Kevin Arceneaux, Ted to the other psychological and social sciences. Brader, Ray Duch, Kevin Esterling, Diana It was this type of prodding that led us to con- Mutz, Mike Neblo, Eric Oliver, Randy ceive of the Cambridge Handbook of Experi- Stevenson, Nick Valentino, and Lynn mental Political Science. Sadly, Lee did not live Vavreck. Don Kinder played a special role to see the completion of the Handbook, but we at the conference, offering his overall assess- hope it approaches the high standards that he ment at the end of the proceedings. A ver- always set. We know we are not alone in say- sion of these thoughts appears as the volume’s ing that he is greatly missed. Afterword.

xiii xiv Acknowledgments

We also owe thanks to the more than thirty Of course, the conference and the pro- graduate students who attended the confer- duction of the volume would not have been ence, met with faculty, and offered their per- possible without generous ﬁnancial support, spectives. These students (many of whom and we gratefully acknowledge the National became professors before the publication of Science Foundation (SES-0851285), North- the volume) included Lene Aarøe, Emily western University’s Weinberg College of Alvarez, Christy Aroopala, Bernd Beber, Arts and Sciences, and the Institute for Policy Toby Bolsen, Kim Dionne, Katie Donovan, Research. Ryan Enos, Brian Falb, Mark Fredrickson, Following the conference, authors en- Fernando Garcia, Ben Gaskins, Seth Gold- gaged in substantial revisions and, along the man, Daniel Hidalgo, Samara Klar, Yanna way, a number of others provided instruc- Krupnikov, Thomas Leeper, Adam Levine, tive comments – including Cengiz Erisen, Peter Loewen, Kristin Michelitch, Daniel Jeff Guse, David Llanos, and the anonymous Myers, Jennifer Ogg Anderson, Spencer Pis- Press reviewers. We also thank the partic- ton, Josh Robison, Jon Rogowski, Mark ipants in Druckman’s graduate experimental Schneider, Geoff Sheagley, Alex Theodor- class who read a draft of the volume and com- idis, Catarina Thomson, Dustin Tingley, mented on each chapter; these impressive stu- Brad Verhulst, and Abby Wood. dents included Emily Alvarez, Toby Bolsen, We thank a number of others who atten- Brian Falb, Samara Klar, Thomas Leeper, ded the conference and offered important Rachel Moskowitz, Taryn Nelson, Christoph comments, including, but not limited to, Nguyen, Josh Robison, and Xin Sun. We have David Austen-Smith, Traci Burch, Fay Cook, no doubt that countless others offered advice Jeremy Freese, Jerry Goldman, Peter Miller, (of which we, as the editors, are not directly Eugenia Mitchelstein, Ben Page, Jenn Riche- aware), and we thank them for their contri- son, Anne Sartori, Victor Shih, and Salvador butions. A special acknowledgment is due to Vazquez del Meracdo. Samara Klar and Thomas Leeper, who have The conference would not have been pos- probably read the chapters more than any- sible without the exceptional contributions of one else and, without fail, have offered help- a number of individuals. Of particular note ful advice and skillful coordination. Finally, are the many staff members of Northwest- it was a pleasure working with Eric Crahan ern’s Institute for Policy Research. We thank and Jason Przybylski at Cambridge Univer- the institute’s director, Fay Cook, for sup- sity Press. porting the conference, and we are indebted We view this Handbook as a testament to to Patricia Reese for overseeing countless the work of many scholars (a number of whom logistics. We also thank Eric Betzold, Arlene are authors in this volume) who set the stage Dattels, Sarah Levy, Michael Weis, and Bev for experimental approaches in political sci- Zack. A number of Northwestern’s politi- ence. Although we cannot be sure what many cal science Ph.D. students also donated their of them will think of the volume, we do hope time to ensure a successful event – includ- that it successfully addresses a question raised ing Emily Alvarez, Toby Bolsen, Brian Falb, by an editor’s (Druckman’s) son who was Samara Klar, Thomas Leeper, and Josh Robi- seven when he asked, “Why is political ‘sci- son. We also thank Nicole, Jake, and Sam ence’ a ‘science’ since it doesn’t do things that Druckman for their patience and help in science does, like run experiments?” ensuring everything at the conference was in place. – James N. Druckman, Donald P. Green, James H. Kuklinski, and Arthur Lupia INTRODUCTION

CHAPTER 1

Experimentation in Political Science

James N. Druckman, Donald P. Green, James H. Kuklinski, and Arthur Lupia

In his 1909 American Political Science Asso- participants) to treatment and control groups. ciation presidential address, A. Lawrence Experiments also guide theoretical develop- Lowell (1910) advised the fledgling discipline ment by providing a means for pinpoint- against following the model of the natural ing the effects of institutional rules, pref- sciences: “We are limited by the impossi- erence configurations, and other contextual bility of experiment. Politics is an observa- factors that might be difficult to assess using tional, not an experimental science . . . ” (7). other forms of inference. Most of all, exper- The lopsided ratio of observational to experiments guide theory by providing stubborn imental studies in political science, over the facts – that is, reliable information about one hundred years since Lowell’s statement, cause and effect that inspires and constrains arguably affirms his assessment. The next theory. hundred years are likely to be different. The Experiments bring new opportunities for number and influence of experimental studies inference along with new methodological are growing rapidly as political scientists dis- challenges. The goal of the Cambridge Hand- cover ways of using experimental techniques book of Experimental Political Science is to help to illuminate political phenomena. scholars more effectively pursue experimen- The growing interest in experimentation tal opportunities while better understand- reflects the increasing value that the disci- ing the challenges. To accomplish this goal, pline places on causal inference and empir- the Handbook offers a review of basic defi- ically guided theoretical refinement. Experi- nitions and concepts, compares experiments ments facilitate causal inference through the with other forms of inference in political sci- transparency and content of their procedures, ence, reviews the contributions of experimen- most notably the random assignment of ob- tal research, and presents important method- servations (a.k.a. subjects or experimental ological issues. It is our hope that discussing these topics in a single volume will help facilitate the growth and development of experi- Parts of this chapter come from Druckman et al. (2006). mentation in political science.

3 4 James N. Druckman, Donald P. Green, James H. Kuklinski, and Arthur Lupia

1. The Evolution and Influence of lution began (e.g., Mahoney and Druckman Experiments in Political Science 1975; Guetzkow and Valadez 1981), and, later, a periodic but now extinct journal, The Social scientists answer questions about Experimental Study of Politics, began publica- social phenomena by constructing theories, tion (also see Brody and Brownstein 1975). deriving hypotheses, and evaluating these These examples are best seen as excep- hypotheses by empirical or conceptual means. tions, however. For much of the discipline’s One way to evaluate hypotheses is to inter- history, experiments remained on the periph- vene deliberately in the social process under ery. In his widely cited methodological paper investigation. An important class of inter- from 1971, Lijphart (1971) states, “The ventions is experiments. An experiment is a experimental method is the most nearly ideal deliberate test of a causal proposition, typi- method for scientific explanation, but unfor- cally with random assignment to conditions.1 tunately it can only rarely be used in polit- Investigators design experiments to evaluate ical science because of practical and ethical the causal impacts of potentially informative impediments” (684). In their oft-used meth- explanatory variables. ods text, King, Keohane, and Verba (1994) Although scientists have conducted exper- provide virtually no discussion of experimen- iments for hundreds of years, modern experi- tation, stating only that experiments are help- mentation made its debut in the 1920s and ful insofar as they “provide a useful model 1930s. It was then that, for the first time, for understanding certain aspects of non- social scientists began to use random assign- experimental design” (125). ment in order to allocate subjects to control A major change in the status of experi- and treatment groups.2 One can find exam- ments in political science occurred during the ples of experiments in political science as early last decades of the twentieth century. Evi- as the 1940s and 1950s. The first experi- dence of the change is visible in Figure 1.1. mental paper in the American Political Science Figure 1.1 comes from a content analysis of Review (APSR) appeared in 1956 (Eldersveld the discipline’s widely regarded flagship jour- 1956).3 In that study, the author randomly nal, the APSR, and shows a sharp increase, assigned potential voters to a control group in recent years, in the number of articles that received no messages or to treatment using a random assignment experiment. In groups that received messages encouraging fact, more than half of the 71 experimen- them to vote via personal contact (which tal articles that appeared in the APSR during included phone calls or personal visits) or via its first 103 years were published after 1992. a mailing. The study showed that more vot- Other signs of the rise of experiments include ers in the personal contact treatment groups the many graduate programs now offering turned out to vote than those in either the courses on experimentation, National Sci- control group or the mailing group; that is, ence Foundation support for experimental personal contact caused a relative increase in infrastructure, and the proliferation of sur- turnout. A short time after Eldersveld’s study, vey experiments in both private and publicly 4 an active research program using experi- supported studies. Experimental approaches ments to study international conflict reso- 4 The number of experiments has not only grown, but experiments appear to be particularly influential in 1 This definition implicitly excludes so-called natural shaping research agendas. Druckman et al. (2006) experiments, where nature initiates a random pro- compared the citation rates for experimental articles cess. We discuss natural experiments in the next published in the APSR (through 2005)withtherates chapter. for 1) a random sample of approximately six nonex- 2 Brown and Melamed (1990) explain that “[r]ando- perimental articles in every APSR volume where at mization procedures mark the dividing line between least one experimental article appeared, 2)thatsame classical and modern experimentation and are of great random sample narrowed to include only quantitative practical benefit to the experimenter” (3). articles, and 3) the same sample narrowed to two arti- 3 Gosnell’s (1926) well-known voter mobilization field cles on the same substantive topic that appeared in the study was not strictly an experiment because it did same year as the experimental article or in the year not employ random assignment. before it appeared. They report that experimental Experimentation in Political Science 5

18 16 14 12 10 8 6 4 Number of articles 2 0

Years Figure 1.1. Experimental Articles in the American Political Science Review have not been confined to single subfields or imental studies that attempt to estimate the approaches. Instead, political scientists have magnitudes of causal parameters, such as the employed experiments across fields and have influence of racial attitudes on policy prefer- drawn on and developed a notable range ences (Gilens 1996) or the price elasticity of of experimental methods. These sources of demand for public and private goods (Green diversity make a unifying Handbook particu- 1992). larly appealing for the purpose of facilitating A second role entails “speaking to theo- coordination and communication across var- rists,” where the goal is “to test the predic- ied projects. tions [or the assumptions] of well articulated formal theories [or other types of theories]. . . . Such experiments are intended to 2. Diversity of Applications feed back into the theoretical literature – i.e., they are part of a dialogue between experi- Political scientists have implemented exper- menters and theorists” (Roth 1995, 22). The iments for various purposes to address a many political science experiments that assess multitude of issues. Roth (1995) identifies the validity of claims made by formal mod- three nonexclusive roles that experiments elers epitomize this type of correspondence can play, and a cursory review makes clear (e.g., Ostrom, Walker, and Gardner 1992; that political scientists employ them in all Morton 1993;Frechette,´ Kagel, and Lehrer three ways. First, Roth describes “search- 2003).5 The third usage is “whispering in ing for facts,” where the goal is to “iso- the ears of princes,” which facilitates “the late the cause of some observed regularity, dialogue between experimenters and policy- by varying details of the way the experi- makers. ...[The]experimentalenvironment ments were conducted. Such experiments are is designed to resemble closely, in certain part of the dialogue that experimenters carry respects, the naturally occurring environment on with one another” (22). These types of that is the focus of interest for the policy pur- experiments often complement observational poses at hand” (Roth 1995, 22). Cover and research (e.g., work not employing random Brumberg’s (1982) field experiment examin- assignment) by arbitrating between conflicting the effects of mail from members of the ing results derived from observational data. U.S. Congress on their constituents’ opinions “Searching for facts” describes many exper- 5 The theories need not be formal; for example, Lodge and his colleagues have implemented a series of articles are cited significantly more often than each experiments to test psychological theories of infor- of the comparison groups of articles (e.g., 47%, 74%, mation processing (e.g., Lodge et al. 1989; Lodge, and 26% more often, respectively). Steenbergen, and Brau 1995). 6 James N. Druckman, Donald P. Green, James H. Kuklinski, and Arthur Lupia exemplifies an experiment that whispers in unaware of this range of activity, which limits the ears of legislative “princes.” the extent to which experimental political sci- Although political scientists might share entists have learned from one another. For rationales for experimentation with other example, scholars studying coalition forma- scientists, their attention to focal aspects tion and international negotiations experi- of politically relevant contexts distinguishes mentally can benefit from talking to one their efforts. This distinction parallels the use another, yet there is little sign of engage- of other modes of inference by political scien- ment between the respective contributors to tists. As Druckman and Lupia (2006) argue, these literatures. Similarly, there are few signs “[c]ontext, not methodology, is what unites of collaboration among experimental scholars ourdiscipline....Politicalscienceisunited who study different kinds of decision making by the desire to understand, explain, and pre- (e.g., foreign policy decision making and vot- dict important aspects of contexts where indi- ing decisions). Of equal importance, schol- vidual and collective actions are intimately ars within specific fields who have not used and continuously bound” (109). The envi- experiments may be unaware of when and ronment in which an experiment takes place how experiments can be effective. A goal of is thus of particular importance to political this Handbook is to provide interested schol- scientists. ars with an efficient and effective way to learn And, although it might surprise some, about a broad range of experimental appli- political scientists have implemented exper- cations, how these applications complement iments in a wide range of contexts. Examples and supplement nonexperimental work, and can be found in every subfield. Applica- the opportunities and challenges inherent in tions to American politics include not only each type of application. topics such as media effects (e.g., Iyengar and Kinder 1987), mobilization (e.g., Gerber and Green 2000), and voting (e.g., Lodge, 3. Diversity of Experimental McGraw, and Stroh 1989), but also studies Methods of congressional and bureaucratic rules (e.g., Eavey and Miller 1984; Miller, Hammond, The most apparent source of variation in and Kile 1996). The field of international political science experiments is where they are relations, in some ways, lays claim to one conducted. To date, most experiments have of the longest ongoing experimental tradi- been implemented in one of three contexts: tions with its many studies of foreign policy laboratories, surveys, and the field. These decision making (e.g., Geva and Mintz 1997) types of experiments differ in terms of where and international negotiations (e.g., Druck- participants receive the stimuli (e.g., messages man 1994). Related work in comparative pol- encouraging them to vote), with that exposure itics explores coalition bargaining (e.g., Riker taking place, respectively, in a controlled set- 1967;Frechette´ et al. 2003) and electoral sys- ting; in the course of a phone, in-person, or tems (e.g., Morton and Williams 1999), and web-based survey; or in a naturally occurring recently, scholars have turned to experiments setting such as the voter’s home (e.g., in the to study democratization and development course of everyday life, and often without the (Wantchekon 2003), culture (Henrich et al. participants’ knowledge).6 2004), and identity (e.g., Sniderman, Hagen- Each type of experiment presents method- doorn, and Prior 2004; Habyarimana et al. ological challenges. For example, scholars 2007). Political theory studies include explo- have long bemoaned the artificial settings rations into justice (Frohlich and Oppen- of campus-based laboratory experiments and heimer 1992) and deliberation (Simon and the widespread use of student-aged sub- Sulkin 2001). jects. Although experimentalists from other Political scientists employ experiments 6 In some cases, whether an experiment is one type or across subfields and for a range of purposes. another is ambiguous (e.g., a web survey administered At the same time, many scholars remain in a classroom); the distinctions can be amorphous. Experimentation in Political Science 7 disciplines have examined implications of mental Economics stated that submissions that running experiments “on campus,” this lit- used deception or did not pay participants erature is not often cited by political sci- for their actions would not be accepted for entists (e.g., Dipboye and Flanagan 1979; publication.8 Kardes 1996;Kuhberger¨ 1998; Levitt and For psychologists and economists, differ- List 2007). Some political scientists claim that ences in experimental traditions reflect differ- the problems of campus-based experiments ences in their dominant paradigms. Because can be overcome by conducting experiments most political scientists seek first and fore- on representative samples. This may be true; most to inform political science debates, however, the conditions under which such norms about what constitutes a valid exper- changes produce more valid results have not iment in economics or psychology are not been broadly examined (see, e.g., Greenberg always applicable. So, for any kind of experi- 1987).7 ment, an important question to ask is: which Survey experiments, although not rely- experimental method is appropriate? ing on campus-based “convenience samples,” The current debate about this question also raise questions about external valid- focuses on more than the validity of the infer- ity. Many survey experiments, for example, ences that different experimental approaches expose subjects to phenomena they might can produce. Cost is also an issue. Survey have also encountered prior to participating and field experiments, for example, can be in an experiment, which can complicate causal expensive. Some scholars question whether inference (Gaines, Kuklinski, and Quirk the added cost of such endeavors (com- 2007). pared to, say, campus-based laboratory exper- Field experiments are seen as a way to iments) is justifiable. Such debates are lead- overcome the artificiality of other types of ing more scholars to evaluate the conditions experiments. In the field, however, there can under which particular types of experiments be less control over what experimental stimuli are cost effective. With the evolution of these subjects observe. It may also be more difficult debates has come the question of whether to get people to participate due to an inability the immediate costs of fielding an experiment to recruit subjects or to subjects’ unwilling- are offset by what Green and Gerber (2002) ness to participate as instructed once they are call the “downstream benefits of experimen- recruited. tation.” Downstream benefits refer to sub- Besides where they are conducted, another sequent outcomes that are set in motion by source of diversity in political science exper- the original experimental intervention, such iments is the extent to which they follow as the transmission of effects from one per- experimental norms in neighboring disci- son to another or the formation of habits. plines, such as psychology and economics. In some cases, the downstream benefits of This diversity is notable because psycholog- an experiment only become apparent decades ical and economic approaches to experimen- afterward. tation differ from each other. For example, In sum, the rise of an experimental polit- where psychological experiments often ical science brings both new opportunities include some form of deception, economists for discovery and new questions about the consider it taboo. Psychologists rarely pay price of experimental knowledge. This Hand- subjects for specific actions they undertake book is organized to make the broad range during an experiment. Economists, in con- of research opportunities more apparent and trast, often require such payments (Smith 1976). Indeed, the inaugural issue of Experi- 8 Of the laboratory experiments identified as appearing in the APSR through 2005, half employed induced 7 As Campbell (1969) states, “ . . . had we achieved one, value theory, such that participants received finan- there would be no need to apologize for a successful cial rewards contingent on their performance in the psychology of college sophomores, or even of North- experiment. Thirty-one percent of laboratory exper- western University coeds, or of Wistar staring white iments used deception; no experiments used both rats” (361). induced value and deception. 8 James N. Druckman, Donald P. Green, James H. Kuklinski, and Arthur Lupia to help scholars manage the challenges with Part II contains four essays written by greater effectiveness and efficiency. prominent scholars who each played an important role in the development of experimental political science.10 These essays 4. The Volume provide important historical perspectives and relevant biographic information on the devel- In concluding his book on the ten most fasci- opment of experimental research agendas. nating experiments in the history of science, The authors describe the questions they Johnson (2008) explains that “I’ve barely hoped to resolve with experiments and why finished the book and already I’m second- they believe that their efforts succeeded and guessing myself” (158). We find ourselves in failed as they did. These essays also document an analogous situation. There are many excit- the role experiments played in the evolution ing kinds of experimental political science on of much broader fields of inquiry. which we can focus. Although the Handbook’s Parts III to VIII of the Handbook explore content does not cover all possible topics, the role of political science experiments we made every effort to represent the broad on a range of scholarly endeavors. The range of activities that contemporary experi- chapters in these parts clarify how exper- mental political science entails. The content iments contribute to scientific and social of the Handbook is as follows. knowledge of many important kinds of We begin, in Part I, with a series of political phenomena. They describe cases chapters that provide an introduction to in which experiments complement nonexperimental methods and concepts. These experimental work, as well as cases where chapters provide detailed discussion of what experiments advance knowledge in ways that constitutes an experiment, as well as the nonexperimental work cannot. Each chap- key considerations underlying experimental ter describes how to think about experimen- designs (i.e., internal and external validity, tation on a particular topic and provides student subjects, payment, and deception). advice about how to overcome practical (and, Although these chapters do not delve into the when relevant, ethical) hurdles to design and details of precise designs and statistical analy- implementation. ses (see, e.g., Keppel and Wickens 2004;Mor- In developing this part of the Handbook, ton and Williams 2010), their purpose is to we attempted to include topics where exper- provide a sufficient base for reading the rest of iments have already played a notable role. the Handbook. We asked the authors of these We devoted less space to “emerging” top- chapters not only to review extant knowledge, ics in experimental political science that have but also to present arguments that help place great potential to answer important questions the challenges of, and opportunities in, exper- but are still in early stages of development. imental political science in a broader perspec- Examples of such work include genetic and tive. For example, our chapters regard ques- neurobiological approaches (e.g., Fowler and tions about external validity (i.e., the extent Schreiber 2008), nonverbal communication to which one can generalize experimental (e.g., Bailenson et al. 2008), emotions (e.g., findings) as encompassing much more than Druckman and McDermott 2008), cultural whether a study employs a representative (or, norms (e.g., Henrich et al. 2004), corruption at least, nonstudent) sample. This approach major issue in political science experimentation. In to the chapters yields important lessons about addition, more general relevant discussions are read- when student-based samples, and other com- ily available (e.g., Singer and Levine 2003; Hauck mon aspects of experimental designs, are and 2008). Also see Halpern (2004) on ethics in clinical 9 trials. Other methodological topics for which we do are not problematic. not have chapters include Internet methodology and quasi-experimental designs. 9 Perhaps the most notable topic absent from our intro- 10 Of course, many others played critical roles in the ductory chapters is ethics and institutional review development of experimental political science, and boards. We do not include a chapter on ethics because we take some comfort that most of these others have it is our sense that, to date, it has not surfaced as a contributed to other volume chapters. Experimentation in Political Science 9

(e.g., Ferraz and Finan 2008; Malesky and duces new inferential power by inducing Samphantharak 2008), ethnic identity (e.g., researchers to exercise control over the sub- Humphreys, Posner, and Weinstein 2002), jects of study, to randomly assign subjects to and elite responsiveness (e.g., Esterling, various conditions, and to carefully record Lazer, and Neblo 2009; Richardson and John observations. Political scientists who learn 2009). Note that the Handbook is written in how to design and conduct experiments care- such a way that any of the included chapters fully are often rewarded with a clearer view can be read and used without having read the of cause and effect. chapters that precede them. Although political scientists disagree about The final part of the book, Part IX, covers a many methodological matters, perhaps there number of advanced methodological debates. is a consensus that political science best serves The chapters in this part address the chal- the public when its findings give citizens and lenges of making causal inferences in complex policymakers a better understanding of their settings and over time. As with the preced- shared environs. When such understandings ing methodological chapters, these chapters require stark and powerful claims about cause do more than review basic issues; they also and effect, the discipline should encourage develop arguments on how to recognize and experimental methods. When designed in a adapt to such challenges in future research. way that target audiences find relevant, exper- The future of experimental political sci- iments can enlighten, inform, and transform ence offers many new opportunities for cre- critical aspects of societal organization. ative scholars. It also presents important challenges. We hope that this Handbook makes the challenges more manageable for you and the References opportunities easier to seize. Bailenson, Jeremy N., Shanto Iyengar, Nick Yee, and Nathan A. Collins. 2008. “Facial Similarity 5. Conclusion between Voters and Candidates Causes Influ- ence.” Public Opinion Quarterly 72: 935–61. In many scientific disciplines, experimental Brody, Richard A., and Charles N. Brownstein. research is the focal form of scholarly activ- 1975. “Experimentation and Simulation.” In ity. In these fields of study, disciplinary norms Handbook of Political Science 7,eds.FredI. and great discoveries are indescribable with- Greenstein and Nelson Polsby. Reading, MA: Addison-Wesley, 211–63. out reference to experimental methods. For Brown, Stephen R., and Lawrence E. Melamed. the most part, political science is not such a 1990. Experimental Design and Analysis. New- science. Its norms and great discoveries often bury Park, CA: Sage. come from scholars who integrate and blend Campbell, Donald T. 1969. “Prospective: Artifact multiple methods. In a growing number of and Control.” In Artifact in Behavioral Research, topical areas, experiments are becoming an eds. Robert Rosenthal and Robert Rosnow. increasingly common and important element New York: Academic Press, 264–86. of a political scientist’s methodological tool Cover, Albert D., and Bruce S. Brumberg. 1982. kit (see also Falk and Heckman 2009). Partic- “Baby Books and Ballots: The Impact of ularly in recent years, there has been a massive Congressional Mail on Constituent Opinion.” 76 347 59 expansion in the number of political scientists American Political Science Review : – . Dipboye, Robert L., and Michael F. Flanagan. who see experiments as useful and, in some 1979. “Research Settings in Industrial and cases, transformative. Organizational Psychology: Are Findings in the Experiments appeal to our discipline Field More Generalizable Than in the Labora- because of their potential to generate stark tory?” American Psychologist 34: 141–50. and powerful empirical claims. Experiments Druckman, Daniel. 1994. “Determinants of Com- can expand our abilities to change how crit- promising Behavior in Negotiation: A Meta- ical target audiences think about important Analysis.” Journal of Conflict Resolution 38: 507– phenomena. The experimental method pro- 56. 10 James N. Druckman, Donald P. Green, James H. Kuklinski, and Arthur Lupia

Druckman, James N., Donald P. Green, James Geva, Nehemia, and Alex Mintz. 1997. Decision- H. Kuklinski, and Arthur Lupia. 2006.“The Making on War and Peace: The Cognitive- Growth and Development of Experimental Rational Debate. Boulder, CO: Lynne Rienner. Research Political Science.” American Political Gilens, Martin. 1996. “‘Race Coding’ and White Science Review 100: 627–36. Opposition to Welfare.” American Political Sci- Druckman, James N., and Arthur Lupia. 2006. ence Review 90: 593–604. “Mind, Will, and Choice.” In The Oxford Hand- Gosnell, Harold F. 1926. “An Experiment in the book on Contextual Political Analysis,eds.Charles Stimulation of Voting.” American Political Sci- Tilly and Robert E. Goodin. Oxford: Oxford ence Review 20: 869–74. University Press, 97–113. Green, Donald P. 1992. “The Price Elasticity Druckman, James N., and Rose McDermott. 2008. of Mass Preferences.” American Political Science “Emotion and the Framing of Risky Choice.” Review 86: 128–48. Political Behavior 30: 297–321. Green, Donald P., and Alan S. Gerber. 2002. Eavey, Cheryl L., and Gary J. Miller. 1984. “The Downstream Beneﬁts of Experimenta- “Bureaucratic Agenda Control: Imposition or tion.” Political Analysis 10: 394–402. Bargaining?” American Political Science Review Greenberg, Jerald. 1987. “The College Sopho- 78: 719–33. more as Guinea Pig: Setting the Record Eldersveld, Samuel J. 1956. “Experimental Pro- Straight.” Academy of Management Review 12: paganda Techniques and Voting Behavior.” 157–59. American Political Science Review 50: 154–65. Guetzkow, Harold, and Joseph J. Valadez, eds. Esterling, Kevin Michael, David Lazer, and 1981. Simulated International Processes. Beverly Michael Neblo. 2009. “Means, Motive, and Hills, CA: Sage. Opportunity in Becoming Informed about Pol- Habyarimana, James, Macartan Humphreys, itics: A Deliberative Field Experiment Involv- Daniel Posner, and Jeremy M. Weinstein. ing Members of Congress and Their Con- 2007. “Why Does Ethnic Diversity Under- stituents.” Unpublished paper, University of mine Public Goods Provision?” American Polit- California, Riverside. ical Science Review 101: 709–25. Falk, Armin, and James J. Heckman. 2009.“Lab Halpern, Sydney A. 2004. Lesser Harms: The Moral- Experiments Are a Major Source of Knowl- ity of Risk in Medical Research. Chicago: The edge in the Social Sciences.” Science 326: 535– University of Chicago Press. 38. Hauck, Robert J.-P. 2008. “Protecting Human Ferraz, Claudio, and Frederico Finan. 2008. Research Participants, IRBs, and Political Sci- “Exposing Corrupt Politicians: The Effects of ence Redux: Editor’s Introduction.” PS: Politi- Brazil’s Publicly Released Audits on Electoral cal Science & Politics 41: 475–76. Outcomes.” Quarterly Journal of Economics 123: Henrich, Joseph, Robert Boyd, Samuel Bowles, 703–45. Colin Camerer, Ernst Fehr, and Herbert Gin- Fowler, James H., and Darren Schreiber. 2008. tis, eds. 2004. Foundations of Human Society. “Biology, Politics, and the Emerging Science Oxford: Oxford University Press. of Human Nature.” Science 322: 912–14. Humphreys, Macartan, Daniel N. Posner, and Frechette,´ Guillaume, John H. Kagel, and Steven Jeremy M. Weinstein. 2002. “Ethnic Iden- F. Lehrer. 2003. “Bargaining in Legisla- tity, Collective Action, and Conﬂict: An Exper- tures.” American Political Science Review 97: imental Approach.” Paper presented at the 221–32. annual meeting of the American Political Sci- Frohlich, Norman, and Joe A. Oppenheimer. ence Association, Boston. 1992. Choosing Justice: An Experimental Approach Iyengar, Shanto, and Donald R. Kinder. 1987. to Ethical Theory. Berkeley: University of Cali- News That Matters: Television and American fornia Press. Opinion. Chicago: The University of Chicago Gaines, Brian J., James H. Kuklinski, and Paul Press. J. Quirk. 2007. “The Logic of the Survey Johnson, George. 2008. The Ten Most Beautiful Experiment Reexamined.” Political Analysis 15: Experiments. New York: Alfred A. Knopf. 1–20. Kardes, Frank R. 1996. “In Defense of Experimen- Gerber, Alan S., and Donald P. Green. 2000.“The tal Consumer Psychology.” Journal of Consumer Effects of Canvassing, Telephone Calls, and Psychology 5: 279–96. Direct Mail on Voter Turnout.” American Polit- Keppel, Geoffrey, and Thomas D. Wickens. 2004. ical Science Review 94: 653–63. Design and Analysis: A Researcher’s Handbook. Experimentation in Political Science 11

4th ed. Upper Saddle River, NJ: Pearson/ Morton, Rebecca B., and Kenneth C. Williams. Prentice Hall. 1999. “Information Asymmetries and Simul- King, Gary, Robert O. Keohane, and Sidney taneous versus Sequential Voting.” American Verba. 1994. Designing Social Inquiry: Scientific Political Science Review 93: 51–67. Inference in Qualitative Research. Princeton, NJ: Morton, Rebecca B., and Kenneth C. Williams. Princeton University Press. 2010. Experimental Political Science and the Study Kuhberger,¨ Anton. 1998. “The Influence of Fram- of Causality: From Nature to the Lab. New York: ing on Risky Decisions.” Organizational Behav- Cambridge University Press. ior and Human Decision Processes 75: 23–55. Ostrom, Elinor, James Walker, and Roy Gard- Levitt, Steven D., and John A. List. 2007. “What ner. 1992. “Covenants with and without a Do Laboratory Experiments Measuring Social Sword.” American Political Science Review 86: Preferences Tell Us about the Real World?” 404–17. Journal of Economic Perspectives 21: 153–74. Richardson, Liz, and Peter John. 2009. “Is Lob- Lijphart, Arend. 1971. “Comparative Politics and bying Really Effective? A Field Experiment the Comparative Method.” American Political of Local Interest Group Tactics to Influ- Science Review 65: 682–93. ence Elected Representatives in the UK.” Lodge, Milton, Kathleen M. McGraw, and Patrick Paper presented at the European Consortium Stroh. 1989. “An Impression-Driven Model of for Political Research Joint Sessions, Lisbon, Candidate Evaluation.” American Political Sci- Portugal. ence Review 83: 399–419. Riker, William H. 1967. “Bargaining in a Three- Lodge, Milton, Marco R. Steenbergen, and Shawn Person Game.” American Political Science Review Brau. 1995. “The Responsive Voter: Campaign 61: 642–56. Information and the Dynamics of Candidate Roth, Alvin E. 1995. “Introduction to Experimen- Evaluation.” American Political Science Review tal Economics.” In The Handbook of Experimen- 89: 309–26. tal Economics, eds. John H. Kagel and Alvin Lowell, A. Lawrence. 1910. “The Physiology of E. Roth. Princeton, NJ: Princeton University Politics.” American Political Science Review 4: 1– Press, 1–110. 15. Simon, Adam, and Tracy Sulkin. 2001.“Habermas Mahoney, Robert, and Daniel Druckman. 1975. in the Lab: An Experimental Investigation of “Simulation, Experimentation, and Context.” the Effects of Deliberation.” Political Psychology Simulation & Games 6: 235–70. 22: 809–26. Malesky, Edmund J., and Krislert Samphan- Singer, Eleanor, and Felice J. Levine. 2003.“Pro- tharak. 2008. “Predictable Corruption and tection of Human Subjects of Research: Recent Firm Investment: Evidence from a Natu- Developments and Future Prospects for the ral Experiment and Survey of Cambodian Social Sciences.” Public Opinion Quarterly 67: Entrepreneurs.” Quarterly Journal of Political 148–64. Science 3: 227–67. Smith, Vernon L. 1976. “Experimental Eco- Miller, Gary J., Thomas H. Hammond, and nomics: Induced Value Theory.” American Eco- Charles Kile. 1996. “Bicameralism and the nomic Review 66: 274–79. Core: An Experimental Test.” Legislative Stud- Sniderman, Paul M., Look Hagendoorn, and ies Quarterly 21: 83–103. Markus Prior. 2004. “Predispositional Factors Morton, Rebecca B. 1993. “Incomplete Informa- and Situational Triggers.” American Political tion and Ideological Explanations of Platform Science Review 98: 35–50. Divergence.” American Political Science Review Wantchekon, Leonard. 2003. “Clientelism and 87: 382–92. Voting Behavior.” World Politics 55: 399–422.

Part I

DESIGNING EXPERIMENTS

CHAPTER 2

Experiments

An Introduction to Core Concepts

James N. Druckman, Donald P. Green, James H. Kuklinski, and Arthur Lupia

The experimental study of politics has support for public officials (Druckman 2004; grown explosively in the past two decades. Tomz 2007). And although the initial wave Part of that explosion takes the form of a dra- of field experiments focused on the effects matic increase in the number of published of campaign communications on turnout and articles that use experiments. Perhaps less evi- voters’ preferences (Eldersveld 1956; Gerber dent, and arguably more important, exper- and Green 2000; Wantchekon 2003), re- imentalists are exploring topics that would searchers increasingly use field experiments have been unimaginable only a few years ago. and natural experiments to study phenomena Laboratory researchers have studied topics as varied as election fraud (Hyde 2009), repre- ranging from the effects of media exposure sentation (Butler and Nickerson 2009), coun- (Iyengar and Kinder 1987) to the conditions terinsurgency (Lyall 2009), and interpersonal under which groups solve collective action communication (Nickerson 2008). problems (Ostrom, Walker, and Gardner With the rapid growth and development 1992), and, at times, have identified empir- of experimental methods in political science ical anomalies that produced new theoret- come a set of terms and concepts that political ical insights (McKelvey and Palfrey 1992). scientists must know and understand. In this Some survey experimenters have developed chapter, we review concepts and definitions experimental techniques to measure preju- that often appear in the Handbook chapters. dice (Kuklinski, Cobb, and Gilens 1997) and We also highlight features of experiments its effects on support for policies such as that are unique to political science. welfare or affirmative action (Sniderman and Piazza 1995); others have explored the ways in which framing, information, and decision 1. What Is an Experiment? cues influence voters’ policy preferences and In contrast to modes of research that We thank Holger Kern for helpful comments. address descriptive or interpretive questions,

15 16 James N. Druckman, Donald P. Green, James H. Kuklinski, and Arthur Lupia researchers design experiments to address cally that watching the debate caused these causal questions. A causal question invites a differences. comparison between two states of the world: In an effort to address such concerns, ob- one in which some sort of intervention is servational researchers often attempt to com- administered (a treated state, i.e., exposing pare treated and untreated people only when a subject to a stimulus) and another in which they share certain attributes, such as age or it is not (an untreated state). The fundamen- ideology. Researchers implement this general tal problem of causal inference arises because approach in many ways (e.g., multiple regres- we cannot simultaneously observe a person sion analysis, case-based matching, case con- or entity in its treated and untreated states trol methodology), but all employ a similar (Holland 1986). underlying logic: find a group of seemingly Consider, for example, the causal effect comparable observations that have received of viewing a presidential debate. Rarely are different treatments, then base the causal the elections of 1960, 1980, 1984,or2000 evaluation primarily or exclusively on these recounted without mentioning the critical observations. role that debates played in shaping voter Such approaches often fail to eliminate opinion. What is the basis for thinking that comparability problems. There might be no viewing a presidential debate influences the way to know whether individuals who look public’s support for the candidates? We do similar in terms of a (usually limited) set not observe how viewers of the debate would of observed attributes would in fact have have voted had they not seen the debate. We responded identically to a particular treat- do not observe how nonviewers would have ment. Two groups of individuals who look voted had they watched (Druckman 2003). the same to researchers could differ in unmea- Nature does not provide us with the obser- sured ways (e.g., openness to persuasion). vations we would need to make the precise This problem is particularly acute when peo- causal comparisons that we seek. ple self-select into or out of a treatment. Social scientists have pursued two empiri- Whether people decide to watch or not cal strategies to overcome this conundrum: watch a debate, for example, might depend observational research and experimental on unmeasured attributes that predict which research. Observational research involves a candidate they support (e.g., people who comparison between people or entities sub- favor the front-running candidate before jected to different treatments (at least, in the debate might be more likely to watch part, of their own choosing). Suppose that the debate than those who expect their can- some people watched a presidential debate, didate to lose). whereas others did not. To what extent can Experimental research differs from obser- we determine the effect of debate watch- vational research in that the entities under ing by comparing the postdebate behav- study are randomly assigned to different iors of viewers and nonviewers? The answer treatments. Here, treatments refer to poten- depends on the extent to which viewers and tially causal interventions. For example, an nonviewers are otherwise similar. It might experimenter might assign some people to be that most debate watchers already sup- watch a debate (one treatment) and assign ported one candidate, whereas most non- others to watch a completely different pro- watchers favored the other. In such cases, gram (a second treatment). Depending on observed differences between the postdebate the experimental design, there may be a con- opinions of watchers and nonwatchers could trol group that does not receive a treatment stem largely from differences in the opin- (e.g., they are neither told to watch nor dis- ions they held before the debate even started. couraged from watching the debate) and/or Hence, to observe that viewers and nonview- an alternative treatment group (e.g., they are ers express different views about a candi- told to watch a different show or a differ- date after a debate does not say unequivo- ent part of the debate). Random assignment Experiments 17 means that each entity being studied has an units of observation (typically, subjects, or equal chance to be in a particular treatment human participants in an experiment) are ran- condition.1 domly assigned to different treatment or con- Albertson and Lawrence (2009) and Mul- trol groups (although see note 2). Exper- lainathan, Washington, and Azari (2010), for imental studies can take many forms. It example, discuss experiments with encourage- is customary to classify randomized stud- ment designs in which the researcher randomly ies according to the settings in which they encourages some survey respondents to view take place: a lab experiment involves an inter- an upcoming candidate debate (treatment vention in a setting created and controlled group) and neither encourages or discourages by the researcher; a field experiment takes others (control group). After the debate, the place in a naturally occurring setting; and researcher conducts a second interview with a survey experiment involves an intervention both groups in order to ascertain whether in the course of an opinion survey (which they watched the debate and to measure their might be conducted in person, over the candidate preferences. phone, or via the web). This classification How does random assignment overcome scheme is not entirely adequate, however, the fundamental problem of causal inference? because studies often blend different aspects Suppose for the time being that everyone of lab, field, and survey experiments. For who was encouraged to view the debate did example, some experiments take place in so and that no one watched unless encour- lab-like settings, such as a classroom, but aged. Although we cannot observe a given require the completion of a survey that con- individual in both his or her treated and tains the experimental treatments (e.g., the untreated states, random assignment enables treatments might entail providing individuals the researcher to estimate the average treat- with different types of information about an ment effect. Prior to the intervention, the ran- issue). domly assigned treatment and control groups have the same expected responses to viewing the debate. Apart from random sam- 2. Random Assignment or pling variability, in other words, random Random Sampling? assignment provides a basis for assuming that the control group behaves as the treatment When evaluating whether a study qualifies group would have behaved had it not received as an experiment, by our definition, random the treatment (and vice versa). By compar- assignment should not be confused with ran- ing the average outcome in the treatment dom sampling. Random sampling refers to a group to the average outcome in the control procedure by which participants are selected group, the experimental researcher estimates for certain kinds of studies. A common ran- the average treatment effect. Moreover, the dom sampling goal is to choose participants researcher can perform statistical tests to clar- from a broader population in a way that gives ify whether the differences between groups every potential participant the same proba- occurred simply by chance (sampling vari- bility of being selected into the study. Ran- ability) rather than as a result of experimental dom assignment differs. It does not require treatments. that participants be drawn randomly from When we speak of an experiment in this some larger population. Experimental par- Handbook, we mean a study in which the ticipants might come from undergraduate courses or from particular towns. The key 1 In the social sciences, in contrast to the physical sciences, experiments tend to involve use of random requirement is that a random procedure, such assignment to treatment conditions. Randomly as a coin flip, determines whether they receive assigned treatments are one type of “independent a particular treatment. Just as an experi- variable.” Another type comprises “covariates” that are not randomly assigned but nonetheless predict ment does not require a random sample, a the outcome. study of a random sample need not be an 18 James N. Druckman, Donald P. Green, James H. Kuklinski, and Arthur Lupia experiment. A survey that merely asks a ran- 3. Internal and External Validity dom sample of adults whether they watched a presidential debate might be a fine study, Random assignment enables the researcher to but it is not an experimental study of the formulate the appropriate comparisons, but effects of debate viewing because watching random assignment alone does not ensure or not watching the debate was not randomly that the comparison will speak convincingly assigned. to the original causal question. The theoret- The typical social science experiment ical interpretation of an experimental result uses a between-subjects design, insofar as the is a matter of internal validity – “did in fact researcher randomly assigns participants to the experimental stimulus [e.g., the debate] distinct treatment groups. An alternative make some significant difference [e.g., in atti- approach is the within-subjects design in which tude toward the candidate] in this specific a given participant is observed before and instance” (Campbell 1957, 297).3 In the pre- after receiving a treatment (e.g., there is no ceding example, the researcher seeks to gauge random assignment between subjects). Intu- the causal effect of viewing a televised debate; itively, the within-subjects design seems to however, if viewers of the debate are inad- overcome the fundamental problem of causal vertently exposed to attitude-changing news inference; in practice, it is often vulnerable to as well, then the estimated effect of viewing confounds – meaning unintended and uncon- the debate will be conflated with the effect of trolled factors that influence the results. For hearing the news. example, suppose that a researcher measures The interpretation of the estimated causal subjects’ attitudes toward a candidate before effect also depends on what the control group they watch a debate, and then again after receives as a treatment. If, in the previous they have watched it, to determine whether example, the control group watches another the debate changed their attitudes. If subjects television program that airs campaign com- should hear attitude-changing news about the mercials, the researcher must understand candidate after the first measurement and the treatment effect as the relative influ- prior to the second, or if simply filling out ence of viewing debates compared to viewing the predebate questionnaire induces them to commercials.4 This comparison differs from a watch the debate differently than they oth- comparison of those who watch a debate with erwise would have watched, a comparison those who, experimentally, watch nothing. of pre- and postattitudes will produce mis- More generally, every experimental treat- leading conclusions about the effect of the ment entails subtle nuances that the rese- 2 debate. archer must know, understand, and explicate. Hence, in the preceding example, he or she must judge whether the causative agent was viewing a debate per se, any 90-minute polit- 2 Natural scientists frequently use within-subjects designs because they seldom contend with prob- ical program, or any political program of any lems of memory and anticipation when working length. Researchers can, and should, conduct with “subjects” such as electrons. Clearly, natural scientists conduct “experiments” (with interventions) even if they do not employ between-subjects 3 Related to internal validity is statistical conclusion random assignment. Social scientists, confronted as validity, defined as “the validity of inferences about they are by the additional complexities of work- the correlation (covariation) between treatment and ing with humans, typically rely on between-subjects outcome” (Shadish et al. 2002, 38). Statistical con- experimental designs, where randomization ensures clusion validity refers specifically and solely to the that the experimental groups are, in expectation, “appropriate use of statistics to infer whether the identical. presumed independent and dependent variables co- Randomization is unnecessary when subjects are vary,” and not at all to whether a true causal relation- effectively identical. In economics (and hence some ship exists (37). of the work discussed in this Handbook), participants 4 Internal validity is a frequent challenge for experi- sometimes are not randomly assigned on the assump- mental research. For this reason, experimental schol- tion that they all respond the same way to the incen- ars often administer manipulation checks, evaluations tives provided by the experimenter (Guala 2005, 79; that document whether subjects experience the treat- Morton and Williams 2010, 28–29). ment as intended by the experimenter. Experiments 19 multiple experiments or experiments with a erally, despite having data on relatively few wide array of different conditions in an effort voters. How far one can generalize from the to isolate the precise causative agent; how- results of a particular experiment is a question ever, at the end of the day, they must rely of external validity: the extent to which the on theoretical stipulations to decide which “causal relationship holds over variations in idiosyncratic aspects of the treatment are rel- persons, settings, treatments, and outcomes” 5 evant and explain why they, and not others, (Shadish, Cook, and Campbell 2002, 83). are relevant. As suggested in the Shadish et al. quote, Two aspects of experimental implementa- external validity covers at least four aspects of tion that bear directly on internal validity are experimental design: whether the participants noncompliance and attrition. Noncompliance resemble the actors who are ordinarily con- occurs when those assigned to the treatment fronted with these stimuli, whether the con- group do not receive the treatment, or when text (including the time) within which actors those assigned to the control group inad- operate resembles the context (and time) of vertently receive the treatment (e.g., those interest, whether the stimulus used in the encouraged to watch do not watch or those study resembles the stimulus of interest in the not encouraged do watch). In this case, the world, and whether the outcome measures randomly assigned groups remain compa- resemble the actual outcomes of theoretical rable, but the difference in their average or practical interest. The fact that several outcomes measures the effect of the exper- criteria come into play means that experi- imental assignment rather than actually ments are difficult to grade in terms of exter- receiving the treatment. The Appendix to this nal validity, particularly because the external chapter describes how to draw causal infer- validity of a given study depends on what ences in such circumstances. kinds of generalizations one seeks to make. Attrition involves the failure to measure Consider the external validity of our exam- outcomes for certain subjects (e.g., some do ple of the debate-watching encouragement not report their vote preference in the follow- experiment. The subjects in encouragement up) and is particularly problematic when it studies come from random samples of the afflicts some experimental groups more than populations of adults or registered voters. others. The danger is that attrition reveals Random sampling bolsters the external valid- something about the potential outcomes of ity of the study insofar as the people in the sur- those who drop out of the study. For exam- vey better reflect the target population. How- ple, if debate viewers become more willing ever, if certain types of people comply with than nonviewers to participate in a postdebate encouragement instructions more than oth- interview and if viewing the debate changes ers, then our post-treatment inferences will subjects’ candidate evaluations, comparisons depend on whether the average effect among between treatment and control group could those who comply with the treatment resem- be biased. Sometimes researchers unwittingly bles the average effect among those groups to contribute to the problem of differential attri- which we hope to generalize. tion by exerting more effort to gather out- A related concern in such experiments is come data from one of the experimental whether the context and time at which par- groups or by expelling participants from the ticipants watch the debate resembles settings study if they fail to follow directions when to which the researcher hopes to generalize. receiving the treatment. Are the viewers allowed to ignore the debate A related concern for experimental rese- and read a magazine if they want (as they archers is external validity. Researchers typi- could outside the study)? Are they watch- cally conduct experiments with an eye toward ing with the same types of people they would questions that are bigger than “What is the causal effect of the treatment on this particu- 5 Related is construct validity, which is “the validity of inferences about the higher order constructs that lar group of people?” For example, they may represent sampling particulars” (Shadish et al. 2002, want to provide insight about voters gen- 38). 20 James N. Druckman, Donald P. Green, James H. Kuklinski, and Arthur Lupia watch with outside the study? There also are and an outcome variable, they often want to questions about the particular debate pro- probe further to understand the mechanisms gram used in the study (e.g., the stimulus): by which the effect is transmitted. For exam- does it typify debates in general? To the ple, having found that watching a televised extent that it does not, it will be more difficult debate increased the likelihood of voting, they to make claims about debate viewing that are ask why it has this effect. Is it because viewers regarded as externally valid. Before general- become more interested in the race? Do they izing from the results of such an experiment, feel more confident about their ability to cast we would need to know more about the tone, an intelligent vote? Do debates elevate their content, and context of the debate.6 feelings of civic duty? Viewing a debate could Finally, suppose our main interest is in how change any of these mediating variables. debate viewing affects Election Day behav- Assessing the extent to which potential iors. If we want to understand how exposure mediating variables explain an experimen- to debates influences voting, then a question- tal effect can be challenging. Analytically, a naire given on Election Day might be a bet- single random assignment (viewing a debate ter measurement than one taken immediately vs. not viewing) makes it difficult, if not after the debate and well before the election; impossible, to isolate the mediating path- that is, behavioral intentions may change after ways of numerous intervening variables. To the debate but before the election. clarify such effects, a researcher needs to Whether any of these concerns make a design several experiments, all with differ- material difference to the external validity ent kinds of treatments. In the debate exam- of an experimental finding can be addressed ple, a researcher could ask subjects to watch as part of an extended research program in different kinds of debates, with some treat- which scholars vary relevant attributes of the ments likely to affect interest in the race research design, such as the subjects targeted and others to heighten feelings of civic duty. for participation, the alternative viewing (or Indeed, an extensive series of experiments reading) choices available (to address the gen- might be required before a researcher can eralizability of effects from watching a par- make convincing causal claims about causal ticular debate in a certain circumstance), the pathways. types of debates watched, and the timing of In addition to identifying mediating vari- postdebate interviews. A series of such exper- ables, researchers often want to understand iments could address external validity con- the conditions under which an experimen- cerns by gradually assessing how treatment tal treatment affects an important outcome. effects vary depending on different attributes For example, do debates only affect (or affect of experimental design. to a greater extent) political independents? Do debates matter only when held in close proximity to Election Day? These are ques- 4. Documenting and Reporting tions about , wherein the treat- Relationships moderation ment’s effect on the outcome differs across levels of other variables (e.g., partisanship, When researchers detect a statistical relation- timing of debate [see Baron and Kenny ship between a randomly assigned treatment 1986]). Documenting moderating relationships typically entails the use of statistical 6 This is related to the aforementioned internal validity concern about whether the content of the debate interactions between the moderating variable itself caused the reaction, or whether any such pro- and the treatment. This approach, however, gramming would have caused it. The internal validity requires sufficient variance on the moderat- concern is about the causal impact of the presumed stimulus – is the cause what we believe it is (e.g., ing variable. For example, to evaluate whether the debate and not any political programming)? The debates affect only independents, the subject external validity concern is about whether that causal population must include sufficient numbers agent reflects the set of causal variables to which we hope to infer (e.g., is the content of the debate rep- of otherwise comparable independents and resentative of presidential debates?). nonindependents. Experiments 21

In practice, pinpointing mediators and journal reviewers, editors, and even authors moderators often requires theoretical guid- themselves. If statistically significant positive ance and the use of multiple experiments results are published and weaker results are representing distinct conditions. This gets at not, then the published literature will give a one of the great advantages of experiments – distorted impression of a treatment’s influ- they can be replicated and extended in order ence. A meta-analysis of results that have to form a body of related studies. Moreover, been published selectively might be quite as experimental literatures develop, they lend misleading. For example, if only experiments themselves to meta-analysis, a form of statisti- documenting that debates affect voter opin- cal analysis that assesses the conditions under ion survive the publication process and those which effects are large or small (Borenstein that report no effects are never published, et al. 2009). Meta-analyses aggregate exper- then the published literature may provide a iments on a given topic into a single dataset skewed view of debate effects. For this reason, and test whether effect sizes vary with certain researchers who employ meta-analysis should changes in the treatments, subjects, context, look for symptoms of publication bias, such as or manner in which the experiments were the tendency for smaller studies to generate implemented. Meta-analysis can reveal sta- larger treatment effects. tistically significant treatment effects from a As the discussions of validity and pub- set of studies that, analyzed separately, would lication bias suggest, experimentation is no each generate estimated treatment effects panacea.9 The interpretation of experimental indistinguishable from zero. Indeed, it is this results requires intimate knowledge of how feature of meta-analysis that argues against and under what conditions an experiment the usual notion that one should always avoid was conducted and reported. For this reason, conducting experiments with low statistical it is incumbent on experimental researchers power, or a low probability of rejecting the null to give a detailed account of the key fea- hypothesis of no effect (when there is in fact tures of their studies, including 1)whothe an effect).7 A set of low power studies taken subjects are and how they came to partici- together might have considerable power, but pate in the study; 2) how the subjects were if no one ever launches a low power study, randomly assigned to experimental groups; this needed evidence cannot accumulate 3) what treatments each group received; 4) (for examples of meta-analyses in political the context in which each group received science, see Druckman 1994; Lau et al. treatments; 5) the outcome measures; and 6) 1999).8 all procedures used to preserve comparabil- Publication bias threatens the accumula- ity between treatment and control groups, tion of experimental evidence through meta- such as outcome measurement that is blind analysis. Some experiments find their way to participants’ experimental assignments into print more readily than others. Those and the management of noncompliance and that generate statistically significant results attrition. and show that the effect of administering a treatment is clearly nonzero are more likely to be deemed worthy of publication by 5. Ethics and Natural Experiments 7 Statistical power refers to the probability that a researcher will reject the null hypothesis of no effect Implementing experiments in ways that speak when the alternative hypothesis is indeed true. convincingly to causal questions is important 8 Early lab and field studies of the mass media fall into this category. Iyengar, Peters, and Kinder’s (1982) influential lab study of television news had less than 9 The volume does not include explicit chapters on twenty subjects in some of the experimental con- meta-analysis or publication bias, reflecting, in part, ditions. Panagopoulos and Green’s (2008) study of the still relatively recent rise in experimental meth- radio advertising comprised a few dozen mayoral ods (i.e., in many areas, there is not yet a sufficient elections. Neither produced overwhelming statisti- accumulation of evidence). We imagine these topics cal evidence on its own, but both have been bolstered will soon receive considerably more attention within by replications. political science. 22 James N. Druckman, Donald P. Green, James H. Kuklinski, and Arthur Lupia and challenging. Experiments that have great natural experiments can provide scholars with clarifying potential can also be expensive and an opportunity to conduct research on topics difficult to orchestrate, particularly in situa- that would ordinarily be beyond an experi- tions where the random assignment of treat- menter’s reach. ments means a sharp departure from what would ordinarily occur. For experiments on certain visible or conflictual topics, ethical 6. Conclusion problems might also arise. Subjects might either be denied a treatment that they would That social science experiments take many ordinarily seek or be exposed to a treatment forms reflects different judgments about how that they would ordinarily avoid. Even if the best to balance various research aims. Some ethical problems are manageable, such situa- scholars prefer laboratory experiments to tions might also require researchers to garner field experiments on the grounds that the lab potential subjects’ explicit consent to partic- offers the researcher tighter control over the ipate in the experimental activities. Subjects treatment and how it is presented to sub- might refuse to consent, or the consent form jects. Others take the opposite view on the might prompt them to think or behave in grounds that generalization will be limited ways they otherwise would not – in both unless treatments are deployed, and outcomes instances, challenging the external validity assessed, unobtrusively in the field. Survey of the experiment. Moreover, some studies experiments are sometimes preferred on the include deception, an aspect of experimental grounds that a large and representative sam- design that raises not only ethical qualms, but ple of people can be presented with a broad also practical concerns about jeopardizing the array of different stimuli in an environment credibility of the experimental instructions in where detailed outcome measures are easily future experiments. gathered. Finally, some scholars turn to nat- Hence, the creative spark required of a ural experiments in order to study historical great experimental study is not just how to interventions or interventions that could not, test an engaging hypothesis, but how to confor practical or ethical reasons, be introduced duct a test while effectively managing prac- by researchers. tical and ethical constraints. In some cases, The diversity of experimental approaches researchers address such practical and ethical reflects in part different tastes about which hurdles by searching for and taking advantage research topics are most valuable, as well of random assignments that occur naturally in as ongoing debates within the experimental the world. These natural experiments include community about how best to attack partic- instances where random lotteries determine ular problems of causal inference. Thus, it is which men are drafted for military service difficult to make broad claims about “the right (e.g., Angrist 1990), which incoming legis- way” to run experiments in many substan- lators enjoy the right to propose legislation tive domains. In many respects, experimenta- (Loewen, Koop, and Fowler 2009), or which tion in political science is still in its infancy, Pakistani Muslims obtain visas allowing them and it remains to be seen which experimen- to make the pilgrimage to Mecca (Clinging- tal designs, or combinations of designs, pro- smith, Khwaja, and Kremer 2009). The term vide the most reliable political insights. That natural experiment is sometimes defined more said, a good working knowledge of this chap- expansively to include events that happen to ter’s basic concepts and definitions can fur- some people and not others, but the hap- ther understanding of the reasons behind the penstance is not random. The adequacy of dramatic growth in the number and scope of this broader definition is debatable; however, experiments in political science, as well as the when the mechanism determining whether ways in which others are likely to evaluate and people are exposed to a potentially relevant learn from the experiments that a researcher stimulus is sufficiently random, then these develops. Experiments 23

Appendix: Introduction to the By comparing Equations (2) and (3), we see Neyman-Rubin Causal Model that the average treatment effect need not be the same as the treatment effect among the The logic underlying randomized experi- treated. ments is often explicated in terms of a nota- This framework can be used to show the tional system that has its origins in Neyman importance of random assignment. When (1923) and Rubin (1974). For each individual treatments are randomly administered, the = i,letY0 be the outcome if i is not exposed group that receives the treatment (Ti 1) has to the treatment and Y1 be the outcome if i the same expected outcome that the group = is exposed to the treatment. The treatment that does not receive the treatment (Ti 0) effect is deﬁned as would if it were treated:

τ = − . | = 1 = | = 0 . 4 i Y i1 Y i0 (1) E (Y i1 Ti ) E (Y i1 Ti ) ( )

In other words, the treatment effect is the Similarly, the group that does not receive the difference between two potential states of the treatment has the same expected outcome, world: one in which the individual receives if untreated, as the group that receives the the treatment and another in which the indi- treatment, if it were untreated: vidual does not. Extending this logic from | = = | = . a single individual to a set of individuals, E (Y i0 Ti 0) E (Y i0 Ti 1) (5) we may define the average treatment effect (ATE) as follows: Equations (4) and (5) follow from what Holland (1986) terms the independence = τ = − . 2 ATE E ( i ) E (Y i1) E (Y i0) ( ) assumption because the randomly assigned value of Ti conveys no information about the The concept of the average treatment effect potential values of Yi. Equations (2), (4), and implicitly acknowledges the fact that the (5) imply that the average treatment effect treatment effect may vary across individuals. may be written as The value of τ i may be especially large, for = τ = | = example, among those who seek out a given ATE E ( i ) E (Y i1 Ti 1) treatment. In such cases, the average treat- − | = . E (Y i0 Ti 0) (6) ment effect in the population may be quite different from the average treatment effect Because E(Y 1|T = 1) and E(Y 0|T = 0)may among those who actually receive the treat- i i i i be estimated directly from the data, Equa- ment. tion (6) suggests a solution to the problem Stated formally, the concept of the average of causal inference. To estimate an average treatment effect among the treated may be treatment effect, we simply calculate the dif- written as ference between two sample means: the aver- = τ | = 1 = | = 1 age outcome in the treatment group minus ATT E ( i Ti ) E (Y i1 Ti ) the average outcome in the control group. − | = 1 , 3 E (Y i0 Ti ) ( ) This estimate is unbiased in the sense that, on average across hypothetical replications of = where Ti 1 when a person receives a treat- the same experiment, it reveals the true aver- | = ment. To clarify the terminology, Yi1 Ti age treatment effect. 1 is the outcome resulting from the treat- Random assignment further implies that ment among those who are actually treated, independence will hold not only for Yi, but | = whereas Yi0 Ti 1 is the outcome that would also for any variable Xi that might be mea- have been observed in the absence of treat- sured prior to the administration of the treatment among those who are actually treated. ment. For example, subjects’ demographic 24 James N. Druckman, Donald P. Green, James H. Kuklinski, and Arthur Lupia attributes or their scores on a pre-test are pre- assume away spillovers (or multiple forms of sumably independent of randomly assigned the treatment) from one experimental subject treatment groups. Thus, one expects the aver- to another. age value of Xi in the treatment group to be the same as in the control group; indeed, the Noncompliance entire distribution of Xi is expected to be the same across experimental groups. This prop- Sometimes only a subset of those who are erty is known as covariate balance. It is possible assigned to the treatment group is actually to gauge the degree of balance empirically by treated, or a portion of the control group comparing the sample averages for the treat- receives the treatment. When those who get ment and control groups. the treatment differ from those who are The preceding discussion of causal effects assigned to receive it, an experiment con- skipped over two further assumptions that fronts a problem of noncompliance. In exper- play a subtle but important role in experimental studies of get-out-the-vote canvass- imental analysis. The first is the idea of ing, for example, noncompliance occurs when an exclusion restriction. Embedded in Equa- some subjects who were assigned to the tion (1) is the idea that outcomes vary as a treatment group remain untreated because function of receiving the treatment per se. they are not reached (see Gerber et al. It is assumed that assignment to the treat- 2010). ment group only affects outcomes insofar as How experimenters approach the problem subjects receive the treatment. Part of the of noncompliance depends on their objec- rationale for using blinded placebo groups tives. Those who want to gauge the effec- in experimental design is the concern that tiveness of an outreach program may be con- subjects’ knowledge of their experimental tent to estimate the intent-to-treat effect, that assignment might affect their outcomes. The is, the effect of being randomly assigned to same may be said for double-blind proce- the treatment. The intent-to-treat effect is dures: when those who implement experi- essentially a blend of two aspects of the exper- ments are unaware of subjects’ experimen- imental intervention: the rate at which the tal assignments, they cannot intentionally or assigned treatment is actually delivered to inadvertently alter their measurement of the subjects and the effect it has on those who dependent variable. receive it. Some experimenters are primar- A second assumption is known as the sta- ily interested in the latter. Their aim is to ble unit treatment value assumption (SUTVA). measure the effects of the treatment on com- In the notation used previously, expectations pliers, people who receive the treatment if | = such as E(Yi1 Ti ti) are all written as if the and only if they are assigned to the treatment expected value of the treatment outcome vari- group. able Yi1 for unit i only depends on whether the When there is noncompliance, a subject’s unit gets the treatment (whether ti equals one group assignment, Zi, is not equivalent to or zero). A more complete notation would Ti. Define a subset of the population, called allow for the consequences of treatments T1 “compliers,” who get the treatment if and through Tn administered to other units. It only if they are assigned to the treatment. = is conceivable that experimental outcomes Compliers are subjects for whom Ti 1 when = = = might depend on the values of t1, t2,...,ti–1, Zi 1 and for whom Ti 0 when Zi 0.Note ti+1,...,tn as well as the value of ti: that whether a subject is a complier is a function of both a subject’s characteristics and the | = , = ,..., = , particular features of the experiment; it is not E(Y i1 T1 t1 T2 t2 Ti−1 ti−1 = , = ,..., = . a fixed attribute of a subject. Ti ti Ti+1 ti+1 Tn tn) When treatments are administered exactly = , ∀ according to plan (Zi Ti i ), the average By ignoring the assignments to all other units causal effect of a randomly assigned treatment | = when we write this as E(Yi1 Ti ti), we can be estimated simply by comparing mean Experiments 25 treatment group outcomes and mean control Knowledge and Attitudes.” American Politics group outcomes. What can be learned about Research 37: 275–300. treatment effects when there is noncompli- Angrist, Joshua D. 1990. “Lifetime Earnings and ance? Angrist et al. (1996) present a set of the Vietnam Era Draft Lottery: Evidence from Social Security Administrative Records.” Amer- sufficient conditions for estimating the aver- 80 313 36 age treatment effect among the subgroup of ican Economic Review : – . Angrist, Joshua D., Guido W. Imbens, and Don- subjects who are compliers. Here we present ald B. Rubin. 1996. “Identification of Causal a description of the assumptions for esti- Effects Using Instrumental Variables.” Journal mating the average treatment effect for the of the American Statistical Association 91: 444–55. compliers. Baron, Reuben M., and David A. Kenny. 1986. To estimate the average treatment effect “The Moderator-Mediator Variable Distinc- among compliers, we must assume that tion in Social Psychological Research: Concep- assignment Z is random. We must also make tual, Strategic, and Statistical Considerations.” four additional assumptions: the exclusion Journal of Personality and Social Psychology 51: 1173 82 restriction, SUTVA, monotonicity, and a – . Borenstein, Michael, Larry V. Hedges, Julian P. nonzero causal effect of the random assign- 2009 ment. The exclusion restriction implies that T. Higgins, and Hannah R. Rothstein. . Introduction to Meta-Analysis. London: Wiley. the outcome for a subject is a function of Butler, Daniel M., and David W. Nickerson. the treatment they receive but is not other- 2009. “How Much Does Constituency Opin- wise influenced by their assignment to the ion Affect Legislators’ Votes? Results from a treatment group. SUTVA implies that a sub- Field Experiment.” Unpublished manuscript, ject’s outcomes depend only on the subject’s Institution for Social and Policy Studies at Yale own treatment and treatment assignment and University. not on the treatments assigned or received Campbell, Donald T. 1957. “Factors Relevant to by any other subjects. Monotonicity means the Validity of Experiments in Social Settings.” 54 297 312 that there are no defiers, that is, no sub- Psychological Bulletin : – . Clingingsmith, David, Asim Ijaz Khwaja, and jects who would receive the treatment if 2009 assigned to the control group and who would Michael Kremer. . “Estimating the Impact of the Hajj: Religion and Tolerance in Islam’s not receive the treatment if assigned to the Global Gathering.” Quarterly Journal of Eco- treatment group. The final assumption is nomics 124: 1133–1170. that the random assignment has some effect Druckman, Daniel. 1994. “Determinants of Com- on the probability of receiving the treat- promising Behavior in Negotiation: A Meta- ment. With these assumptions in place, the Analysis.” Journal of Conflict Resolution 38: 507– researcher may estimate the average treat- 56. ment effect among compliers in a manner Druckman, James N. 2003. “The Power of Televi- that will be increasingly accurate as the num- sion Images: The First Kennedy-Nixon Debate 65 559 71 ber of observations in the study increases. Revisited.” Journal of Politics : – . 2004 Thus, although the problem of experimental Druckman, James N. . “Political Preference crossover constrains a researcher’s ability to Formation: Competition, Deliberation, and the (Ir)relevance of Framing Effects.” American draw inferences about the average treatment Political Science Review 98: 671–86. effect among the entire population, accu- Eldersveld, Samuel J. 1956. “Experimental Pro- rate inferences can often be obtained with paganda Techniques and Voting Behav- regard to the average treatment effect among ior.” American Political Science Review 50: compliers. 154–65. Gerber, Alan S., and Donald P. Green. 2000.“The Effects of Canvassing, Telephone Calls, and References Direct Mail on Voter Turnout.” American Polit- ical Science Review 94: 653–63. Albertson, Bethany, and Adria Lawrence. 2009. Gerber, Alan S., Donald P. Green, Edward H. “After the Credits Roll: The Long-Term Kaplan, and Holger L. Kern. 2010. “Baseline, Effects of Educational Television on Public Placebo, and Treatment: Efficient Estimation 26 James N. Druckman, Donald P. Green, James H. Kuklinski, and Arthur Lupia

for Three-Group Experiments.” Political Anal- Mullainathan, Sendhil, Ebonya Washington, and ysis 18: 297–315. Julia R. Azari. 2010. “The Impact of Electoral Guala, Francesco. 2005. The Methodology of Exper- Debate on Public Opinions: An Experimental imental Economics. New York: Cambridge Uni- Investigation of the 2005 New York City May- versity Press. oral Election.” In Political Representation, eds. Holland, Paul W. 1986. “Statistics and Causal Ian Shapiro, Susan Stokes, Elizabeth Wood, Inference.” Journal of the American Statistical and Alexander S. Kirshner. New York: Cam- Association 81: 945–60. bridge University Press, 324–42. Hyde, Susan D. 2009. “The Causes and Con- Neyman, Jerzy. 1923. “On the Application of sequences of Internationally Monitored Elec- Probability Theory to Agricultural Experi- tions.” Unpublished manuscript, Yale Univer- ments: Essay on Principles.” Statistical Science sity. 5: 465–80. Iyengar, Shanto, and Donald R. Kinder. 1987. Nickerson, David W. 2008. “Is Voting Con- News That Matters: Television and American tagious?: Evidence from Two Field Experi- Opinion. Chicago: The University of Chicago ments.” American Political Science Review 102: Press. 49–57. Iyengar, Shanto, Mark D. Peters, and Donald Ostrom, Elinor, James Walker, and Roy Gardner. R. Kinder. 1982. “Experimental Demonstra- 1992. “Covenants with and without a Sword: tions of the ‘Not-So-Minimal’ Consequences Self-Governance Is Possible.” American Political of Television News Programs.” American Polit- Science Review 86: 404–17. ical Science Review 76: 848–58. Panagopoulos, Costas, and Donald P. Green. Kuklinski, James H., Michael D. Cobb, and Martin 2008. “Field Experiments Testing the Impact Gilens. 1997. “Racial Attitudes and the New of Radio Advertisements on Electoral Compe- South.” Journal of Politics 59: 323–49. tition.” American Journal of Political Science 52: Lau, Richard R., Lee Sigelman, Caroline Held- 156–68. man, and Paul Babbitt. 1999. “The Effects Rubin, Donald B. 1974. “Estimating Causal of Negative Political Advertisements: A Meta- Effects of Treatments in Randomized and Analytic Review.” American Political Science Nonrandomized Studies.” Journal of Educa- Review 93: 851–75. tional Psychology 66: 688–701. Loewen, Peter John, Royce Koop, and James Shadish, William R., Thomas D. Cook, and Don- H. Fowler. 2009. “The Power to Propose: A ald T. Campbell. 2002. Experimental and Quasi- Natural Experiment in Politics.” Unpublished Experimental Designs for Generalized Causal paper, University of British Columbia. Inference. Boston: Houghton Mifﬂin. Lyall, Jason. 2009. “Does Indiscriminant Violence Sniderman, Paul M., and Thomas Piazza. 1995. Incite Insurgent Attacks?” Journal of Conﬂict The Scar of Race. Cambridge, MA: Harvard Uni- Resolution 53: 331–62. versity Press. McKelvey, Richard D., and Thomas R. Palfrey. Tomz, Michael. 2007. “Domestic Audience Costs 1992. “An Experimental Study of the Cen- in International Relations: An Experimental tipede Game.” Econometrica 4: 803–36. Approach.” International Organization 61: 821– Morton, Rebecca B., and Kenneth C. Williams. 40. 2010. Experimental Political Science and the Study Wantchekon, Leonard. 2003. “Clientelism and of Causality: From Nature to the Lab. New York: Voting Behavior.” World Politics 55: 399– Cambridge University Press. 422. CHAPTER 3

Internal and External Validity

Rose McDermott

One of the challenges in conducting interdis- depends more on the primary function of ciplinary work, or in attempting to commu- the experiment. Because both internal and nicate across disciplinary boundaries, relates external validity remain important in assess- to the implicit norms that infuse different ing the quality, accuracy, and utility of any fields. Much like trying to speak across cul- given experimental design, it facilitates opti- tures, it often becomes frustrating to trans- mal experimental design to concentrate on late or make explicit differing assumptions attempting to maximize both, but the nature underlying appropriate inferential methods of the enterprise often requires explicit con- and strategies. Status differentials often exac- sideration of the trade-offs between them. erbate these divergences, privileging one set The purpose of an experiment informs of disciplinary norms over another, so that the degree to which emphasis should be decisions about ideal methods do not always placed on internal versus external validity. rest entirely on the appropriateness of a par- In some cases, as when studying a univer- ticular technique for a given project. sal human experience such as vision, col- Such differences clearly affect the imple- lege students are unlikely to significantly mentation of experimental methodology differ from the broader population on the across the fields of psychology, economics, dimension under investigation. Therefore, and political science. One of the areas in the additional external validity that would be which these biases inflict misunderstanding provided by replication across time and pop- surrounds issues related to internal and exter- ulation will not be as important. In other cir- nal validity. In political science, concerns with cumstances, such as a study of the effect of external validity often border on the mono- testosterone on decision making in combat, maniacal, leading to the neglect, if not the external validity depends on finding partic- complete dismissal, of attention to the impor- ipants in combat; otherwise, the entire pur- tant issues involved in internal validity. In pose of the study becomes vitiated. Of course, psychology, the reverse emphasis predom- the primary purpose of such a study would inates. In behavioral economics, the focus not aim for external validity, but rather for

27 28 Rose McDermott a deeper understanding of the effects of the her findings result from experimental manip- endocrine system on social relationships. ulations and even if he or she still remains Recognition of the methodological goals uncertain as to how the mechanism might promoted by internal and external validity work in various settings or across diverse indi- remains critical to the enterprise of achiev- viduals. Shadish, Cook, and Campbell (2002) ing robust experimental findings. Perhaps are careful to note that internally valid find- the best way to conceptualize the balance ings remain discrete to the specific experi- between internal and external validity in mental context in which they are explored; experimental design is to think about them generalization of any uncovered causal phe- in a two-step temporal sequence. Internal nomena then depends on extensions to other validity comes first, both sequentially and populations, contexts, and situations that practically. Without first establishing inter- involve attempts to achieve external validity. nal validity, it remains unclear what process Internal validity remains intrinsically tied should be explored in the real world. An to experimental, as opposed to mundane, experimenter has to know that the conclu- realism (Aronson et al. 1990; McDermott sions result from the manipulations imposed 2002). To the extent that subjects become before trying to extrapolate those findings psychologically engaged in the process they into other contexts. External validity follows, confront, internal validity intensifies. Simi- as replications across time and populations larly, internal validity diminishes in the face seek to delineate the extent to which these of subject disengagement, just as one might conclusions can generalize. expect any action that would distract a sub- This chapter proceeds in four parts. Sepa- ject would rob the study of its ability to specify rate discussions of internal and external valid- and consolidate the causal factor of interest. ity encompass Sections 1 and 2.Section3 If subjects approach a task with skepticism or notes the trade-offs in value and practical detachment, then genuine responses fade and logistics between the two. Section 4 con- strategic incentives come to the fore. This cludes with a meditation on future prospects raises the possibility that measures obtained for improving validity. do not accurately reflect the process being manipulated, but rather manifest a different underlying construct altogether. It does not 1. Internal Validity matter whether the experimental environment does not overtly mimic the real-world Campbell (1957) considered an experiment setting as long as the subject experiences the internally valid if the experimenter finds a relevant forces that the investigator seeks to significant difference between the treatment elicit. Because of this, the internal experience and control conditions. These differences of the experiment for the subject need not are then assumed to provide a meaningful necessarily reflect outside appearances. reflection of the causal processes at play. The success of the experiment depends As long as no reason exists to assume that on the subject taking the task seriously, and some extraneous mediating factor systemati- experimenters can foster such engagement to cally influenced subjects’ responses, observers the degree that they can create and establish a can attribute changes in the dependent vari- situation that forces psychological investment able to systematic manipulations across the on the part of subjects. For example, if an independent variables. From this perspective, experimenter wants to study, say, processes of internal validity is enhanced by experiments cooperation, then it does not matter whether that are well designed, carefully controlled, the subject would be unlikely to run across and meticulously measured so that alternative the actual partners presented, as long as he or explanations for the phenomena under con- she responds to other subjects just as he or she sideration can be excluded. In other words, would to any other potential ally. Similarly, internal validity refers to the extent to which it should not matter in a study of aggression an experimenter can be confident that his or that subjects are unlikely to have money taken Internal and External Validity 29 from them as a result of their behavior in a that resulted in inconsistent or inconclusive simple economic game, as long as this behav- findings. Often in experimental economics ior stimulates anger in them the way an actual these studies evolve almost as conversations theft or injustice would. The critical operative between scholars using different theoretical feature in such experimental designs revolves or methodological tools to examine their vari- around the ability of the experimenter to cre- ables of interest from differing perspectives. ate a psychological situation that realistically Many of the studies in experimental eco- elicits the dynamics under consideration. In nomics that seem to undertake endless varia- other words, internal validity equates to the tions on the theme of how people behave in manipulation of a psychological response. the ultimatum game are motivated, at least in part, by the desire of investigators to define precisely those conditions under which a par- Comparisons with Experimental Economics ticular behavior, such as fairness, will emerge, Roth (1995) described three main purposes for sustain, or dissipate. Sequences of studies can experiments in economics, and this analysis generate new hypotheses or reveal novel areas was extended and applied to political science of inquiry. Fehr’s work on altruistic punish- by Druckman et al. (2006). These goals in- ment followed such a sequence of inquiry cluded 1) extending theoretical models, which into the extent and bounds of human coop- he referred to as “speaking to theorists”; 2) eration (Fehr and Gachter¨ 2002; Fehr and data generation, which he called “search- Fischbacher 2003; de Quervain et al. 2004). ing for facts”; and 3) searching for mean- Such experimental research often points to ing or policy applications, which he described alternative explanations for enduring puzzles, as “whispering in the ears of princes” (22). as when neuroeconomists began to explore Importantly, each function requires slightly potential biological and genetic exigencies different foci and may engender greater con- in motivating particular economic behavior, cern with one type of validity over another. including risk (Sokol-Hessner et al. 2009). In speaking to theorists, at least in experi- Similar progress can be seen in psychology mental economics, the focus typically re- within the dialogue that emerged following volves around providing an experimental test the original studies of judgmental heuristics of a formal model. In this context, economists and biases conducted by Tversky and Kah- tend to secure internal validity within the neman (1974). Gigerenzer (1996) and others experimental paradigm itself. In such cases, a began to challenge the assumption that biases formal mathematical model will be generated, represented mistakes in the human inferential and its predictions will be tested in an experi- process, and instead demonstrated that when mental setting to see how well or how closely such dynamics were experimentally tested in actual human behavior conforms to the hypo- ecologically valid contexts (e.g., when people thetical expectation. The formal model may are called on to detect cheaters), ostensible then be adjusted to accommodate the find- errors in judgment evaporate. ings brought forth by the experiment. In such Finally, whispering in the ears of princes tests, focus on internal validity would remain speaks to the ways in which experimental almost exclusive because scholars remain less designs can be generated to address cen- concerned with the extent of generalization tral concerns held by policymakers and other outside the lab and more interested in the decision makers. Here the obvious empha- performance of the model. sis would lie more in the realm of external In searching for facts, the purpose of validity because results would need to speak the experiment revolves around generating to broad populations in order to be of use new data. This goal can take several forms. to policymakers. However, importantly, such Sometimes investigators are inspired by pre- studies retain no real utility to the extent vious experimental results, failure, or lacu- that they do not first measure what they nae to explore an aspect of previous stud- claim to examine. Herein lies an important ies or theory that did not make sense, or trade-off between attempting to create an 30 Rose McDermott experiment whose characteristics resemble receive the treatment. This source of invalid- situations familiar and important to many ity (nonexposure to treatment) thus remains individuals – involving perhaps choices correctable under certain assumptions (Nick- between political candidates, financial risk, erson 2005). For example, if an experimenter or choices over health care or employment does not come to your door, then the experi- benefits – and retaining control over the menter knows that he or she did not have an manipulation and measurement of complex effect on the measured outcome. Therefore, or multidimensional variables. if an effect is observed, then the experimenter must search for the cause elsewhere. If scholars prove willing to make some assumptions Threats to Internal Validity about the lack of effect on those who were Campbell and Stanley (1966) delineated nine not treated, then it becomes possible to sta- primary threats to internal validity that often tistically back out the effect of the putative lie outside the experimenter’s ability to con- causal variable on those who were treated. In trol. Their discussion remains the defini- that way, observers can see the raw difference, tive characterization of the kinds of prob- and not merely the effects of biases in validity lems that most risk confidence in attribut- within the design itself, on the people who ing changes in the dependent variable to received it. manipulations of the independent variable. This can be particularly problematic in These challenges include selection, history, medical studies if a certain drug causes pro- maturation, repeated testing, instrumenta- hibitive side effects that preclude a high per- tion, regression toward the mean, mortality, centage of people from continuing treatment. experimenter bias, and selection–maturation In such cases, subject mortality itself consti- interaction. Each threat presents a critical tutes an important dependent variable in the challenge to good experimental design. The experiment. In medical research, this issue is most important for purposes of examining sometimes discussed in terms of intent-to- validity relates to attrition – or mortality treat effects, where the normative prescrip- effects – and subject noncompliance because tion requires analyzing data subsequent to high rates of either can influence both inter- randomization regardless of a subject’s adher- nal and external validity. ence to treatment or subsequent withdrawal, Mortality or attrition effects occur when noncompliance, or deviation from the exper- subjects drop out of an experiment or are oth- imental protocol. This can bias results if erwise lost to follow-up. This only poses a the withdrawal from the experiment resulted threat to internal validity to the extent that from the treatment itself and not from some it occurs subsequent to random assignment other extraneous factor. (Kiesler, Collins, and Miller 1969). If sub- Of course, subject mortality poses a much jects drop out prior to such assignment, then more severe threat in between-subject it may constitute a threat to the external valid- designs, especially in matched or blocked ity of the experiment, but not to the inter- designs where the loss of one subject becomes nal validity of its findings. However, if sub- equivalent to the loss of two. This does not jects in one condition are dropping out of mean that a block design allows experi- an experiment at a higher rate than those in menters to drop blocks in the face of another condition, then it may be the case attrition. Drawing on examples from field that such attrition is in fact potentiated by experiments in voter mobilization campaigns, the treatment itself. This relates to the issue Nickerson (2005) demonstrates how this of intention to treat (Angrist, Imbens, and strategy produces bias, except under special Rubin 1996). When dealing with questions circumstances. of noncompliance, observers must estimate Proxy outcomes do not always constitute the weighted average of two different effects, a valid measure of the topic of interest. For where it is usual to divide the apparent effect example, in AIDS research, focusing on num- by the proportion of the people who actually bers of T cells as indicators of immune system Internal and External Validity 31 function may prove helpful for researchers, Another problem is active or passive but only remains significant for patients to the forms of noncompliance. Many standard extent that these values correlate significantly forms of analysis assume that everyone who with clinical outcomes of interest, such as receives treatment experiences similar levels quality of life, or longevity itself. In medicine, of engagement with the protocol, but this it is often the case that proxy measures that are assumption often remains faulty because non- assumed to correlate with outcome measures compliance rates can be nontrivial. Non- of concern actually represent orthogonal val- compliance raises the prospect of artificially ues; the best recent example relates to find- inducing treatment values into the estimated ings that even well-controlled blood sugar outcome effects. If this occurs, then it may not does not fully mitigate the risk of heart disease necessarily introduce systematic bias; how- among diabetics. Political discussions suffer ever, it may also not provide any useful infor- from a similar dynamic. In work on the influ- mation about the process the experimenter ence of negative advertising on voting, the seeks to understand. If subjects are not pay- important effect not only exerts an immedi- ing attention or are thinking about some- ate influence on voter choice, but also relates thing unrelated to the task at hand, then it to downstream effects, such as suppression of will remain unclear whether the manipula- overall voter turnout. In turn, scholars might tion actually exerted an effect. Null results not be so interested in this variable if they are under such conditions may represent true not focused on vote choice, but remain more negatives, implying no effect where one may concerned with the sources of public opinion exist if subjects attended to the manipulation on its own. Often researchers care about the as intended. Angrist et al. (1996) propose use reaction to the stimulus as much as they care of instrumental variables regression in order about its immediate effect. Whether ques- to circumvent this problem. Arceneaux, Ger- tions surrounding internal or external validity ber, and Green (2006) demonstrate the supe- in circumstances involving proxy, interven- riority of the instrumental variables approach ing, and intermediate variables remain prob- over a matching analysis in a large-scale voter lematic often depends on whether a scholar mobilization experiment. is interested in the immediate or downstream Instrumental variables are sometimes used effect of a variable. when there is concern that the treatment and In addition, mortality often arises as an unobserved factors that might affect the out- issue in iterated contexts not so much because come (i.e., the disturbance term) are corre- subjects die, but rather because long-term lated in some significant way. To be clear, the follow-up is often so costly and difficult that issue is not the assigned treatment, but rather interim measures are instituted as depen- the actual treatment as received and experi- dent variables to replace the real variables enced by the subject. When this occurs, the of interest. In the statistics literature, these concern arises that the treatment received by are referred to as “principal surrogates.” So, the subject is somehow related to the distur- for example, in medical experiments, blood bance term. If this occurs, then we could not sugar, rather than longevity, is treated as know the true effect regardless of the analysis the variable of interest in studies of diabetes. because the disturbance remains unobserved. Transitional voter turnout remains monoton- An instrumental variable is one that exerts its ically related to political legitimacy; however, effect through an influence on the indepen- it is not obvious that an observer gets more dent variable but has no direct effect on the purchase on such larger overall issues using dependent variable, nor is it systematically proxy variables. Such a focus hid the finding related to unobserved causes of the depen- that many blood sugar medications, which dent variable. In other words, it is only related indeed controlled blood sugar, nonetheless to the dependent variable through the medi- potentiated higher rates of cardiovascular- ating effect of the endogenous independent related deaths than uncontrolled blood variable. Gerber and Green (2000) provide an sugar. example of this with regard to voter turnout 32 Rose McDermott effects. In their study, random assignment to an effective experimental protocol and merit treatment determines whether a subject is some consideration as well. Importantly, dif- successfully canvassed, which in turn appears ferent kinds of experimental design pose to affect turnout. Assignment to the condi- greater risks in some areas than in others, tion of being canvassed, which is random, and recognizing which designs present which remains unrelated to the disturbance term. challenges can prevent inadvertent error. The assignment to the condition of being Several particularly problematic areas canvassed only influences turnout through exist. Pseudoexperimental designs, where the actual contact with voters; no backdoor paths researcher is not able to manipulate the inde- exist that would cause people to be affected, pendent variable, as well as experimental either directly or indirectly, by their assign- designs, which do not allow for the ran- ment to being canvassed or not, other than domization of subjects across condition for their contact with the canvasser. Because of practical or ethical reasons, present greater the expense involved in large experimental challenges for internal validity than more studies, researchers sometimes apply instru- controlled laboratory experiments. In addi- mental variables regression to observational tion, field experiments raise the specter data to gain traction on the problem at of subject noncompliance to a higher hand. In other words, they tend to be used degree than more restricted laboratory to try to infer causality by imagining that settings. the independent variable is near random Furthermore, certain content areas of (Sovey and Green 2011) when experimen- investigation may pose greater threats to tal studies designed to determine true causa- internal validity than other topics. Honesty tion might not be possible for one reason or in investigating socially sensitive issues, such another. as race or sex, may be compromised by the Of course, the key to success depends on subjects’ desire for positive impression man- selecting valid variables. This can be difficult; agement. They may not want to admit the however, often nature or government can extent to which they harbor or espouse views supply a useful instrument. For example, that they know others may find offensive. Miguel, Satyanath, and Sergenti (2004)use Less obtrusive measurements, such as those weather as an instrumental variable to exam- involving reaction time tests or implicit ine the relationship between economic shocks association measure, may help circumvent and civil conflict. Certainly, hurricanes, fires, this problem. Alternatively, techniques that or policy changes in large programs such do not rely on subject report, such as analyses as welfare might serve similar purposes. of brain waves or hormonal or genetic Such instrumental variables are feasible to factors, may obviate the need for subject the extent that the independent variable honesty, depending on the topic under provides consistent estimates of causal effects investigation. when the instruments are independent of the Regardless, certain realities inevitably con- disturbance term and correlated in a substan- strain the ability of an investigator to know tial way with the endogenous independent whether subjects are sufficiently engaged in variable of interest. Successfully identifying an experimental task so as to justify reliance such an instrument can help triangulate on on the data generated by them. Any system- the relationships and treatments of interest, atic restrictions in the performance of sub- but often finding such instruments can jects can potentially contaminate the internal prove challenging. Alternative strategies, validity of the results obtained. The follow- such as intent-to-treat effects, for dealing ing discussion highlights some of the ways with problems where the treatment and the in which subjects can, intentionally or other- outcome may be correlated exist, as discussed wise, impede internal validity in experimental previously. tests. Some of these categories overlap with Other unrelated concerns can also com- Campbell and Stanley (1966), whereas others promise prospects for the development of introduce additional concerns. Internal and External Validity 33

Good experimentalists should strive for Problems affecting prospects for internal the goal of trying to design an experiment validity arise when anything interferes with from the subject’s perspective, with an eye the ability to attribute changes in the depen- toward understanding what the person will dent variables to manipulations in the inde- see, hear, and experience in the setting cre- pendent variable. Sometimes, but not always, ated, and not with the singular purpose of this can occur if subjects intentionally change achieving the fastest, most efficient way to their behavior to achieve a particular effect. collect the data they need. This becomes par- Perhaps they want to give the experimenter ticularly important in behaviorally oriented what they believe that she wants, although tasks. Subject involvement is not an altruis- they can easily be wrong about the experi- tic goal, but rather one that should be moti- menter’s goals. Or they may want to inten- vated entirely by enlightened self-interest. tionally thwart the investigator’s purpose. To the extent that the experimentalist can Intentional subject manipulation of outcomes create an engaging, involving, and interest- can even artificially induce treatment values ing task environment, many of the follow- into the experiment. ing issues may be ameliorated. True, most Most of these concerns related to sub- subjects remain motivated to participate in jects strategically trying to manipulate their experiments because of the incentives offered, responses for reasons having nothing to do whether money, credit, or some other bene- with the experimental treatment can be obvi- fit. But it behooves any conscientious exper- ated by randomization, except to the extent imenter to keep in mind that many, if not that such attempts at deceiving the experi- most, subjects will want to figure out what menter are either systematic or widespread the experiment is “really” about and what the in effect. Sometimes such efforts only affect experimenter wants so that they can comply, the inferential process to the extent that sub- resist, or simply satisfy their curiosity. The jects are able to successfully guess the investi- job of the experimenter is to make the task gator’s hypotheses. Often it does not matter sufficiently absorbing that the subject finds it whether the subject knows the purpose of an more interesting to concentrate on the task experiment; observers want to test straight- at hand than to try to game the experiment. forward conscious processes. However, if One of the most effective and efficient ways knowledge of the experimenter’s hypothesis to achieve this goal is to engage in preexper- encourages subjects to consciously attempt to imental pilot testing to see which stimuli or override their more natural instincts within tasks elicit the most subject engagement. Par- the confines of the experiment, then the pos- ticularly entrepreneurial experimenters can sibility of systematic interference with inter- corral friends and relatives to test alterna- nal validity arises. Obviously, this is most tive scenarios to enhance subjects’ psycho- problematic under conditions that require logical involvement in a study. If an experi- experimental deception. Because more than mentalist invokes this strategy of pilot test- eighty percent of psychology experiments in ing, then it becomes absolutely imperative to top journals use some kind of deception and solicit information from subjects in post-test almost no experiments in economics journals debriefing in order to learn how they per- do, the issue of subjects guessing an experi- ceived the situation, how they understood menter’s hypothesis remains more problem- their task, what systematic biases or misin- atic in some disciplines than in others. terpretations might have emerged from the If this is a concern, then one way to poten- experimenter’s perspective, and how the pro- tially control for such effects is to probe sub- cedure might be improved. Asking pilot subjects after the experiment to see whether they jects what they believe might have increased guessed deception was operative or discerned their interest can prove to be a disarm- the true purpose of the experiment. Every ingly straightforward and surprisingly suc- effort should be made to keep subjects within cessful strategy for enhancing future subject the analysis, but skeptical subjects can be engagement. compared separately with more susceptible 34 Rose McDermott subjects to determine any differences in sively, on problems associated with external response. If no differences exist, then data validity. External validity refers to the gen- can be collapsed; if differences emerge, then eralizability of findings from a study, or the they might be reported, especially if they sug- extent to which conclusions can be applied gest a bias in treatment effect resulting from across different populations or situations. individual differences in acceptance of the Privileging of external validity often results protocol. If there is no deception but the sub- from a misunderstanding that generalizabil- ject knows what the experiment is about, then ity can result from, or be contained within, the problem of subjects intentionally trying a single study, as long as it is large enough to manipulate results is not eliminated; how- or broad enough. This is almost never true. ever, again, to the extent that such efforts External validity results primarily from repli- are small and random, they should be min- cation of particular experiments across diverse imized by processes of randomization across populations and different settings, using a condition. variety of methods and measures. As Aronson In addition, of course, any of these chal- et al. (1990) state succinctly, “No matter how lenges to internal validity can be exacerbated similar or dissimilar the experimental context to the extent that they occur concomitantly or is to a real-life situation, it is still only one interact in unexpected or unpredictable ways. context: we cannot know how far the results will generalize to other contexts unless we carry on an integrated program of systematic Ways to Improve replication” (77). Many ways to circumvent challenges to Some of the reason for the difference internal validity have been alluded to in in disciplinary emphasis results from diver- the course of the preceding discussion. gent purposes. Most psychologists, like many Well-designed experiments with strong con- economists, use experiments primarily to test trol, careful design, and systematic mea- theory, rather than to make generalizations surement go a long way toward alleviating of such theory to broader populations. Their these concerns. primary research goal focuses on explicat- Perhaps the single most important strating and elucidating basic operating principles egy experimenters can employ to avoid risks underlying common human behaviors, such to internal validity is to develop procedures as cooperation or discrimination, and then that optimize experimental realism for sub- distilling these processes through the crys- jects. Designing an experiment that engages talline filter of replication to delineate the subjects’ attention and curiosity will ensure boundaries of their manifestation and expres- that the dynamic processes elicited in the sion in real-world contexts. Replication that experimental condition mimic those that are establishes external validity can, and should, evoked under similar real-world conditions. take many forms. If a genuine cause-and- No amount of time that goes into trying to effect relationship exists across variables, then develop an involving experiment from the it should emerge over time, within different perspective of the subject will be wasted contexts, using various methods of measure- in terms of maximizing the resemblance ment, and across population groups, or the between the psychological experience in the boundaries of their operation should become experiment and that of the unique real-world defined (Smith and Mackie 1995). Aronson environment they both inhabit and create. et al. describe this process best: Bringing the research out of the laboratory does not necessarily make it more generaliz- 2. External Validity able or “true”; it simply makes it different. The question of which method – “artificial” Although psychologists pay primary atten- laboratory experiments versus experiments tion to issues associated with internal validity, conducted in the real world – will provide political scientists tend to focus, almost exclu- the more generalizable results is simply the Internal and External Validity 35

wrong question. The generalizability of any trivial tasks presented to subjects offer a research finding is limited. This limitation poor analogue to the real-world experiences can be explicated only by systematically testing that individuals confront in trying to traverse the robustness of research results across differ- their daily political and social environments. ent empirical realizations of both the indepen- This characterization of a controlled labo- dent and dependent variables via systematic ratory experiment, although often accurate, replication to test the extent to which different reflects a privileging of mundane as opposed translations of abstract concepts into concrete realizations yield similar results. (Aronson to experimental realism. The benefit of such a et al. 1990, 82) stripped-down, stylized setting is that it offers the opportunity to carefully operationalize Of course, it remains important to exam- and measure the variables of interest and ine the extent to which the outcomes mea- then, through multiple tests on numerous sured in a laboratory setting find analogues populations, to begin to define the condi- in real-world contexts. External validity can tions under which generality might obtain. be examined in multiple ways, including mea- The reason it becomes so critical to uncover suring treatment effects in real-world envi- these mechanisms is because unless an inves- ronments, exploring the diverse contexts in tigator knows the underlying principles oper- which these variables emerge, investigating ating in a given dynamic, it will prove the various populations it affects, and looking simply impossible to ascertain which aspect at the way basic phenomena might change in of behavior is causing which effect within response to different situations. Some of these the context of real-world settings where factors can be explored in the context of a many other variables and interactions occur controlled laboratory setting, but some might simultaneously. be more profitably addressed in field con- One of the most dramatic examples of texts. However, experimenters should remain this process occurred in the famous Mil- aware of the trade-offs involved in the abil- gram (1974) experiment. Milgram set out to ity to control and measure carefully defined explain the compliance of ordinary Germans variables with a richer understanding of the with Nazi extermination of the Jews. Test- extent to which these factors might interact ing at Yale was designed to provide a control with other unknowns outside the laboratory condition for later comparison with German setting. and Japanese subjects. Prior to the experi- Although the concerns regarding exter- ment, every psychiatrist consulted predicted nal validity certainly remain legitimate, it is that only the worst, most rare psychopaths important to keep it mind that they should would administer the maximum amount of only arise to the extent that sufficient prior shock. In these experiments, subjects arrived attention has been paid to ensuring that and were told they were going to participate a study embodies internal validity first. As in an experiment about “learning.” They were Aronson et al. (1990) rightly state, “Internal to play either a teacher or a learner, and they validity is, of course, the more important, for drew straws to determine their role. Unbe- if random or systematic error makes it impos- knownst to them, the straws were rigged so sible for the experimenter even to draw any that the subject was always the “teacher.” conclusions from the experiment, the ques- In this role, subjects were told to teach a tion of the generality of these conclusions series of word pairs to their “learner,” who never arises” (75). was in fact an experimental confederate. All “learner” replies were taped to ensure that every subject experienced the same condi- Threats to External Validity tions. When the learner apparently got a Most concerns that political scientists express wrong answer, subjects were told to shock regarding external validity reflect their recog- the learner, and with each sequential mistake nition of the artificial nature of the labo- to up the voltage one notch, equal to 15 volts. ratory setting. The notion here is that the At a prearranged point, the “learner” starts 36 Rose McDermott screaming, asks to be let out, says he has a mortality may compromise the representa- heart condition, and finally ceases respond- tiveness of the study population. ing. Subjects who showed hesitation were Selection bias, in terms of nonrandom encouraged to continue by the experimenter, sampling, represents another threat to exter- who said he would take responsibility. The nal validity that also threatens internal valid- vast majority of subjects went all the way to ity, as described previously. If subjects are the top of the machine, equal to 450 volts and drawn from too restrictive a sample or clearly marked “XXX.” The careful design an unrepresentative sample, then obviously of the experiment allowed Milgram to begin more replication will be required to gen- to uncover the subtle and powerful effects of eralize the results with confidence. This is obedience on behavior (for a discussion of becoming an increasing concern with the the experiment, see Dickson’s chapter in this huge increase in Internet samples, where volume). investigator knowledge and control of their Experimental realism remains more im- subject populations can become extremely portant than mundane realism in maximiz- restricted. It can be virtually impossible to ing prospects for internal validity because it is know whether the person completing the more likely to elicit the critical dynamic under Internet survey is who they say they are, much investigation; more highly stylized or abstract less whether they are attending to the tasks in experimental protocols can risk both internal any meaningful way. With unrepresentative and external validity by failing to engage sub- samples of students, it is still possible to rea- jects’ attention or interest. Creative design son about the ways in which they may not can sometimes overcome these difficulties, be representative of a larger population (e.g., even in laboratory settings, because clever by having superior abstract cognitive skills). experimentalists have found ways to simu- But with an Internet sample, even this ability late disrespect – for example, by having a to determine the ways in which one sample confederate “accidentally” bump into a sub- may differ from another becomes extremely ject rudely (Cohen and Nisbett 1994)or challenging. by simulating an injury (Darley and Latane The so-called Hawthorne effect poses 1968). another threat to external validity. This phe- Restricted subject populations can also nomenon, named after the man who precipi- limit the degree of potential generalizability tated its effect when it was first recognized, from studies, although the degree to which refers to the way in which people change their this problem poses a serious threat varies with behavior simply because they know they are the topic under investigation. Although it is being monitored (Roethlisberger and Dick- generally better to have more subjects across son 1939). Surveillance alone can change a wider demographic range, depending on behavior in ways that can influence the vari- the content of the study, it may be more able being measured. Without such moni- important to obtain more subjects, rather toring and observation, behavior, and thus than explicitly diverse ones. Common sense, results, might be quite different. Sometimes in combination with practical logistics such this effect can be desirable, such as when as costs, should guide judgment concerning observation is used to enforce compliance how best to orchestrate this balance. with protocols, reminding subjects to do a Several other threats to external validity certain thing in a particular way or at a specific exist, some of which mirror those that can time. Technology such as personal handheld compromise internal validity. Subject mortal- devices can help facilitate this process as well. ity raises a concern for external validity to the However, such self-consciousness can some- extent that such mortality takes place prior times affect outcome measures in more biased to randomization; recall that mortality subse- ways. quent to randomization compromises inter- External interference, or interference be- nal validity. Prior to randomization, subject tween units, presents a complex and nuanced Internal and External Validity 37 potential confound (see Sinclair’s chapter in investigation can make the ostensible artifi- this volume). Behavior in the real world oper- ciality worthwhile. ates differently than in more sanitized settings precisely because there is more going Ways to Improve on at any given time and, often, more is at stake in the psychic world of the subject. Given that political scientists tend to be Moreover, the same variables may not operate united in their concern for politically relevant the same way in two different situations pre- contexts (Druckman and Lupia 2006), exter- cisely because other factors may exacerbate nal validity will continue to serve as a central or ameliorate the appearance of any given focus of concern for those interested in exper- response within a particular situation. Inter- imental relevance for broader societal conference thus can occur not only as a result texts. Several strategies can help maximize the of what happens within a given experimen- potential for increasing such relevance and tal context, but also as a consequence of the broader applicability, first and foremost being way such responses can change when inter- reliance on replication across subjects, time, acting with diverse and unpredictable addi- and situation. In general, anything that multi- tional variables in real-world contexts that plies the ways in which a particular dynamic is can operate to suppress, potentiate, or other- investigated can facilitate prospects for exter- wise overwhelm the expression of the relevant nal validity. To be clear, external validity processes. occurs primarily as a function of this strategy Perhaps the biggest concern that politi- of systematic replication. Conducting a series cal scientists focus on with regard to exter- of experiments that include different popu- nal validity revolves around issues related to lations, involve different situations, and use either the artificiality or triviality of the exper- multiple measurements establishes the fun- imental situation, although it is certainly pos- damental basis of external validity. A sin- sible for the opposite criticism, that subjects gle study, regardless of how many subjects pay too much attention to stimulus materi- it encompasses or how realistic the environ- als in experiments, to be leveled as well. For ment, cannot alone justify generalization out- example, in looking at the effect of television side the population and domain in which it advertising, subjects may pay much closer was conducted. attention to such ads in the lab than they One of the most important ways to en- would in real life, where they might be much hance external validity involves increasing more likely to change channels or walk out the heterogeneity of the study populations, of the room when the ads come on. Clearly, unless one is only trying to generalize to it would be next to impossible for experi- homogenous populations, such as veterans, menters to replicate most aspects of real life republicans, or women, in which case the in a controlled way. Time constraints, cul- study of focal populations remains optimal. tural norms, and subject investment preclude Including subjects from different age groups, such mirroring (Walker 1976). However, it is sexes, races, and socioeconomic or educa- often the case that such cloning is not nec- tional statuses, for example, increases the essary in order to study a particular aspect representativeness of the sample and poten- of human behavior, and the ability to isolate tiates prospects for generalization. Again, such phenomena, and explore its dimensions, common sense should serve as a guide as to can compensate for the more constrained which populations should be studied for any environmental setting by allowing investi- given topic. Studies involving facial recog- gators to delineate the precise microfoun- nition of emotion, for example, can benefit dational mechanisms underlying particular greatly from employing subjects with focal attitudes and behaviors of interest. The ben- brain lesions because their deficits in recog- efits that can derive from locating such speci- nition can inform researchers of the pro- ficity in the operation of the variable under cesses necessary for intact processing of these 38 Rose McDermott attributes. In this case, fewer subjects with It remains important to explicitly and clearly particularly illuminating characteristics can recognize the inherent nature of the trade- provide greater leverage than a larger number offs between them. of less informative ones. Two principal trade-offs exist between Increasing the diversity of circumstances internal and external validity. First, the bal- or situations in which a particular phe- ance between these types of validity clearly nomenon is investigated can also heighten reflects a difference in value. Attention to external validity. Exploring a particular pro- internal validity optimizes the ability of cess, such as cooperation, in a variety of set- an investigator to achieve confidence that tings can prove particularly helpful for dis- changes in the dependent variables truly covering contextual boundaries on particular resulted from the manipulation of the inde- processes, locating environmental cues that pendent variable. In other words, method- trigger such dynamics, and illustrating the ological and theoretical clarity emerge from particular dimensions of its operation. careful and conscientious documentation of Using multiple measures, or multiple types variables and measures. The experimenter of measures, as well as doing everything possi- can rest assured that the processes investi- ble to improve the quality of measures gated returned the results produced; in other employed, can greatly enhance external valid- words, investigators can believe that they ity. This might involve finding multiple studied what they intended and that any effect dependent measures to assess downstream was produced by the manipulated purported effects, either over time or across space cause. (Green and Gerber 2002). In many medi- In contrast, concentration on external cal studies, intervening variables are often validity by expanding subject size or rep- used as proxies to determine intermediary resentativeness can increase confidence in effects if the critical outcome variable does generalizability, but only to the extent that not occur frequently or takes a long time to extraneous or confounding hypotheses can be manifest, although some problems associated eliminated or excluded from contention. If with this technique were noted previously as sufficient attention has gone into securing well. Proper and careful definition of the vari- details assuring internal validity, then the ables under consideration – both those explic- window between the laboratory and the out- itly being studied and measured and those side world can become more transparent. expected to affect these variables differen- Second, trade-offs between internal and tially in a real-world setting – remains cru- external validity exist in practical terms as cial to isolating the conditions under which well. Internal validity can take time and particular processes are predicted to occur. attention to detail in operationalizing variables, comparing measures, and contrasting the implications of various hypotheses. Most 3. Balance between Internal of this effort occurs prior to actually conduct- and External Validity ing the experiment. Working toward enhancing external validity requires more enduring Obviously, it is best to strive to maximize effort because, by definition, the effort must both internal and external validity. But some- sustain beyond a single study and encompass times this is not possible within the practical a sequence of experiments. In addition, secur- and logistical constraints of a given experi- ing additional populations or venues may take mental paradigm. Maximizing internal valid- time after the experiment is designed. ity may diminish the ability to extrapolate the Such trade-offs between internal and findings to situations and populations outside external validity emerge inevitably over the those specifically studied. Privileging external course of experimental work. Depending on validity often neglects important aspects of the topic, a given experimenter may concen- internal experimental control so that the true trate on maximizing one concern over the cause of reported findings remains unclear. other within the context of any particular Internal and External Validity 39 study. But awareness of the requisite trade- tion, can also be employed for this pur- offs in value and practice can be important in pose. Clearly, the most common technology balancing the intent and implementation of at the moment in both psychology and neu- any given experiment. roscience involves functional magnetic resonance imagery to locate particular geogra- phies in the brain. Blood and saliva tests for 4. Future Work hormonal or genetic analysis can also provide useful and effective, albeit more intru- Striving to maximize internal and exter- sive, indirect measures of human function- nal validity in every experiment remains a ing. I have leveraged these measures in my laudable goal, even if optimizing both can own work exploring the genetic basis of sometimes prove unrealistic in any particular aggression (McDermott et al. 2009) and sex study. Certain things can be done to improve differences in aggression (McDermott and and increase validity in general, depending on Cowden 2001); in this way, such biological the substance of investigation. measures can be used to explore some factors Particular methodological techniques or underlying conflict. These technologies offer technologies may make it easier to enhance the advantage of circumventing notoriously validity in part by making measures less unreliable or deceptive self-report to obtain obtrusive, allowing fewer channels for responses that can be compared either within subjects to consciously accommodate or or between subjects to determine potential resist experimental manipulation. These can sources for particular attitudes and behav- become particularly useful when studying iors of interest. Such efforts can enhance socially sensitive topics such as race or sex. prospects for internal validity and increase the Such strategies include implicit association ease and speed with which external validity tests and other implicit measures of atten- can be achieved as well. tion (see Lodge and Taber’s chapter in this volume). These instruments can allow investigators to obtain responses from subjects without the subjects necessarily being aware References of either the topic or being able to control their reactions consciously. Similarly, reac- Angrist, Joshua, Guido Imbens, and Donald 1996 tion time tests can also provide measures of Rubin. . “Identification of Causal Effects speed and accuracy of association in ways Using Instrumental Variables.” Journal of the American Statistical Association 91: 444–55. that can bypass subjects’ conscious attempts Arceneaux, Kevin, Alan Gerber, and Donald to deceive or manipulate observers. Even Green. 2006. “Comparing Experimental and though some subjects may be able to con- Matching Methods Using a Large-Scale Voter sciously slow their response to certain stimuli Mobilization Experiment.” Political Analysis 14: (and it is unclear why they would choose to 37–62. do so), it may be impossible for them to per- Aronson, Elliott, Phoebe C. Ellsworth, J. Mer- form more rapidly than their inherent capac- rill Carlsmith, and Marti Hope Gonzales. 1990. ity allows. Methods of Research in Social Psychology. 2nd ed. Of course, other forms of subliminal tests New York: McGraw-Hill. 1957 exist, although they tend to be less widely Campbell, Donald T. . “Factors Relevant to known or used in political science than in Validity of Experiments in Social Settings.” Psychological Bulletin 54: 297–312. other fields such as psychology or neuro- Campbell, Donald T., and Julian C. Stanley. 1966. science. Neuroscientists, for example, often Experimental and Quasi-Experimental Designs for use eye tracking devices in order to follow Research. Chicago: Rand McNally. what a subject observes without having to Cohen, Dov, and Richard E. Nisbett. 1994. “Self- rely on less accurate self-report measures. Protection and the Culture of Honor: Explain- Physiological measures, including heart rate, ing Southern Violence.” Personality and Social galvanic skin response, or eye blink func- Psychology Bulletin 20: 551–67. 40 Rose McDermott

Darley, John M., and Bibb Latane. 1968. McDermott, Rose, Dustin Tingley, Jonathan A. “Bystander Intervention in Emergencies: Dif- Cowden, Giovanni Frazzetto, and Dominic fusion of Responsibility.” Journal of Personality D. P. Johnson. 2009. “Monoamine Oxidase and Social Psychology 8: 377–83. A Gene (MAOA) Predicts Behavioral Aggres- de Quervain, Dominique J.-F., Urs Fischbacher, sion Following Provocation.” Proceedingsofthe Valerie Treyer, Melanie Schellhammer, Ulrich National Academy of Sciences 106: 2118–23. Schnyder, Alfred Buck, and Ernst Fehr. 2004. Miguel, Edward, Shanker Satyanath, and Ernest “The Neural Basis of Altruistic Punishment.” Sergenti. 2004. “Economic Shocks and Civil Science 305 (5688): 1254–58. Conflict: An Instrumental Variable Approach.” Druckman, James N., Donald P. Green, James Journal of Political Economy 112: 725–53. H. Kuklinski, and Arthur Lupia. 2006.“The Milgram, Stanley. 1974. Obedience to Authority. Growth and Development of Experimental New York: Harper & Row. Research in Political Science.” American Politi- Nickerson, David W. 2005. “Scalable Protocols cal Science Review 100: 627–35. Offer Efficient Design for Field Experiments.” Druckman, James N., and Arthur Lupia. 2006. Political Analysis 13: 233–52. “Mind, Will and Choice.” In The Oxford Hand- Roethlisberger, Fritz J., and William J. Dickson. book on Contextual Political Analysis,eds.Charles 1939. Management and the Worker. Cambridge, Tilly and Robert E. Goodin. Oxford: Oxford MA: Harvard University Press. University Press, 97–113. Roth, Alvin E. 1995. “Introduction to Experi- Fehr, Ernst, and Urs Fischbacher. 2003.“The mental Economics.” In Handbook of Experimen- Nature of Human Altruism.” Nature 425: 785– tal Economics, eds. John H. Kagel and Alvin 91. E. Roth. Princeton, NJ: Princeton University Fehr, Ernst, and Simon Gachter.¨ 2002. “Altruistic Press, 3–110. Punishment in Humans.” Nature 415: 137–40. Shadish, William R., Thomas D. Cook, and Gerber, Alan, and Donald P. Green. 2000.“The Donald T. Campbell. 2002. Experimental Effects of Canvassing, Telephone Calls, and and Quasi-Experimental Designs for Generalized Direct Mail on Voter Turnout: A Field Exper- Causal Inference. Boston: Houghton-Mifflin. iment.” American Political Science Review 94: Smith, Eliot R., and Diane M. Mackie. 1995. Social 653–63. Psychology. New York: Worth. Gigerenzer, Gerd. 1996. “On Narrow Norms and Sokol-Hessner, Peter, Ming Hsu, Nina G. Curley, Vague Heuristics: A Reply to Kahneman and Mauricio R. Delgado, Colin F. Camerer, and Tversky.” Psychological Review 103: 592–96. Elizabeth A. Phelps. 2009. “Thinking Like a Green, Donald P., and Alan Gerber. 2002.“The Trader Selectively Reduces Individuals’ Loss Downstream Benefits of Experimentation.” Aversion.” Proceedings of the National Academy of Political Analysis 10: 394–402. Sciences 106: 5035–40. Kiesler, Charles A., Barry E. Collins, and Norman Sovey, Allison J., and Donald P. Green. 2011. Miller. 1969. Attitude Change: A Critical Analysis “Instrumental Variables Estimation in Political of Theoretical Approaches. New York: Wiley. Science: A Reader’s Guide.” American Journal McDermott, Rose. 2002. “Experimental Method- of Political Science 55: 188–200. ology in Political Science.” Political Analysis 10: Tversky, Amos, and Daniel Kahneman. 1974. 325–42. “Judgment under Uncertainty: Heuristics and McDermott, Rose, and Jonathan A. Cowden. Biases.” Science 185: 1124–31. 2001. “The Effects of Uncertainty and Sex in a Walker, T. 1976. “Microanalytic Approaches to Simulated Crisis Game.” International Interac- Political Decision Making.” American Behav- tions 27: 353–80. ioral Science 20: 93–110. CHAPTER 4

Students as Experimental Participants

A Defense of the “Narrow Data Base”

James N. Druckman and Cindy D. Kam

An experiment entails randomly assigning when student subjects are likely to constrain participants to various conditions or manip- experimental inferences. We show that such ulations. Given common consent require- situations are relatively limited; any conve- ments, this means experimenters need to nience sample poses a problem only when recruit participants who, in essence, agree to the size of an experimental treatment effect be manipulated. The ensuing practical and depends on a characteristic on which the con- ethical challenges of subject recruitment have venience sample has virtually no variance. led many researchers to rely on convenience Third, we brieﬂy survey empirical evidence samples of college students. For political sci- that provides guidance on when researchers entists who put particular emphasis on gen- should be particularly attuned to taking steps eralizability, the use of student participants to ensure appropriate generalizability from often constitutes a critical, and according to student subjects. We conclude with a discus- some reviewers, fatal problem for experimen- sion of the practical implications of our ﬁnd- tal studies. ings. In short, we argue that student subjects In this chapter, we investigate the extent to are not an inherent problem to experimen- which using students as experimental partic- tal research; moreover, the burden of proof – ipants creates problems for causal inference. articulating and demonstrating that student First, we discuss the impact of student sub- subjects pose an inferential problem – should jects on a study’s internal and external valid- lie with critics rather than experimenters. ity. In contrast to common claims, we argue that student subjects do not intrinsically pose a problem for a study’s external validity. Sec- 1. The “Problem” of Using ond, we use simulations to identify situations Student Subjects

Although internal validity may be the sine We thank Kevin Arceneaux, Don Green, Jim Kuklinski, Peter Loewen, and Diana Mutz for helpful advice, and qua non of experiments, most researchers Samara Klar and Thomas Leeper for research assistance. use experiments to make generalizable causal

41 42 James N. Druckman and Cindy D. Kam inferences (Shadish, Cook, and Campbell student subjects; for example, Kam et al. 2002, 18–20). For example, suppose one report that from 1990 through 2006, one implements a laboratory study with students fourth of experimental articles in general and finds a causal connection between the political science journals relied on student experimental treatment (e.g., a media story subjects, whereas more than seventy percent about welfare) and an outcome of inter- did so in more specialized journals (Kam et al. est (e.g., support for welfare). An obvious 2007, 419–20; see also Druckman et al. 2006). question is whether the relationship found Are the results from these studies of ques- in the study exists within a heterogeneous tionable validity? Second, there are practi- population, in various contexts (e.g., a large cal issues. A common rationale for moving media marketplace), and over time. This is away from laboratory studies – in which stu- an issue of external validity, which refers dent subjects are relatively common – to to the extent to which the “causal rela- survey and/or field experiments is that these tionship holds over variations in persons, latter venues facilitate using nonstudent par- settings, treatments [and timing], and out- ticipants. When evaluating the pros and cons comes” (Shadish et al. 2002, 83). McDermott of laboratory versus survey or field exper- (2002) explains,“Externalvalidity...tend[s] iments, should substantial weight be given to preoccupy critics of experiments. This near to whether participants are students? Sim- obsession ...tend[s] to be used to dismiss ilarly, those implementing lab experiments experiments” (334). have increasingly put forth efforts (and paid A point of particular concern involves gen- costs) to recruit nonstudent subjects (e.g., eralization from the sample of experimental Lau and Redlawsk 2006, 65–66;Kam2007). participants – especially when, as is often Are these costs worthwhile? To address these the case, the sample consists of students – questions, we next turn to a broader discus- to a larger population of interest. Indeed, sion of what external validity demands. this was the focus of Sears’ (1986)widely cited article, “College Sophomores in the Dimensions of External Validity Laboratory: Influences of a Narrow Data base on Social Psychology’s View of Human To assess the external validity or generaliz- Nature.”1 Many political scientists simply ability of a causal inference, one must con- assume that “a student sample lacks external sider from what we are generalizing and to generalizability” (Kam, Wilking, and Zech- what we hope to generalize. When it comes to meister 2007, 421). Gerber and Green (2008) “from what,” a critical, albeit often neglected, explain, “If one seeks to understand how point is that external validity is best under- the general public responds to social cues or stood as being assessed over a range of stud- political communication, the external valid- ies on a single topic (McDermott 2002, 335). ity of lab studies of undergraduates has Assessment of any single study, regardless of inspired skepticism” (Sears 1986; Benz and the nature of its participants, must be done in Meier 2008, 358). In short, social scientists light of the larger research agenda to which it in general and political scientists in particular hopes to contribute.2 view student subjects as a major hindrance Moreover, when it comes to generalization to drawing inferences from experimental from a series of studies, the goal is to gen- studies. eralize across multiple dimensions. External Assessing the downside of using student subjects has particular current relevance. 2 This is consistent with a Popperian approach to cau- First, many political science experiments use sation that suggests causal hypotheses are never confirmed and evidence accumulates via multiple tests, even if these tests have limitations. Campbell (1969) 1 Through 2008,Sears’(1986) article was cited an offers a fairly extreme stance on this when he states, impressive 446 times according to the Social Sci- “Had we achieved one, there would be no need ence Citation Index. It is worth noting that Sears’ to apologize for a successful psychology of college argument is conceptual – he does not offer empirical sophomores, or even of Northwestern University evidence that student subjects create problems. coeds, or of Wistar staring white rats” (361). Students as Experimental Participants 43 validity refers to generalization not only of for their opinions about a publicly owned individuals, but also across settings/contexts, gambling casino, which was a topic of “real- times, and operationalizations. There is little world” ongoing political debate. Prior to doubt that institutional and social contexts expressing their opinions, respondents ran- play a critical role in determining political domly received no information (i.e., con- behavior and that, consequently, they can trol group) or information that emphasized moderate causal relationships. One power- either economic benefits or social costs (e.g., ful recent example comes from the politi- addiction to gambling). Druckman shows that cal communication literature; a number of the opinions of attentive respondents (i.e., experiments, using both student and non- respondents who regularly read newspaper student subjects, show that when exposed coverage of the campaign) in the economic to political communications (e.g., in a lab- information condition did not significantly oratory), individuals’ opinions often reflect differ from attentive individuals in the control the content of those communications (e.g., group.4 The noneffect likely stemmed from Kinder 1998; Chong and Druckman 2007b). the economic information – which was avail- The bulk of this work, however, ignores able outside the experiment in ongoing polit- the contextual reality that people outside the ical discussion – having already influenced all controlled study setting have choices (i.e., respondents. Another exposure to this infor- they are not captive). Arceneaux and John- mation in the experiment did not add to the son (2008) show that as soon as participants prior, pre-treatment effect. In other words, in communication experiments can choose the ostensible noneffect lacked external valid- whether to receive a communication (i.e., ity, not because of the sample but because it the captive audience constraint is removed), failed to account for the timing of the treat- results about the effects of communications ment and what had occurred prior to that time drastically change (i.e., the effects become less (also see Slothuus 2009).5 dramatic). In this case, ignoring the contex- A final dimension of external validity tual reality of choice appears to have consti- involves how concepts are employed. Find- tuted a much greater threat to external valid- ing support for a proposition means look- ity than the nature of the subjects.3 ing for different ways of administering and Timing also matters. Results from exper- operationalizing the treatment (e.g., deliver- iments implemented at one time may not ing political information via television ads, hold at other times, given the nature of world newspaper stories, interpersonal communica- events. Gaines, Kuklinski, and Quirk (2007) tions, survey question text) and operationaliz- further argue that survey experiments in paring the dependent variables (e.g., behavioral, ticular may misestimate effects due to a fail- attitudinal, physiological, implicit responses). ure to consider what happened prior to the In short, external validity does not simply study (also see Gaines and Kuklinski’s chap- refer to whether a specific study, if rerun ter in this volume). Building on this insight, on a different sample, would provide the Druckman (2009) asked survey respondents same results. It refers more generally to whether “conceptually equivalent” (Ander- 1997 21 3 A related example comes from Barabas and Jerit’s son and Bushman , ) relationships (2010) study, which compares the impact of com- can be detected across people, places, times, munications in a survey experiment against analo- and operationalizations. This introduces the gous dynamics that occurred in actual news coverage. They find that the survey experiment vastly other end of the generalizability relation- overstated the effect, particularly among certain sub- ship – that is, “equivalent” to what? For many, groups. Sniderman and Theriault (2004)andChong and Druckman (2007a) also reveal the importance of context; both studies show that prior work that limits 4 For reasons explained in his paper, Druckman (2009) competition between communications (i.e., by only also focuses on individuals more likely to have formed providing participants with a single message rather prior opinions about the casino. than a mix that is typically found in political contexts) 5 Another relevant timing issue concerns the duration likely misestimates the impact of communications on of any experimental treatment effect (see, e.g., Gaines public opinion. et al. 2007; Gerber et al. 2007). 44 James N. Druckman and Cindy D. Kam the “to what” refers to behavior as observed and 3) the goal of the study (e.g., to build outside the study, but this is not always a theory or to generalize one). the case. Experiments have different purposes; Roth (1995) identifies three nonexclu- Evaluating External Validity sive roles that experiments can play: search for facts, speaking to theorists,orwhispering in The next question is how to evaluate exter- the ears of princes,(22) which facilitates “the nal validity. Although this is best done over a dialogue between experimenters and policy- series of studies, we acknowledge the need to makers” (see also Guala 2005, 141–60). These assess the strengths of a particular study with types likely differ in the target of generaliza- respect to external validity. Individual studies tion. Of particular relevance is that theory- can be evaluated in at least two ways (Aron- oriented experiments are typically not meant son and Carlsmith 1968; Aronson, Brewer, to “match” behaviors observed outside the and Carlsmith 1985; Aronson, Wilson, and study per se, but rather the key is to gen- Brewer 1998). First, experimental realism eralize to the precise parameters put forth in refers to whether “an experiment is realistic, the given theory. Plott (1991) explains, “The if the situation is involving to the subjects, if experiment should be judged by the lessons they are forced to take it seriously, [and] if it teaches about the theory and not by its it has impact on them” (Aronson, Brewer, similarity with what nature might have hap- and Carlsmith 1985, 485). Second, mundane pened to have created” (906). This echoes realism concerns “the extent to which events Mook’s (1983) argument that much experi- occurring in the research setting are likely to mental work is aimed at developing and/or occur in the normal course of the subjects’ testing a theory, not at establishing gener- lives, that is, in the ‘real world’” (Aronson alizability. Experiments that are designed to et al. 1985, 485).7 demonstrate “what can happen” (e.g., Mil- Much debate about samples focuses on gram 1963; Zimbardo 1973) can still be use- mundane realism. When student subjects do ful, even if they do not mimic everyday life.6 not match the population to which a causal In many instances, the nature of the sub- inference is intended, many conclude that the jects in the experiments are of minimal rele- study has low external validity. Emphasis on vance, particularly given experimental efforts mundane realism, however, is misplaced (e.g., to ensure that their preferences and/or moti- McDermott 2002; Morton and Williams vations match those in the theory (e.g., see 2008, 345); of much greater importance is Dickson’s chapter in this volume). experimental realism. Failure of participants Assessment of how student subjects influ- to take the study and treatments “seriously” ence external validity depends on three con- compromises internal validity, which in turn siderations: 1) the research agenda on which renders external validity of the causal rela- the study builds (e.g., has prior work already tionship meaningless (e.g., Dickhaut, Living- established a relationship with student sub- stone, and Watson 1972, 477; Liyanarachchi jects, meaning incorporating other popula- 2007, 56).8 In contrast, at worst, low levels tions may be more pressing?); 2) the relative generalizability of the subjects, compared to 7 A third evaluative criterion is psychological realism, the setting, timing, and operationalizations which refers to “the extent to which the psychological (e.g., a study using students may have more processes that occur in an experiment are the same as psychological processes that occur in everyday life” leeway to control these other dimensions); (Aronson et al. 1998, 132). The relevance of psychological realism is debatable and depends on one’s philosophy of science (c.f. Friedman 1953;Simon1963, 6 Aronson, Wilson, and Brewer (1998) explain that it 1979, 475–76; also see MacDonald 2003). “is often assumed (perhaps mindlessly!) that all stud- 8 By “seriously,” we mean analogous to how individu- ies should be as high as possible in external validity, als treat the same stimuli in the settings to which one in the sense that we should be able to generalize the hopes to generalize (and not necessarily “serious” in a results as much as possible across populations and technical sense). We do not further discuss steps that settings and time. Sometimes, however, the goal of can be taken to ensure experimental realism because the research is different” (132). this moves into the realm of other design issues Students as Experimental Participants 45 of mundane realism simply constrain the to the treatment group.11 Suppose the true breadth of any generalization but do not make data-generating process features a homoge- the study useless. neous treatment effect: Moreover, scholars have yet to specify = β + β + ε . clear criteria for assessing mundane realism, yi 0 T Ti i (1) and, as Liyanarachchi (2007) explains, “Any superficial appearance of reality (e.g., a high Assuming that all assumptions of the classical level of mundane realism) is of little comfort, linear regression model are met, the ordinary because the issue is whether the experiment β least squares estimate for T is unbiased, con- ‘captures the intended essence of the theoret- 12 1975 106 57 9 sistent, and efficient. The results derived ical variables” (Kruglanski , )( ). from estimation on a given sample would be That said, beyond superficiality, we recog- fully generalizable to those that would result nize that student subjects – although hav- from estimation on any other sample. ing no ostensibly relevant connection with 10 Specific samples will differ in their distri- experimental realism – may limit mundane butions of individual covariates. Continuing realism that constrains generalizations of a with our running example, samples may dif- particular study. This occurs when character- fer in the distribution of attitude crystallization istics of the subjects affect the nature of the (i.e., an attitude is increasingly crystallized causal relationship being generalized. When when it is stronger and more stable).13 Stu- this happens, and with what consequences, dent samples may yield a disproportionately are questions to which we now turn. large group of subjects who are low in crystallization. A random sample from the general population might generate a group that 2. Statistical Framework is normally distributed and centered at the

In this section, we examine the use of stu- 11 For ease of exposition, our example only has one dent samples from a statistical point of view. treatment group. The lessons easily extend to multi- This allows us to specify the conditions under ple treatment groups. 12 We could have specified a data-generating process which student samples might constrain causal that also includes a direct relationship between y and generalization (in the case of a single exper- some individual-level factors such as partisanship or iment). Our focus, as in most political sci- sex (consider a vector of such variables, X). Under random assignment, the expected covariance between ence analyses of experimental data, is on the the treatment and X is zero. Hence, if we were to magnitude of the effect of some experimental estimate the model without X, then omitted variable treatment, T, on an attitudinal or behavioral bias would technically not be an issue. If the data- generating process does include X, and even though dependent measure, y. Suppose, strictly for we might not have an omitted variable bias problem, presentational purposes, we are interested in including X in the model may still be advisable. Inclu- the effect of a persuasive communication (T) sion of relevant covariates (i.e., covariates that, in the data-generating process, actually have a nonzero on a subject’s poststimulus policy opinion ( ). y effect on y) will reduce ei (the difference between T takes on a value of 0 for subjects randomly the observed and predicted y), which in turn will 2 reduce s , resulting in more precise estimated stan- assigned to the control group and takes on 1991 1 dard errors for our coefficients (see Franklin ). a value of for subjects randomly assigned Moreover, it is only in expectation that Cov(X,T ) = 0. In any given sample, Cov(X,T ) may not equal zero. (e.g., subject payments, incentives; see Dickson’s Inclusion of covariates can mitigate against inciden- chapter in this volume). tal variation in cell composition. In advising inclu- 9 Berkowitz and Donnerstein (1982) explain, “The sion of control variables, Ansolabehere and Iyengar meaning the subjects assign to the situation they (1995) note, “Randomization does not always work. are in and the behavior they are carrying out [i.e., Random assignment of treatments provides a gen- experimental realism] plays a greater part in deter- eral safeguard against biases but it is not foolproof. mining generalizability of an experiment’s outcome By chance, too many people of a particular type may than does the sample’s demographic representatives end up in one of the treatment groups, which might or the setting’s surface realism” (249). skew the results” (172; see also Bowers’ chapter in 10 This claim is in need of empirical evaluation because this volume). it may be that students are more compliant, and this 13 This example is inspired by Sears’ (1986) discussion may have an impact on realism. of “Uncrystallized Attitudes.” 46 James N. Druckman and Cindy D. Kam middle of the range. A sample from politi- should produce an unbiased estimate of that single cally active individuals (e.g., conventioneers) treatment effect, and, thus, the results from any might result in a group that is disproportion- convenience sample should generalize easily to any ately high in crystallization.14 other group. Consider the following samples with vary- Suppose, however, the “true” underlying ing distributions on attitude crystallization. data-generating process contains a heteroge- In all cases, n = 200, and treatment is ran- neous treatment effect: that is, the effect of domly assigned to half the cases. Attitude the treatment is moderated16 by individual- crystallization ranges from 0 (low) to 1 (high). level characteristics. The size of the treatment Consider a “student sample,” where ninety effect might depend on some characteristic, percent of the sample is at a value of “0” and such as gender, race, age, education, sophis- ten percent of the sample is at a value of “1.” tication, etc. Another way to say this is that Consider a “random sample,” where the sam- there may be an “interaction of causal rela- ple is normally distributed and centered on tionship with units” (Shadish et al. 2002, 87). 0.5, with standard deviation of 0.165. Finally, As one method of overcoming this issue, a consider a “conventioneers sample,” where researcher can randomly sample experimen- ten percent of the sample is at a value of “0” tal subjects. By doing so, the researcher can and ninety percent of the sample is at a value be assured that of “1.”15 β Suppose the true treatment effect ( T) the average causal relationship observed in takesavalueof“4.” We set up a Monte Carlo thesamplewillbethesameas(1)theaver- experiment that estimated Equation (1) 1,000 age causal relationship that would have been times, each time drawing a new ε term. We observed in any other random sample of per- repeated this process for each of the three sons of the same size from the same population and (2) the average causal relationship that types of samples (student, random, and con- would be been observed across all other persons ventioneers). The sampling distributions for 4 1 in that population who were not in the orig- bT appear in Figure . . inal random sample. (Shadish et al. 2002, The results demonstrate that when the 91) true data-generating process produces a single treatment effect, estimates from any sam- Although random sampling has advantages ple will produce an unbiased estimate of for external validity, Shadish et al. note that the true underlying treatment effect. Perhaps “it is so rarely feasible in experiments” (91). this point seems obvious, but we believe it The way to move to random sampling might has escaped notice from many who criticize be to use survey experiments, where respon- experiments that rely on student samples. We dents are (more or less) a random sample repeat: if the underlying data-generating process of some population of interest. We say a is characterized by a homogeneous treatment effect bit more about this possibility later in the (i.e., the treatment effect is the same across the chapter. For now, let us assume that a given entire population), then any convenience sample researcher has a specific set of reasons for not using a random sample (cost, instrumen- 14 And, of course, crystallization might vary across dif- tation, desire for laboratory control, etc.), ferent types of issues. On some issues (e.g., financial and let’s examine the challenges a researcher aid policies), students might have highly crystallized views, whereas conventioneers might have less crystallized views. 16 See Baron and Kenny (1986) for the distinction 15 Now, if our goal was to use our three samples to make between moderation and mediation. Psychologists descriptive inferences about the general population’s refer to the case where Z affects the effect of X as mod- mean level of attitude crystallization, then both the eration (i.e., an interaction effect). Psychologists refer student sample and the conventioneers sample would to mediation when some variable X influences the be inappropriate. The goal of an experimental design level of some variable Z,wherebyX affects Y through is expressly not to undertake this task. Instead, the its effect on the level of Z. For an extended treatment goals of an experimental design are to estimate the of interaction effects in regression analysis, see Kam causal effect of some treatment and then to generalize and Franzese (2007). For a discussion of mediation, it. see Bullock and Ha’s chapter in this volume. Students as Experimental Participants 47

Student Sample Percent 02 4 6 810 3.6 3.8 4 4.2 4.4

Random Sample Percent 0246810 3.6 3.8 4 4.2 4.4

Conventioneers Sample Percent 02 4 6 810 3.6 3.8 4 4.2 4.4

Figure 4.1. Sampling Distribution of bT, Single Treatment Effect Note: 1,000 iterations, estimated using Equation (1). using a convenience sample might face in this If our sample includes sufficient variance on framework. this moderator, and we have ex ante theo- We revise our data-generating process to rized that the treatment effect depends on reflect a heterogeneous treatment effect by this moderating variable, Z, then we can (and taking Equation (1) and modeling how some should) estimate the interaction. If, however, individual-level characteristic, Z (e.g., atti- the sample does not contain sufficient vari- tude crystallization), influences the magni- ance, not only can we not identify the moder- tude of the treatment effect: ating effect, but we may also misestimate the on-average effect depending on what specific β = γ + γ . range of Z is present in our sample. 1 10 11 Zi (2) The question of generalizing treatment effects reduces to asking whether there is a We also theorize that Z might influence the single treatment effect or a set of treatment intercept: effects, the size of which depends on some (set of) covariate(s). Note that this is a theoretically β = γ + γ . 0 00 01 Zi oriented question of generalization. It is not just whether “student samples are generaliz- Substituting into Equation (1), able,” but rather what particular characteristics of student samples might lead us to won- = γ + γ + γ + γ + ε der whether the causal relationship detected yi ( 00 01 Zi ) ( 10 11 Zi ) Ti i = γ + γ + γ + γ ∗ + ε . in a student sample experiment would be sys- yi 00 01 Zi 10Ti 11 Zi Ti i tematically different from the causal relation- (3) ship in the general population. 48 James N. Druckman and Cindy D. Kam

Revisiting our running example, suppose model based on Equation (1). Equation (1) we believe that a subject’s level of attitude should only be estimated when the data- crystallization (Z) inﬂuences the effect of a generating process produces a single treat- persuasive communication (T) on a subject’s ment effect: the value of β1. However, we poststimulus policy opinion (y). The more have “mistakenly” estimated Equation (1) crystallized someone’s attitude is, the smaller when the true data-generating process pro- the treatment effect should be, whereas the duces a series of treatment effects (governed β = − less crystallized someone’s attitude is, the by the function 1 5 5Zi). The sampling greater the treatment effect should be. Using distributions in Figure 4.2 provide the “aver- this running example, based on Equation (3), age” treatment effect, which depends directly assume that the true relationship has the fol- on the mean value of Z within a given sample: ∗ lowing (arbitrarily selected) values: 5 − 5 E(Z). Are the results from one sample more γ00 = 0 trustworthy than the results from another 2002 γ01 = 0 sample? As Shadish et al. ( ) note, con- γ = 5 ducting an experiment on a random sample 10 will produce an “average” treatment effect; γ11 =−5 hence, to some degree, the results from the random sample might be more desirable than Let Z, attitude crystallization, range from 0 the results from the other two convenience (least crystallized) to 1 (most crystallized). γ 10 samples. However, all three sets of results tells us the effect of the treatment when Z = reﬂect a fundamental disjuncture between the 0, that is, the treatment effect among the least model that is estimated and the true data- crystallized subjects. γ 11 tells us how crystal- generating process. If we have a theoreti- lization moderates the effect of the treatment. cal reason to believe that the data-generating First, consider what happens when we process is more complex (i.e., the treatment estimate Equation (1), the simple (but theo- depends on an individual-level moderator), retically incorrect, given that it fails to model then we should embed this theoretical model the moderating effect) model that looks for into our statistical model. the “average” treatment effect. We estimated To do so, we returned to Equation (3) and this model 1,000 times, each time drawing estimated the model 1,000 times, each time anewε term. We repeated this process for drawing a new ε term. We repeated this pro- each of the three samples. The results appear cess three times, for each of the three samples. in Figure 4.2. The results appear in Figure 4.3. When we estimate a “simple” model look- First, notice that the sampling distribu- ing for an average treatment effect, our esti- tions for bT are all centered on the same value, mates for β1 diverge from sample to sample. 5, and the sampling distributions for bTZ are In cases where we have a student sample, and also all centered on the same value, −5.In where low levels of crystallization increase the other words, Equation (3) produces unbiased β β treatment effect, we systematically overesti- point estimates for T and TZ, regardless of mate the treatment effect relative to what we which sample is used. We uncover unbiased would get in estimating the same model on a point estimates even where only ten percent random sample with moderate levels of crys- of the sample provides key variation on Z (stu- tallization. In the case of a conventioneers dent sample and conventioneers sample). sample, where high levels of crystallization Second, notice the spread of the sampling depress the treatment effect, we systemati- distributions. We have the most certainty cally underestimate the treatment effect, rel- about bT in the student sample and substan- ative to the estimates obtained from the gen- tially less certainty in the random sample eral population. and the conventioneers sample. The greater We have obtained three different results degree of certainty in the student sample across the samples because we estimated a results from the greater mass of the sample Students as Experimental Participants 49

Student Sample 10 5 Percent 0 0 1 2 3 4 5

Random Sample 10 5 Percent 0 0 1 2 3 4 5

Conventioneers Sample 10 5 Percent 0 0 1 2 3 4 5

Figure 4.2. Sampling Distribution of bT, Heterogeneous Treatment Effects Note: 1,000 iterations, estimated using Equation (1). 10 10 8 8 6 6 4 4 Percent Percent Student Sample 2 2 0 0 -10 -5 0 5 10 -10 -5 0 5 10 10 10 8 6 5 4 Percent Percent Random Sample 2 0 0 -10 -5 0 5 10 -10 -5 0 5 10 8 8 6 6 4 4 Percent Percent 2 2 0 0

Conventioneer Sample -10 -5 0 5 10 -10 -5 0 5 10

Figure 4.3. Sampling Distributions of bT and bTZ, Heterogeneous Treatment Effects Note: 1,000 iterations, estimated using Equation (3). 50 James N. Druckman and Cindy D. Kam that is located at 0 in the student sam- dent samples to offer more constructive, more β ple (because the point estimate for T,the theoretically oriented critiques that reflect uninteracted term in Equation (3), represents the possibility that student samples may be the effect of T when Z takes on the value of 0). problematic if the magnitude and direction For the sampling distribution of bTZ,we of the treatment effect depend on a partic- have higher degrees of certainty (smaller ular (set of) covariate(s) that are peculiarly standard errors) in the student sample and distributed within a student sample. the conventioneers sample. This is an inter- In sum, we have identified three distinct esting result. By using samples that have situations. First, in the homogeneous case – higher variation on Z, we yield more precise where the data-generating process produces point estimates of the heterogeneous treat- a single treatment effect – we showed that ment effect.17 Moreover, we are still able the estimated treatment effect derived from a to uncover the interactive treatment effect student sample is an unbiased estimate of the because these samples still contain some vari- true treatment effect. Second, when there is ation across values of Z. a heterogeneous case (where the treatment How much variation in Z is sufficient? As effect is moderated by some covariate Z) and long as Z varies to any degree in the sam- the researcher fails to recognize the contin- ple, the estimates for bT and bTZ will be gent effect, a student sample (indeed, any unbiased. Being “right on average” may be convenience sample) may misestimate the of little comfort if the degree of uncertainty average treatment effect if the sample is non- around the point estimate is large. If Z does representative on the particular covariate Z. not vary very much in a given sample, then the However, in this case, even a representa- estimated standard error for bTZ will be large. tive sample would misspecify the treatment But concerns about uncertainty are “run of effect due to a failure to model the inter- the mill” when estimating a model on any action. Third, when the researcher appro- dataset: more precise estimates arise from priately models the heterogeneity with an analyzing datasets that maximize variation in interaction, then the student sample, even if our independent variables. it is nonrepresentative on the covariate Z, will Our discussion thus suggests that experi- misestimate the effect only if there is virtually mentalists (and their critics) need to consider no variance (i.e., literally almost none) on the the underlying data-generating process: that moderating dynamic. Moreover, a researcher is, theory is important. If a single treatment can empirically assess the degree of vari- effect is theorized, then testing for a single ance on the moderator within a given sample treatment effect is appropriate. If a hetero- and/or use simulations to evaluate whether geneous treatment effect is theorized, then limited variance poses a problem for uncov- researchers should explicitly theorize how the ering the interactive effect. An implication is treatment effect should vary along a spe- that the burden, to some extent, falls on an cific (set of) covariate(s), and researchers can experiment’s critic to identify the moderating thereby estimate such relationships as long factor and demonstrate that it lacks variance as there is sufficient variation in the spe- in an experiment’s sample. cific (set of) covariate(s) in the sample. We hope to push those who launch vague criticisms regarding the generalizability of stu- 3. Contrasting Student Samples with Other Samples 17 Uncovering more certainty in the student and conventioneers samples (compared to the random sam- We argue that a given sample constitutes only ple) derives from the specific ways in which we have constructed the distributions of Z. If the random one – and arguably not the critical one – sample were, say, uniformly distributed rather than of many considerations when it comes to normally distributed along Z, then the same result assessing external validity. Furthermore, a would not hold. The greater precision in the estimates depends on the underlying distribution of Z in student sample only creates a problem when a given sample. a researcher 1) fails to model a contingent Students as Experimental Participants 51 causal effect (when there is an underlying compared individuals currently enrolled in heterogeneous treatment effect), and 2)the college against the general population. The students differ from the target population Web Appendix19 contains question wording with regard to the distribution of the mod- for each item. erating variable. This situation, which we As we can see from Table 4.1,inmost acknowledge does occur with nontrivial fre- cases, the difference in means for students and quency, leads to the question of just how often the nonstudent general population are indis- student subjects empirically differ from rep- tinguishable from zero. Students and the non- resentative samples. The greater such differ- student general population are, on average, ences, the more likely problematic inferences indistinguishable when it comes to partisan- will occur. ship (we find this for partisan direction and Kam (2005) offers some telling evidence intensity), ideology, importance of religion, comparing student and nonstudent samples belief in limited government, views about on two variables that can affect information homosexuality as a way of life, contributions processing: political awareness and need for of immigrants to society, social trust, degree cognition. She collected data from a student of following and discussing politics, and over- sample using the same items that are used all media use. Students are distinguishable in the American National Election Study’s from nonstudents in religious attendance, (ANES’s) representative sample of adult cit- level of political information as measured in izens. She found that the distributions for this dataset,20 and specific types of media use. both variables in the student sample closely Overall, however, we are impressed by just resemble those in the 2000 ANES. This near- how similar students are to the nonstudent identical match in distribution allowed Kam general population on key covariates often of to generalize more broadly results from an interest to political scientists. experiment on party cues that she ran with In cases where samples differ on vari- the student subjects. ables that are theorized to influence the size Kam (2005) focuses on awareness and need and direction of the treatment effect, the for cognition because these variables plau- researcher should, as we note previously, sibly moderate the impact of party cues; as model the interaction. The researcher might explained, in comparing student and non- also consider cases where students – despite student samples, one should focus on pos- differing on relevant variables – might be sible differences that are relevant to the study advantageous. In some situations, students in question. Of course, one may nonetheless facilitate testing a causal proposition. Stu- wonder whether students differ in other ways dents are relatively educated, in need of small that could matter (e.g., Sears 1986, 520). This amounts of money, and accustomed to fol- requires a more general comparison, which lowing instructions (e.g., from professors) we undertake by turning to the 2006 Civic (Guala 2005, 33–34). For these reasons, stu- and Political Health of the Nation Dataset dent samples may enhance the experimental (collected by CIRCLE) (for a similar exer- realism of experiments that rely on induced cise, see Kam et al. 2007). value theory (where monetary payoffs are These data consist of telephone and web used to induce preferences) and/or involve interviews with 2,232 individuals aged 15 relatively complicated, abstract instructions years and older living in the continental (Friedman and Sunder 1994, 39–40).21 The United States. We limited the analysis to individuals aged 18 years and older. We 19 Available at http://faculty.wcas.northwestern.edu/∼ selected all ostensibly politically relevant pre- jnd260/publications.html. 18 20 The measure of political information in this dataset is dispositions available in the data and then quite different from that typically found in the ANES; it is heavier on institutional items and relies more on 18 We did this before looking at whether there were dif- recall than recognition. ferences between students and the nonstudent gen- 21 We suspect that this explains why the use of student eral population sample; that is, we did not selectively subjects seems to be much less of an issue in experi- choose variables. mental economics (e.g., Guala 2005). 52 James N. Druckman and Cindy D. Kam

Table 4.1: Comparison of Students versus Nonstudent General Population

Nonstudent General Students Population p Value

Partisanship 0.47 0.45 ns (0.02)(0.01) Ideology 0.50 0.52 ns (0.01)(0.01) Religious attendance 0.56 0.50 <.01 (0.02)(0.01) Importance of religion 0.63 0.62 ns (0.02)(0.02) Limited government 0.35 0.33 ns (0.03)(0.02) Homosexuality as a way of life 0.60 0.62 ns (0.03)(0.02) Contribution of immigrants to society 0.62 0.63 ns (0.03)(0.02) Social trust 0.34 0.33 ns (0.03)(0.02) Follow politics 0.68 0.65 ns (0.02)(0.01) Discuss politics 0.75 0.71 ns (0.01)(0.01) Political information (0–6 correct) 2.53 1.84 <.01 (0.11)(0.07) Newspaper use (0–7 days) 2.73 2.79 ns (0.14)(0.11) National television news (0–7 days) 3.28 3.63 <.05 (0.15)(0.10) News radio (0–7 days) 2.47 2.68 ns (0.16)(0.11) Web news (0–7 days) 3.13 2.18 <.01 (0.16)(0.10) Overall media use 2.90 2.83 ns (0.09)(0.06)

Notes: ns, not signiﬁcant. Weighted analysis. Means with standard errors in parentheses. See the Web Appendix for variable coding and question text. Source: Lopez, Mark Hugo, Peter Levine, Deborah Both, Abby Kiesa, Emily Kirby, and Karlo Marcelo. 2006.“The2006 Civic and Political Health of the Nation Survey: A Detailed Look at How Youth Participate in Politics and Communities.” October. Retrieved from www.civicyouth.org/PopUps/ 2006 CPHS Report update.pdf (October 30, 2010).

goal of many of these experiments is to test of utmost importance (rather than mundane theory, and, as mentioned, the match to the realism). theoretical parameters (e.g., the sequence of Alternatively, estimating a single treat- events if the theory is game theoretic) is ment effect on a student sample subject pool Students as Experimental Participants 53 can sometimes make it more difficult to find tion of what would be found in the general effects. For example, studies of party cues population. examine the extent to which subjects will follow the advice given to them by political parties. Strength of party identification 4. Conclusion might be a weaker cue for student subjects, whose party affiliations are still in the for- As mentioned, political scientists are guilty mative stages (Campbell et al. 1960). If this of a “near obsession” with external validity were the case, then the use of a student sam- (McDermott 2002, 334) and this obsession ple would make it even more difficult to dis- focuses nearly entirely on a single dimension cover party cue effects. To the extent that of external validity – who is studied. Our goal party cues work among student samples, these in this chapter has been to situate the role of likely underestimate the degree of cue taking experimental samples within a broader frame- that might occur among the general popula- work of how one might assess the generaliz- tion, whose party affiliations are more deeply ability of an experiment. Our key points are grounded. Similarly, students seem to exhibit as follows: relatively lower levels of self-interest and sus- r ceptibility to group norms (Sears 1986, 524), The external validity of a single experi- meaning that using students in experiments mental study must be assessed in light of on these topics increases the challenge of an entire research agenda and in light of identifying treatment effects.22 the goal of the study (e.g., testing a theory Finally, it is worth mentioning that if the or searching for facts). r goal of a set of experiments is to generalize Assessment of external validity involves a theory, then testing the theory across a set multiple dimensions, including the sam- of carefully chosen convenience samples may ple, context, time, and conceptual opera- even be superior to testing the theory within tionalization. There is no reason per se to a single random sample.23 A theory of the prioritize the sample as the source of an moderating effect of attitude crystallization inferential problem. Indeed, we are more on the effects of persuasive communications likely to lack variance on context and tim- might be better tested on a series of different ing because these are typically constants in samples (and possibly different student sam- the experiment. r ples) that vary on the key covariate of interest. In assessing the external validity of the Researchers need to consider what partic- sample, experimental realism (as opposed ular student sample characteristics might lead to mundane realism) is critical, and there is a causal relationship discovered in the sample nothing inherent to the use of student sub- to differ systematically from what would be jects that reduces experimental realism. r found in the general population. Researchers The nature of the sample – and the use of then need to elaborate on the direction of the students – matters in certain cases. How- bias: the variation might facilitate the assess- ever, a necessary condition is a hetero- ment of causation, and/or it might lead to geneous (or moderated) treatment effect. either an overestimation or an underestima- Then the impact depends on the following: r If the heterogeneous effect is theorized, 22 As explained, students also tend to be more suscep- then the sample only matters if there tible to persuasion (Sears 1986). This makes them a is virtually no variance on the mod- more challenging population on which to experiment if the goal is to identify conditions where persuasive erator. If there is even scant variance, messages fail (e.g., Druckman 2001). then the treatment effect will not only 23 Convenience samples might be chosen to represent be correctly estimated but may also be groups that are high and low on a particular covariate of interest. This purposive sampling might yield more estimated with greater confidence. The rewards than using a less informative random sample. suitability of a given sample can be 54 James N. Druckman and Cindy D. Kam

assessed (e.g., empirical variance can be feasible, researchers can take a second-best analyzed). approach by utilizing question wordings that r The range of heterogeneous, nontheo- match those in general surveys (thereby facil- rized cases may be much smaller than is itating comparisons). often believed. Indeed, when it comes to a Third, we hope for more discussion about host of politically relevant variables, stu- the pros and cons of alternative modes dent samples do not significantly differ of experimentation, which may be more from nonstudent samples. amenable to using nonstudent subjects. Al- r There are cases where student samples are though we recognize the benefits of using desirable because they facilitate causal tests survey and/or field experiments, we should or make for more challenging assessments. not be overly sanguine about their advantages. For example, the control available in Our argument has a number of practi- laboratory experiments enables researchers to cal implications. First, we urge researchers maximize experimental realism (e.g., by using to attend more to the potential moder- induced value or simply by more closely mon- ating effects of the other dimensions of itoring the subjects). Similarly, there is less generalizability: context, time, and conceptu- concern in laboratory settings about compli- × alization. The past decade has seen an enor- ance treatment interactions that become mous increase in survey experiments due in problematic in field experiments or spillover no small way to the availability of more effects in survey experiments (Transue, Lee, 2009 representative samples. Yet, scholars must and Aldrich ; also see Sinclair’s chapter account for the distinct context of the survey in this volume). The increased control offered interview (e.g., Converse and Schuman 1974; by the laboratory setting often affords greater Zaller 1992, 28). Sniderman, Brody, and Tet- ability to manipulate context and time, which, lock (1991) elaborate that “the conventional we argue, deserves much more attention. survey interview, though well equipped to Finally, when it comes to the sample, atten- assess variations among individuals, is poorly tion should be paid to the nature of any equipped to assess variation across situations” sample and not just student samples. This (265). Unlike most controlled lab settings, includes consideration of nonresponse biases 2008 researchers using survey experiments have in surveys (see Groves and Peytcheva ) limited ability to introduce contextual vari- and the impact of using “professional” survey respondents that are common in many ations. 25 Second, we encourage the use of dual sam- web-based panels. In short, the nature of ples of students and nonstudents. The dis- any particular sample needs to be assessed in covery of differences should lead to serious consideration of what drives distinctions (i.e., employed, and the reactions they displayed. Mintz et al. conclude that “student samples are often inap- what is the underlying moderating dynamic, propriate, as empirically they can lead to divergence and can it be modeled?). The few studies in subject population results” (769). We would argue that compare samples (e.g., Gordon et al. that this conclusion is premature. Although their 1986 2001 2001 results reveal differences, on average, between the ; James and Sonner ; Peterson ; samples, the authors leave unanswered why the differ- Mintz, Redd, and Vedlitz 2006; Depositario ences exist. Mintz et al. speculate that the differences 2009 may stem from variations in expertise, age, account- et al. ; Henrich, Heine, and Noren- 769 2009 ability, and gender ( ). A thorough understanding zayan ), although sometimes reporting of the heterogeneity in the treatment effects (which, differences, rarely explore the nature of the as explained, is the goal of any experiment) would, differences.24 When dual samples are not thus, require exploration of these moderators. Our simulation results suggest that even if the student sample exhibited limited variation on these variables, 24 For example, Mintz, Redd, and Vedlitz (2006)imple- it could have isolated the same key treatment dynam- mented an experiment, with both students and mil- ics that would be found in the military sample. itary officers, about counterterrorism decision mak- 25 The use of professional, repeat respondents raises ing. They found that the two samples significantly similar issues to those caused by repeated use of par- differed, on average, in the decisions they made, the ticipants from a subject pool (e.g., Stevens and Ash information they used, the decision strategies they 2001). Students as Experimental Participants 55 light of various trade-offs, including consid- Journal of Personality and Social Psychology 51: eration of an experiment’s goal, costs of dif- 1173–82. ferent approaches, and other dimensions of Benz, Matthias, and Stephan Meier. 2008.“Do generalizability. People Behave in Experiments as in the Field?: Evidence from Donations.” Experimental Eco- We have made a strong argument for the 11 268 81 increased usage and acceptance of student nomics : – . Berkowitz, Leonard, and Edward Donnerstein. subjects, suggesting that the burden of proof 1982. “External Validity Is More Than Skin be shifted from the experimenter to the critic Deep: Some Answers to Criticisms of Labo- 1994 16 (also see Friedman and Sunder , ). We ratory Experiments.” American Psychologist 37: recognize that many will not be persuaded; 245–57. however, at the very least, we hope to stimu- Campbell, Angus, Philip Converse, Warren late increased discussion about why and when Miller, and Donald Stokes. 1960. The Ameri- student subjects may be problematic. can Voter. Chicago: The University of Chicago Press. [Reprinted in 1980.] Campbell, Donald T. 1969. “Prospective: Artifact References and Control.” In Artifact in Behavioral Research, eds. Robert Rosenthal and Robert Rosnow. Anderson, Craig A., and Brad J. Bushman. 1997. New York: Academic Press, 351–82. “External Validity of ‘Trivial’ Experiments: Chong, Dennis, and James N. Druckman. 2007a. The Case of Laboratory Aggression.” Review “Framing Public Opinion in Competitive of General Psychology 1: 19–41. Democracies.” American Political Science Review Ansolabehere, Stephen, and Shanto Iyengar. 1995. 101: 637–55. Going Negative: How Political Advertisements Chong, Dennis, and James N. Druckman. 2007b. Shrink and Polarize the Electorate. New York: “Framing Theory.” Annual Review of Political Free Press. Science 10: 103–26. Arceneaux, Kevin, and Martin Johnson. 2008. Converse, Jean M., and Howard Schuman. 1974. “Choice, Attention, and Reception in Political Conversations at Random: Survey Research as Communication Research.” Paper presented at Interviewers See It. New York: Wiley. the annual meeting of the International Society Depositario, Dinah Pura T., Rodolfo M. Nayga for Political Psychology, Paris. Jr., Ximing Wu, and Tiffany P. Laude. 2009. Aronson, Elliot, Marilynn B. Brewer, and J. Mer- “Should Students Be Used as Subjects in Exper- ill Carlsmith. 1985. “Experimentation in Social imental Auctions?” Economic Letters 102: 122– Psychology.” In Handbook of Social Psychol- 24. ogy. 3rd ed., eds. Gardner Lindzey and Elliot Dickhaut, John W., J. Leslie Livingstone, and Aronson. New York: Random House, 441– David J. H. Watson. 1972. “On the Use of Sur- 86. rogates in Behavioral Experimentation.” The Aronson, Elliot, and J. Merill Carlsmith. 1968. Accounting Review 47 (Suppl): 455–71. “Experimentation in Social Psychology.” In Druckman, James N. 2001. “On the Limits of Handbook of Social Psychology. 2nd ed., eds. Gard- Framing Effects.” Journal of Politics 63: 1041– ner Lindzey and Elliot Aronson. Reading, MA: 66. Addison-Wesley, 1–79. Druckman, James N. 2009.“CompetingFramesin Aronson, Elliot, Timothy D. Wilson, and Mar- a Campaign.” Unpublished paper, Northwest- ilynn B. Brewer. 1998. “Experimentation in ern University. Social Psychology.” In The Handbook of Social Druckman, James N., Donald P. Green, James Psychology. 4th ed., eds. Daniel T. Gilbert, H. Kuklinski, and Arthur Lupia. 2006.“The Susan T. Fiske, and Gardner Lindzey. Boston: Growth and Development of Experimental McGraw-Hill, 99–142. Research Political Science.” American Political Barabas, Jason, and Jennifer Jerit. 2010. “Are Sur- Science Review 100: 627–36. vey Experiments Externally Valid?” American Franklin, Charles H. 1991. “Efficient Estimation Political Science Review 104: 226–42. in Experiments.” The Political Methodologist 4: Baron, Reuben M., and David A. Kenny. 1986. 13–15. “The Moderator–Mediator Variable Distinc- Friedman, Daniel, and Shyam Sunder. 1994. tion in Social Psychological Research: Concep- Experimental Economics: A Primer for Economists. tual, Strategic, and Statistical Considerations.” New York: Cambridge University Press. 56 James N. Druckman and Cindy D. Kam

Friedman, Milton. 1953. Essays in Positive Eco- Kinder, Donald R. 1998. “Communication and nomics. Chicago: The University of Chicago Opinion.” Annual Review of Political Science 1: Press. 167–97. Gaines, Brian J., James H. Kuklinski, and Paul J. Kruglanski, Arie W. 1975. “The Human Subject Quirk. 2007. “The Logic of the Survey Exper- in the Psychology Experiment: Fact and Arti- iment Reexamined.” Political Analysis 15: 1–20. fact.” In Advances in Experimental Social Psy- Gerber, Alan S., James G. Gimpel, Donald P. chology, ed. Leonard Berkowitz. New York: Green, and Daron R. Shaw. 2007.“TheInflu- Academic Press, 101–47. ence of Television and Radio Advertising on Lau, Richard R., and David P. Redlawsk. 2006. Candidate Evaluations: Results from a Large How Voters Decide: Information Processing in Elec- Scale Randomized Experiment.” Unpublished tion Campaigns. Cambridge: Cambridge Uni- paper, Yale University. versity Press. Gerber, Alan S., and Donald P. Green. 2008. Liyanarachchi, Gregory A. 2007. “Feasibility of “Field Experiments and Natural Experiments.” Using Student Subjects in Accounting Experi- In The Oxford Handbook of Political Methodol- ments: A Review.” Pacific Accounting Review 19: ogy, eds. Janet M. Box-Steffensmeier, Henry 47–67. E. Brady, and David Collier. Oxford: Oxford MacDonald, Paul. 2003. “Useful Fiction or Mir- University Press, 357–81. acle Maker: The Competing Epistemologi- Gordon, Michael E., L. Allen Slade, and Neal cal Foundations of Rational Choice Theory.” Schmitt. 1986. “The ‘Science of the Sopho- American Political Science Review 97: 551–65. more’ Revisited: From Conjecture to Empiri- McDermott, Rose. 2002. “Experimental Method- cism.” Academy of Management Review 11: 191– ology in Political Science.” Political Analysis 10: 207. 325–42. Groves, Robert M., and Emilia Peytcheva. 2008. Milgram, Stanley. 1963. “Behavioral Study of “The Impact of Nonresponse Rates on Nonre- Obedience.” Journal of Abnormal and Social Psy- sponse Bias: A Meta-Analysis.” Public Opinion chology 67: 371–78. Quarterly 72: 167–89. Mintz, Alex, Steven B. Redd, and Arnold Vedlitz. Guala, Francesco. 2005. The Methodology of Exper- 2006. “Can We Generalize from Student imental Economics. New York: Cambridge Uni- Experiments to the Real World in Political Sci- versity Press. ence, Military Affairs, and International Rela- Henrich, Joseph, Steven J. Heine, and Ara Noren- tions?” Journal of Conflict Resolution 50: 757–76. zayan. 2009. “The Weirdest People in the Mook, Douglas G. 1983. “In Defense of External World: How Representative Are Experimental Invalidity.” American Psychologist 38: 379–87. Findings from American University Students? Morton, Rebecca B., and Kenneth C. Williams. What Do We Really Know about Human Psy- 2008. “Experimentation in Political Science.” chology?” Unpublished paper, University of In The Oxford Handbook of Political Methodol- British Columbia. ogy, eds. Janet M. Box-Steffensmeier, Henry E. James, William L., and Brenda S. Sonner. 2001. Brady, and David Collier. Oxford: Oxford Uni- “Just Say No to Traditional Student Samples.” versity Press, 339–56. Journal of Advertising Research 41: 63–71. Peterson, Robert A. 2001. “On the Use of College Kam, Cindy D. 2005. “Who Toes the Party Students in Social Science Research: Insights Line?: Cues, Values, and Individual Differ- from a Second-Order Meta-Analysis.” Journal ences.” Political Behavior 27: 163–82. of Consumer Research 28: 450–61. Kam, Cindy D. 2007. “When Duty Calls, Do Cit- Plott, Charles R. 1991. “Will Economics Become izens Answer?” Journal of Politics 69: 17–29. an Experimental Science?” Southern Economic Kam, Cindy D., and Robert J. Franzese, Jr. 2007. Journal 57: 901–19. Modeling and Interpreting Interactive Hypotheses Roth, Alvin E. 1995. “Introduction to Experimen- in Regression Analysis. Ann Arbor: University of tal Economics.” In The Handbook of Experimen- Michigan Press. tal Economics, eds. John H. Kagel and Alvin Kam, Cindy D., Jennifer R. Wilking, and Eliza- E. Roth. Princeton, NJ: Princeton University beth J. Zechmeister. 2007. “Beyond the ‘Nar- Press, 3–109. row Data Base’: Another Convenience Sample Sears, David O. 1986. “College Sophomores in for Experimental Research.” Political Behavior the Laboratory: Influence of a Narrow Data 29: 415–40. Base on Social Psychology’s View of Human Students as Experimental Participants 57

Nature.” Journal of Personality and Social Psy- Sniderman, Paul M., and Sean M. Theriault. 2004. chology 51: 515–30. “The Structure of Political Argument and the Shadish, William R., Thomas D. Cook, and Don- Logic of Issue Framing.” In Studies in Public ald T. Campbell. 2002. Experimental and Quasi- Opinion, eds. Willem E. Saris and Paul M. Sni- Experimental Designs for Generalized Causal derman. Princeton, NJ: Princeton University Inference. Boston: Houghton Mifﬂin. Press, 133–65. Simon, Herbert A. 1963. “Problems of Method- Stevens, Charles D., and Ronald A. Ash. 2001. ology Discussion.” American Economic Review “The Conscientiousness of Students in Subject Proceedings 53: 229–31. Pools: Implications of ‘Laboratory’ Research.” Simon, Herbert A. 1979. “Rational Decision Mak- Journal of Research in Personality 35: 91– ing in Business Organizations.” American Eco- 97. nomic Review 69: 493–513. Transue, John E., Daniel J. Lee, and John H. Slothuus, Rune. 2009. “The Political Logic of Aldrich. 2009. “Treatment Spillover Effects Party Cues in Opinion Formation.” Paper pre- across Survey Experiments.” Political Analysis sented at the annual meeting of the Midwest 17: 143–61. Political Science Association, Chicago. Zaller, John. 1992. The Nature and Origins of Sniderman, Paul M., Richard A. Brody, and Mass Opinion. New York: Cambridge Univer- Philip E. Tetlock. 1991. Reasoning and Choice: sity Press. Explorations in Political Psychology. Cambridge: Zimbardo, Phillip. 1973. “A Pirandellian Prison.” Cambridge University Press. New York Times Magazine, April 8. CHAPTER 5

Economics versus Psychology Experiments

Stylization, Incentives, and Deception

Eric S. Dickson

In this chapter, I follow other authors (e.g., Section 3 considers the use of deception. Kagel and Roth 1995; McDermott 2002; The psychological school tends to see decep- Camerer 2003; Morton and Williams 2010) tion as a useful tool in experimentation, and, in focusing on a few key dimensions of differ- at times, a necessary one. In contrast, the eco- ence between experiments in the economic nomic school by and large considers decep- and psychological traditions. tion to be taboo. Section 1 considers the level of stylization These basic differences in research style typical in economics and psychology exper- highlight the historical divide between psy- imentation. Although research in the polit- chological and economic – alternatively, ical psychology tradition tends to place an behavioral and rational choice – scholarship emphasis on the descriptive realism of lab- in political science. Over the years, scholars oratory scenarios, work in experimental eco- have tended to peer across this divide with nomics tends to proceed within a purposefully more mistrust than understanding, and intel- abstract, “context free” environment. lectual interchange between the different Section 2 considers the kinds of incentives schools has been lamentably limited in scope. offered to subjects by experimentalists from However, the difference in approaches bet- these two schools of thought. Experimental ween psychologists and economists reflects economists generally offer subjects monetary more than the sociology of their respective incentives that depend on subjects’ choices traditions; many of the norms characteristic in the laboratory – and, in game-theoretic of each field have evolved in response to experiments, the choices of other subjects as the specific nature of theory and of inquiry well. In contrast, psychology research tends within the separate disciplines. not to offer inducements that are condi- To say that each school of experimenta- tional on subjects’ actions, instead giving sub- tion has categorical strengths and weaknesses jects fixed cash payments or fixed amounts of would perhaps be too strong a claim. Rather, course credit. in this chapter, I argue that the advantages

58 Economics versus Psychology Experiments 59 and disadvantages associated with specific rather than in a naturalistic manner. The design choices may play out differently, roles assumed by subjects, and the alternatives depending on the nature of the research ques- that subjects face, are generally described tion being posed, the theory being tested, and using neutral terminology with a minimum even the results that are ultimately obtained. of moral or emotional connotations; experi- In this chapter, I organize my discussion mental instructions are written in a techno- around the logic of inference in economics- cratic style. For example, in their landmark and psychology-style experiments. Going study of punishment in games of public goods down this path leads me to several conclu- provision, Fehr and Gachter¨ (2000)employ sions that may at first seem counterintuitive. an experimental frame using strictly neutral For example, I will argue that stylized, language, never once mentioning the word economics-style experimentation can some- “punishment” or other potentially charged times be particularly valuable in the study terms such as “fairness” or “revenge.” In a of essentially psychological research ques- similar way, Levine and Palfrey (2007)usethe tions. Contrary to the way our discipline labels X and Y, rather than terms like “vote” has traditionally been organized around sepa- and “abstain,” in their experimental study on rate schools of methodological practice, strat- voter turnout; the cost of voting is translated egy and psychology are inextricably bound into a “Y bonus” accruing only to individ- together in virtually all political phenom- uals who choose Y, that is, do not vote. In ena that we desire to understand. The multi- their study of deliberation, Dickson, Hafer, faceted nature of our objects of study, along and Landa (2008) model individual decisions with the varying strengths and weaknesses to communicate in a stylized environment; of different research methods in attacking the “arguments” exchanged during delibera- different problems, highlight the advantages tion are represented using simple single-digit of methodological pluralism in building an numbers. intellectually cumulative literature in experi- The abstract experimental tasks associ- mental political science. ated with this form of stylization are used in part because of a desire to maintain experimental control. Researchers in this tradi- 1. Stylized versus Contextually Rich tion generally believe that the use of nor- Experimental Scenarios matively charged terms such as punishment, fairness, or revenge may evoke reactions in A first salient dimension of difference subjects whose source the analyst cannot between economics and psychology experi- fathom and that the analyst cannot properly ments is rooted in the basic nature of the measure. Experimental economists would experimental scenarios presented to subjects. generally argue that such loss of control With some exceptions, economics experi- would limit the generalizability, and thus the ments tend to be carried out in a highly usefulness, of their findings in the laboratory. stylized environment in which the scenar- According to this argument, the descrip- ios presented to subjects are purposefully tively appealing complexity of highly con- abstract, whereas experiments in psychology textual experiments comes with strings tend to evoke more contextually rich settings. attached when it comes to inference. Sup- Because the economic style of experimenta- pose that a particular effect is measured in a tion is likely to be more foreign to many read- contextually rich setting. More or less by ers, this discussion begins by describing some definition, contextually rich settings con- arguments that have been given in support of tain many features that could potentially stylization in laboratory experiments. claim subjects’ attention or influence subjects’ behavior or cognition. Given this, how could Logic of Stylization we know which feature of the setting – or Research in the economic style tends to which combination of features – led to the frame experimental scenarios in an abstract effect that we observed? 60 Eric S. Dickson

In contrast, it is argued that a similar effect Such arguments in favor of stylization measured in a stylized setting may have have, in fact, even been employed from wider lessons to teach. One argument for this time to time within social psychology itself. claim can be explicated through the use of The minimal group experimental paradigm two examples. First, consider the Fehr and (Tajfel et al. 1971; Tajfel and Turner 1986) Gachter¨ (2000) experiment, which demon- demonstrated that social identities can moti- strated that many experimental subjects are vate individual behavior even when those willing to undertake costly punishment of social identities were somewhat laughable counterparts who fail to make adequate con- constructs artificially induced within a styl- tributions to a public good, even under con- ized setting, for instance, dividing subjects ditions where such punishment is costly and based on their tendency to overcount or no benefit from punishment can accrue to the undercount dots on a screen or their pref- punisher. Because this result was obtained in erence for paintings by one abstract painter such an abstract choice environment, which (Klee) over another (Kandinsky). The finding did not directly prime subjects to think in that even these social identities could affect terms of punishment or fairness, the result behavior helped establish social identity the- seems unlikely to be merely an artifact of ory and motivate a vast field of research. some abstruse detail of the experimental frame presented to subjects. A more natural Limits of Stylization interpretation of the study’s findings is that a willingness to punish the violation of norms is The first and perhaps most obvious point is a basic feature of human nature that comes to that certain research questions – particularly be expressed even in novel settings in which certain research questions in political psy- subjects lack experience or obvious referents. chology – cannot reasonably be posed both in As such, the use of an abstract, stylized envi- stylized and in contextually rich settings. Just ronment in the study arguably strengthens to take one clear-cut example, Brader (2005) rather than weakens the inferences we make studies the effects of music within political from its result. advertisements on voters’ propensities to turn Second, Dickson et al. (2008) demonstrate out, on voters’ seeking additional political that many subjects “overspeak” compared information, and on other dependent vari- to a benchmark equilibrium prediction; that ables. It would obviously make little sense is, subjects often choose to exchange argu- to attempt to translate such a study into a ments during the course of deliberation even highly stylized setting because the psycholog- when they are more likely to alienate listen- ical mechanisms Brader explores are so deeply ers than persuade them. This finding sug- rooted in the contextual details of his experi- gests that deliberation may unfold in a man- mental protocol. ner more compatible with the deliberative Many other research questions, however, democratic ideal of a “free exchange of argu- could potentially lend themselves to explo- ments” than a fully strategic model would ration either in stylized or in highly contex- be likely to predict. In their study, styliza- tual contexts. In considering the advantages tion has at least two distinct advantages. The and disadvantages of stylization in such cases, use of a stylized, game-theoretic environment a natural question to ask is whether experi- allowed for the definition of a rational choice mental results obtained using both methods benchmark in the first place – without which tend to lead to similar conclusions. overspeaking could not have been defined For at least some research questions, the or identified. And the finding that individu- evidence suggests that stylization may lead als overspeak, even in a stylized environment to conclusions that are misleading or at least without obvious normative referents, under- incomplete. A classic example comes from the scores the behavioral robustness of individ- psychology literature on the Wason selection ual willingness to exchange arguments with task. In Wason’s (1968) original study, sub- others. jects were given a number of cards, each of Economics versus Psychology Experiments 61 which had a number on one side and a letter guidance as to when stylization might pose on the other, and a rule that had to be tested, greater problems for external validity. Many namely, that every card with a vowel on one scholars find that stylization can be beneficial, side has an even number on the other side. given their research questions – because of a Given a selection of cards labeled E, K, 4, perceived higher degree of experimental con- and 7, subjects were required to answer which trol, because stylization can sometimes allow cards must be turned over in order to test for a clearer definition of theoretical bench- the rule. In this study, only a small fraction marks than might be the case in a highly of subjects gave the correct answer (E and contextual environment, or because stylized 7); especially few noted that the rule could environments can sometimes pose a “tough be falsified by turning over the 7 and finding test” for measuring behavioral or psycholog- a vowel, whereas others included 4 in their ical phenomena, as in the Fehr and Gachter¨ answer in an apparent search for informa- (2000) and Dickson et al. (2008) studies. At tion confirming the rule. This finding is often the same time, a literature consisting wholly taken as clear evidence for a confirmatory bias of such studies would be widely met with jus- in hypothesis testing. tifiable skepticism about external validity. At The Wason (1968) selection task became least for many research areas within political a popular paradigm in the aftermath of the science, the best progress is likely to be made original study, and parallel versions have been most quickly when research in both traditions carried out in many different settings. Inter- is carried out – and when scholars commu- estingly, subjects’ performance at the task nicate about their findings across traditional appears to be highly variable, depending on dividing lines. When research using differ- the context in which the task is presented. In ent techniques tends to point in the same another well-known study, Griggs and Cox direction, we can have more confidence in (1982) present subjects with a selection task the results than we could have if only one logically equivalent to Wason’s, but rather research method had been employed. When than using abstract letters and numbers as research using different techniques instead labels, the task is framed as a search for vio- points in different directions, the details of lators of a social norm: underage drinking. In these discrepancies may prove invaluable in this study, most subjects are readily able to provoking new theoretical explanations for answer correctly that people who are drink- the phenomenon at hand, as scholars attempt ing and people who are known to be underage to understand the discrepancies’ origins. are the ones whose age or behavior needs to be examined when searching for instances of underage drinking. 2. Use of Monetary Incentives Results such as these suggest that subjects may sometimes think about problems In most economics experiments, subjects quite differently, depending on the frame in receive cash payments that depend on their which the problem is presented, an intuition own choices in the laboratory and, in the that seems natural to scholars with a back- case of game-theoretic experiments, on the ground in psychology. At the same time, such choices of other people. In contrast, subjects results by no means imply that stylized stud- who take part in political psychology exper- ies yield different results from highly con- iments are generally compensated in a way textual ones more generally. To take fram- that does not depend on the choices they ing effects themselves as an example, parallel make, typically either a fixed cash payment or literatures within economics and psychology a fixed amount of course credit. What moti- suggest that frames can affect choice behav- vates experimentalists from these two tradi- ior in similar ways both in stylized and highly tions to take different approaches to motivat- contextual environments. ing subjects? As of now, there is nothing like a gen- The most obvious point to make is that eral theory that would give experimentalists many research studies in political psychology 62 Eric S. Dickson are not well suited to the use of monetary viduals learn from the political communica- incentives because the relevant quantities of tions to which they are exposed and whether interest cannot be monetized in a reason- citizens are actually able to learn what they able way. For example, in a framing study need to in order to make reasoned choices by Druckman and Nelson (2003), subjects (Lupia and McCubbins 1998). In pursuit report their attitudes on political issues after of these objectives, a number of scholars exposure to stimuli in the form of newspaper have devised stylized experimental settings in articles. Clearly, in studies with a dependent which subjects receive messages whose infor- variable, offering subjects financial incentives mational value can be objectively weighed to report one opinion as opposed to another using Bayes’ rule in the context of a signaling would be of no help whatsoever in study- game equilibrium (e.g., Lupia and McCub- ing framing effects or the formation of public bins 1998; Dickson in press). Subjects then opinion. receive monetary rewards that depend on the Of course, the same is not true of all degree of fit between their own posterior research questions of interest to politi- beliefs and the “correct” beliefs implied by cal experimentalists, political psychologists Bayesian rationality in equilibrium. included. As such, experimenters sometimes have a real choice to make in deciding Monetary Incentives as a Means whether to motivate subjects with monetary of Controlling for Preferences incentives. In considering the implications of this choice, it is useful to review some of the Many experiments in political economy focus varied purposes for which monetary incen- on the effects of institutions in shaping indi- tives have been used in experiments. vidual behavior. Such experiments are typically organized as tests of predictions from game-theoretic models. Of course, actors’ Monetary Incentives as a Means of preferences over different possible outcomes Rewarding Accuracy or Reducing Noise are primitive elements of such models. As One potential use for monetary incentives in such, to expose a game-theoretic model to experiments is to reward accuracy. Experi- an experimental test, it must be that there mentalists want to ensure that subjects actu- is some means of inducing subjects to share ally pay attention and properly engage the the preferences of actors in the theoreti- tasks that they are meant to perform. In set- cal model. In economics experiments, this is tings where a “right answer” is both defin- done through the use of monetary incentives able and, at least in principle, achievable for subjects. by the subject – a setting very unlike the It is instructive to highlight the dif- Druckman and Nelson (2003) article cited ference between this approach and typical previously – financial inducements can help research methods in the psychological tra- fulfill this role. For example, in a survey dition. In political psychology experiments, experiment on political knowledge, Prior and direct inquiry into the nature of individ- Lupia (2008) find that monetary rewards ual motivations, preferences, and opinions motivate subjects to respond more accurately is often the goal. In contrast, for the pur- and to take more time considering their poses of testing a game-theoretic model, responses. This result suggests that financial economics experiments generally prefer to inducements can sometimes help elicit more control for individual motivations by manip- accurate measures of knowledge and reduce ulating them exogenously, to the extent that levels of noise in survey responses. this is possible. By controlling for prefer- A natural, and related, setting for the ences using monetary incentives, experimen- use of such methods in political experiments tal economists attempt to focus on testing involves the study of political communica- other aspects of their theoretical models, tion. Scholars want to understand how indi- such as whether actors make choices that Economics versus Psychology Experiments 63 are consistent with a model’s equilibrium Does the Scale of Monetary Incentives predictions, or the extent to which actors’ Matter? cognitive skills enable them to make the optimal choices predicted by theory. If an experimentalist decides that motivating subjects with monetary incentives is appropriate for his or her study, one basic ques- Monetary Incentives as a Means tion of implementation involves the appro- of Measuring Social Preferences priate scale for monetary incentives. It is not Finally, it might also be noted that the use unusual for experimental economics labs to of monetary incentives can be beneficial for have informal norms that subjects’ expected the study of subjects’ intrinsic motivations. earnings should not fall below some mini- Consider, for example, the Fehr and Gachter¨ mum rate of compensation; the maintenance (2000) study cited previously. Subjects inter- of a willing subject pool requires that “cus- acted within a stylized environment, mak- tomers” be reasonably happy overall with ing public goods contributions decisions and their experiences in the lab. Morton and 2010 choosing whether to punish others based on Williams ( ) summarize existing norms by their behavior. In the experiment, both kinds estimating that payments are typically struc- 50 100 of decisions were associated with monetary tured to average around to percent incentives; a decision to punish another sub- above the minimum wage for the time spent ject, for example, came at a (monetary) cost to in the lab. Such considerations aside, resource the punisher. That individuals are willing to constraints give experimentalists a natural engage in punishment, even when this has a incentive to minimize the scale of payoffs in monetary cost and when no future monetary order to maximize the amount of data that can benefit can possibly accrue, strengthens our be selected – as long as the payments that sub- sense of how strong subjects’ intrinsic motiva- jects receive are sufficient to motivate them in tions to punish may be. Certainly, this finding the necessary way. is more telling than would be a parallel result A recent voting game study by Bassi, 2007 from an analogous experiment in which sub- Morton, and Williams ( ) suggests that jects’ decisions were hypothetical and did not the scale of financial incentives can affect bear any personal material cost for punish- experimental results. In their study, the ing others. In principle, this methodology can inducements offered to subjects varied across potentially allow us to measure the strength three treatments, involving a flat fee only, a of this intrinsic motivation by varying the scale typical of many experimental economics scale of the monetary incentives. Thus, stud- studies, and a larger scale offering subjects ies such as Fehr and Gachter¨ can allow us to twice as much. The fit between subjects’ learn about individuals’ intrinsic motivations behavior and game-theoretic predictions by observing deviations from game-theoretic became monotonically stronger as incen- predictions about how completely (monetar- tives increased; suggestively, this pattern was ily) self-interested actors would behave. found to be most prominent for the most cog- Other studies have taken a similar app- nitively challenging tasks faced by subjects. roach, allowing for inquiry into tradition- These results suggest that, at least in some ally psychological topics within the context settings, higher rates of payment to subjects of game-theoretic experiments. A prominent can increase subjects’ level of attention to the example is Chen and Li (2009), who trans- experiment in a way that may affect behav- late the study of social identities into a lab ior, a result consistent with intuitions derived 2008 environment where subjects play games for from Prior and Lupia ( ), as well as related monetary incentives, thereby offering a novel studies in economics (e.g., Camerer and 1999 tool for measuring the strength of identi- Hogarth ). 2000 ties and the effects of identities on social Gneezy and Rustichini ( ) carried preferences. out a study on IQ test performance that 64 Eric S. Dickson communicates a compatible message. Their account fully for a speaker’s strategic incen- experiment varied financial incentives for tives when inferring the information content correct answers across four distinct treat- of communications. Of course, proper cali- ments. They found performance to be iden- bration of financial incentives to real-world tical in the two treatments offering the least motivations would always be an ideal. How- incentives for performance (one of which sim- ever, for a study whose central result demon- ply involved a flat show-up fee) and identi- strates “poor” performance or the existence cal in the two treatments offering the high- of a “bias” in subject behavior, confidence in est incentives, but performance in the higher external validity is likely to be stronger when incentive treatments exceeded that in the the experimenter errs on the side of making lower incentive treatments. This finding, financial incentives too large rather than too along with Prior and Lupia (2008) and Bassi small. That is, our confidence that a particu- et al. (2007), suggests that a higher scale of lar form of bias actually exists will be stronger incentives can increase attention, at least up to if it persists even when subjects have extra a point, and that higher attention can increase incentives to perform a task well in the labo- performance, at least up to a point that is ratory relative to the weaker incentives they determined in part by the difficulty of the face in real-world settings. This logic under- problem. scores the extent to which simple decisions This pattern has implications for the of experimental design may have powerful kinds of inferences that can be made from effects on the inferences we can draw from studies employing monetary incentives. The an experiment, even when the results are the nature of these implications can be reasonably same across different designs. A given finding expected to differ depending on the nature will generally be more impressive when the of the experimental findings. Consider some experimental design is more heavily stacked of the political communication studies cited against the emergence of that finding. previously. In the scenarios of Lupia and McCubbins (1998), for example, subjects are Potential Problems with Use quite good at inferring the informational con- of Monetary Incentives tent of communications they receive from strategically motivated speakers. In such As noted previously, monetary incentives may instances, confidence in a result’s external be a nonstarter for some research questions, validity may depend to some extent on the but there may be arguments in favor of their “calibration” between the financial incentives use for other research questions. Are there in play and the stakes involved in receiving potential problems with the use of monetary analogous communications in the real world. incentives that may argue against their use in The incentives offered by Lupia and McCub- certain settings? bins appear to be quite appropriate in scale. One potential issue involves interactions However, consider a counterfactual experi- between subjects’ intrinsic motivations and ment in which the monetary stakes for sub- the external motivation that they receive jects were much larger. If, in this counterfac- from financial incentives. Some research in tual experiment, subjects were substantially psychology suggests that financial incentives more motivated to pay attention and make can “crowd out” intrinsic motivations, lead- proper inferences by the monetary induce- ing to somewhat counterintuitive patterns ments in the laboratory than they would of behavior. One of the best-known exam- have been by naturalistic considerations in ples of crowding out comes from Titmuss the real world, then a clear issue would arise (1970), who showed that offering financial in extrapolating from “good” performance compensation for blood donations can lead in the laboratory to predictions about real- to lower overall contribution levels. The world performance. In contrast, in the “cheap standard interpretation is that individuals talk and coordination” scenario of Dick- who donate blood are typically motivated to son (in press), subjects systematically fail to do so for altruistic reasons; when financial Economics versus Psychology Experiments 65 incentives are offered, individuals’ mode of This section describes potential advantages engagement with the blood donation system and disadvantages of using deception from changes, with marketplace values coming to a methodological and inferential perspective. the fore while intrinsic motivations such as Ethical considerations are not discussed here altruism are crowded out. due to space limitations (for a recent review, Whether crowding out poses a problem see Morton and Williams 2010). for the use of monetary incentives is likely to depend on the nature of the research ques- Lack of Deception in tion being explored. For the purposes of game Experimental Economics theory testing, the crowding out of intrinsic motivations can actually be considered Deep-seated opposition to the use of decep- desirable because the experimenter wants to tion has become a feature of various insti- exogenously assign preferences to subjects in tutions within the economics discipline. It is order to instantiate the experimental game in common for experimental economics labora- the laboratory. In contrast, suppose that social tories to publicize and enforce bans on deceiv- interactions within some real-world setting ing subjects; in fact, a strong norm among of interest are believed to depend heavily on practitioners and journal editors makes exper- individuals’ intrinsic motivations. In translat- iments employing deception de facto unpub- ing this real-world setting into the laboratory, lishable in major economics journals. injudicious use of monetary incentives could Before describing the motivations for potentially crowd out the intrinsic motiva- these norms, it is worth describing what tions that are central to the phenomenon “deception” means, and does not mean, to being studied. experimental economists. A rough distinction This potential problem with the use of can be made between sins of commission and monetary incentives is in some instances a sins of omission. Describing features of the challenging one because it can be difficult experimental scenario in a way that is either to anticipate to what extent such incen- explicitly dishonest or actively misleading – tives might cause a transformation in sub- a sin of commission – would be straight- jects’ modes of engagement with the experi- forwardly considered a taboo act of decep- mental scenario. This concern goes hand in tion by experimental economists. In contrast, hand with understandable questions about a failure to fully describe some features of the extent to which stylized economic and the experimental scenario – a sin of omis- contextually rich psychological experiments sion – would not necessarily be counted as a actually investigate the same cognitive mech- deceptive act. As Hey (1998) puts it, “There anisms, an important and understudied mat- is a world of difference between not telling ter that may be illuminated more thoroughly subjects things and telling them the wrong in the future by across-school collaborations, things. The latter is deception, the former is as well as by neuroscientific and other frontier not” (397). Thus, in several studies of pub- research methods. lic goods provision, experimentalists employ a “surprise restart,” in which a second, previously unannounced public goods game is 3. Use of Deception played after the completion of the first. As long as subjects are not actively misled by the In few regards is the difference between the wording of the experimental protocol, such economic and psychological schools as stark a procedure is not considered to be decep- as in attitudes about deceiving subjects. The tive. And, of course, few scholars would argue more-or-less consensus view on deception in that it is necessary to explicitly inform sub- the experimental economics subfield is sim- jects about the purpose of the study in which ple: just don’t do it. In contrast, deception has they are taking part. been and remains fairly commonplace within What arguments do experimental econo- the political psychology research tradition. mists present against the use of deception? 66 Eric S. Dickson

Both Bonetti (1998) and Morton and researchers indeed do, as Hey fears, develop Williams (2010) cite Ledyard (1995) as offer- a reputation for employing deception in their ing a standard line of reasoning: experiments, then subjects may develop heterogeneous beliefs about what is really going It is believed by many undergraduates that on in the laboratory – while also being aware psychologists are intentionally deceptive in that other subjects are doing the same. At the most experiments. If undergraduates believe end of the day, subjects could effectively find the same about economists, we have lost control. It is for this reason that modern exper- themselves playing a wholly different game imental economists have been carefully nur- than the one the experimenter had intended. turing a reputation for absolute honesty in The conjectures within subjects’ minds about all their experiments. . . . [i]f the data are to the true nature of the game would, of course, be valid. Honesty in procedures is absolutely be essentially unknowable not only to one crucial. Any deception can be discovered and another, but also to the analyst. contaminate a subject pool not only for the Ledyard’s (1995) opinion also reflects experimenter but for others. Honesty is a a common viewpoint among experimental methodological public good and deception is not economists, namely, that a lab can bene- contributing. (134) fit from maintaining a reputation for trans- At the heart of this case is the fear that the use parency with its subject pool. Such a reputa- of deception will lead to a loss of experimen- tion, it is argued, could quickly be squandered tal control; as we have seen, many features if deception takes place in the laboratory; the of economics-style experimentation, includ- subject pool may become “tainted” with sub- ing the use of stylized experimental scenar- jects who have themselves experienced decep- ios and monetary incentives, are designed to tion or who have been told about it by friends. help maintain experimental control of differ- This argument is reasonable, but the ques- ent kinds. Hey (1991) articulates the specific tion it bears on is ultimately an empirical nature of this concern: one. Relatively little systematic research has explored this point, but there is some evi- [I]t is crucially important that economics dence that the experience of deception in experiments actually do what they say they the laboratory may affect individual subjects’ do and that subjects believe this. I would not propensities to participate in future exper- like to see experiments in economics degenerate iments as well as their behavior in future to the state witnessed in some areas of experi- experiments (Jamison, Karlan, and Schechter mental psychology where it is common knowl- 2008). To my knowledge, there has been no edge that the experimenters say one thing and systematic research into a related issue: the do another . . . [O]nce subjects start to distrust the experimenter, then the tight control that extent to which experimental economics lab- is needed is lost. (171–73) oratories who do ban deception actually attain the reputations to which they aspire – that This kind of concern about experimental is, to what extent subjects are aware of lab control is quite natural given the typical policies on deception in general or actually nature of research questions in experimental believe that they are never being deceived economics. As noted previously, most eco- while taking part in particular experiments in nomics experiments either test the predic- “no deception” labs. Economists’ arguments tions of game-theoretic models or explore about the sanctity of subject pools further the nature of behavior in game-theoretic tend to presuppose that psychology depart- settings. Crucially, the most common con- ments do not exist, or at least that they draw cepts of equilibrium in games, from which from a disjointed set of participants. If psy- predictions are derived, assume that actors chology and economics labs operate simulta- share common knowledge about basic fea- neously at the same university, to what extent tures of the game being played. Of course, do undergraduate subjects actually perceive experimental subjects learn about “the rules them as separate entities with distinct repu- of the game” through the experimenter. If tations? Does the physical proximity of the Economics versus Psychology Experiments 67 labs to each another affect subject percep- on whether that stimulus is framed as being tions (e.g., if they are in the same build- “real” as opposed to hypothetical. If the ing as opposed to different buildings)? It answer to this question is “yes” – and if this would appear that such questions remain to would make a substantial enough difference be answered. for measurements of the quantities of interest – then at the least a benefit from deception will have been identified. Ultimately, of Use of Deception in Experimental course, in any given setting it is an empiri- Political Psychology cal question whether the answer will be “yes” In contrast, the use of deception is quite com- or “no.” To my knowledge, however, no sys- mon in political psychology, as it is in social tematic studies have been carried out measur- psychology. As we have seen, the reasons for ing the effects, if any, of choosing deceptive this difference can be understood as spring- as opposed to explicitly hypothetical experi- ing from the distinctive natures of inquiry mental scenarios. and theory testing in the two schools. Impor- Taking Druckman and Nelson’s (2003) tantly, the ability to induce common knowl- design as an example, though, it at least seems edge of an experimental scenario within a plausible that the difference may sometimes group of subjects is usually not nearly so cru- be considerable. An individual picking up cial for experiments in the political psychol- what he or she believes to be an article from ogy tradition, which typically do not involve the New York Times will respond to frames tests of game-theoretic models. This subsec- and other cues in a way that depends directly tion reconsiders the advantages and disadvan- on his or her relationship with the New York tages of deception in the context of political Times – his or her sense of the newspaper’s psychology research questions. reliability, the fit of its ideology with his or One prominent class of examples can be her own, and so forth. In contrast, a hypo- found in the study of political communi- thetical exercise of the form “suppose the cation, in which scholars quite frequently New York Times reported ...”could insert in present subjects with stimuli that are fab- the subject’s mind a mysterious intermediary ricated or falsely attributed. Thus, Brader between the newspaper and the subject. Who (2005) presents experimental political adver- is it that is doing this supposing, and what tisements to subjects as though they were are they up to? Alternatively, the subject may genuine ads from a real, ongoing campaign; simply attend differently to the article, paying meanwhile, Druckman and Nelson (2003) it less heed or greeting it with less trust, if he present experimental newspaper stories to or she knows from the offset that it is a fic- subjects as though they came from well- tion. Under such circumstances, it would not known outlets such as the New York Times. be unreasonable to suppose that a given arti- In the following paragraphs, I use these cle might have less of an effect than it would articles as examples in discussing poten- have had it been described as a “real” arti- tial advantages of deception. Throughout, cle. Although economically inclined schol- I take as the salient alternative an other- ars might tend to doubt whether experiments wise identical experimental design in which employing deception can ever gain a full mea- the same stimuli are presented to subjects, sure of experimental control, it is arguable in but explicitly labeled as “hypothetical” cam- this setting that more control might be lost paign ads, newspaper stories, etc. Of course, with an explicitly hypothetical stimulus than in certain circumstances, different counter- with a deceptive one. Whether this is true, factual designs might also be reasonably of course, depends on the extent to which considered. subjects were actually successfully deceived. In judging the potential usefulness of This, however, is the sort of question that can deception, then, a natural question to ask often be addressed through the use of simple is whether an individual’s mode of psycho- manipulation checks by the experimenter. At logical engagement with a stimulus depends least in this example, the treatment effects 68 Eric S. Dickson in Druckman and Nelson’s findings strongly believe – at least to a considerable extent – suggest that the deceptive manipulation did that their actions were causing bodily harm indeed have the desired effect on subjects. to another human. An otherwise comparable In a similar way, it seems plausible that study involving an explicitly hypothetical sce- deception may be a useful element of Brader’s nario would, for obvious reasons, have been (2005) design. In part, this is arguable because far less convincing, even if it yielded the same of the nature of some of Brader’s dependent results. It could be easily argued that Mil- variables. Among other things, Brader shows gram’s (1974) act of deception was central to that the use of music in contrived political the lasting influence of Milgram’s study. advertising can affect subjects’ self-reported level of inclination to seek more information about an election campaign, whereas the References idea of asking subjects to report their level of inclination to seek more information about a Bassi, Anna, Rebecca Morton, and Kenneth hypothetical campaign that means nothing to Williams. 2007. “The Effects of Identities, them seems straightforwardly problematic. Incentives, and Information on Voting.” Paper These examples suggest that deception presented at the Political Economy Seminar may offer access to certain research ques- Series, Kellogg School of Management, North- tions that would remain inaccessible in its western University, Evanston, IL. Retrieved absence. Psychologists also claim that decep- from www.kellogg.northwestern.edu/research/ 0502072 tion may be necessary at times to conceal fordcenter/documents/morton .pdf (October 30, 2010). the purpose of an experiment from subjects 1998 (Bortolotti and Mameli 2006), and they are Bonetti, Shane. . “Experimental Economics and Deception.” Journal of Economic Psychology frequently concerned about the possibility of 19: 377–95. “Hawthorne effects,” through which subjects Bortolotti, Lisa, and Matteo Mameli. 2006. attempt to meet what they perceive to be “Deception in Psychology: Moral Costs the experimenter’s expectations. Such effects and Benefits of Unsought Self-Knowledge.” can be particularly worrisome in sensitive Accountability in Research 13: 259–75. research areas, such as the study of racial pol- Brader, Ted. 2005. “Striking a Responsive Chord: itics. How Political Ads Motivate and Persuade Vot- Finally, it could be argued that the use ers by Appealing to Emotions.” American Jour- 49 388 405 of deception can sometimes strengthen the nal of Political Science : – . 2003 inferences that are possible from a given piece Camerer, Colin F. . Behavioral Game Theory. of research. Among the most famous exper- Princeton, NJ: Princeton University Press. Camerer, Colin F., and Robin M. Hogarth. iments in social psychology is the seminal 1999 1974 . “The Effects of Financial Incentives in Milgram ( ) experiment on obedience and Experiments: A Review and Capital-Labor- authority. In the experiment, subjects were Production Framework.” Journal of Risk and deceived into believing that they could, with Uncertainty 19: 7–42. the twist of a knob, deliver electric shocks of Chen, Yan, and Sherry Xin Li. 2009. “Group Iden- increasing magnitude to another person; an tity and Social Preferences.” American Economic authority figure urged subjects to deliver such Review 99: 431–57. shocks in the context of a staged scenario. In Dickson, Eric S. in press. “Leadership, Follower- the end, a large fraction of subjects did con- ship, and Beliefs about the World: An Experi- form to the authority figure’s commands, to ment.” British Journal of Political Science. the point of delivering highly dangerous volt- Dickson, Eric S., Catherine Hafer, and Dim- itri Landa. 2008. “Cognition and Strategy: A ages. Deliberation Experiment.” Journal of Politics 70: This is a rather shocking result, one 974–89. that had a profound effect on the study of Druckman, James N., and Kjersten R. Nelson. authority specifically and on social psychol- 2003. “Framing and Deliberation: How Cit- ogy more generally. Its power, of course, izens’ Conversations Limit Elite Influence.” comes from our sense that subjects really did American Journal of Political Science 47: 729–45. Economics versus Psychology Experiments 69

Fehr, Ernst, and Simon Gachter.¨ 2000. “Cooper- Lupia, Arthur, and Mathew D. McCubbins. 1998. ation and Punishment in Public Goods Exper- The Democratic Dilemma: Can Citizens Learn iments.” American Economic Review 90: 980–94. What They Need to Know? Cambridge: Cam- Gneezy, Uri, and Aldo Rustichini. 2000.“Pay bridge University Press. Enough or Don’t Pay at All.” Quarterly Journal McDermott, Rose. 2002. “Experimental Method- of Economics 115: 791–810. ology in Political Science.” Political Analysis 10: Griggs, Richard A., and James R. Cox. 1982.“The 325–42. Elusive Thematics Material Effect in Wason’s Milgram, Stanley. 1974. Obedience to Authority. Selection Task.” British Journal of Psychology 73: New York: Harper & Row. 407–20. Morton, Rebecca B., and Kenneth C. Williams. Hey, John D. 1991. Experiments in Economics. 2010. Experimental Political Science and the Study Oxford: Blackwell. of Causality: From Nature to the Lab. New York: Hey, John D. 1998. “Experimental Economics and Cambridge University Press. Deception: A Comment.” Journal of Economic Prior, Markus, and Arthur Lupia. 2008.“Money, Psychology 19: 397–401. Time, and Political Knowledge: Distinguish- Jamison, Julian, Dean Karlan, and Laura ing Quick Recall and Political Learning Skills.” Schechter. 2008. “To Deceive or Not to American Journal of Political Science 52: 169–83. Deceive: The Effect of Deception on Behavior Tajfel, Henri, Michael Billig, R. Bundy, and in Future Laboratory Experiments.” Journal of Claude L. Flament. 1971. “Social Catego- Economic Behavior and Organization 68: 477–88. rization and Inter-Group Behavior.” European Kagel, John H., and Alvin E. Roth. 1995. The Journal of Social Psychology 1: 149–77. Handbook of Experimental Economics. Princeton, Tajfel, Henri, and John Turner. 1986. “The Social NJ: Princeton University Press. Identity Theory of Intergroup Behavior.” In Ledyard, John O. 1995. “Public Goods: A Survey Psychology of Intergroup Relations, eds. Stephen of Experimental Research.” In The Handbook Worchel and William G. Austin. Chicago: of Experimental Economics, eds. John H. Kagel Nelson-Hall. and Alvin E. Roth. Princeton, NJ: Princeton Titmuss, Richard M. 1970. The Gift Relationship: University Press, 111–94. From Human Blood to Social Policy. London: Levine, David K., and Thomas R. Palfrey. 2007. George Allen & Unwin. “The Paradox of Voter Participation: A Labo- Wason, P. C. 1968. “Reasoning about a Rule.” ratory Study.” American Political Science Review Quarterly Journal of Experimental Psychology 20: 101: 143–58. 273–81.

Part II

THE DEVELOPMENT OF EXPERIMENTS IN POLITICAL SCIENCE

CHAPTER 6

Laboratory Experiments in Political Science

Shanto Iyengar

Until the mid-twentieth century, the disci- more, many political scientists viewed experi- pline of political science was primarily qual- ments – which typically necessitate the decep- itative – philosophical, descriptive, legalistic, tion of research subjects – as an inherently and typically reliant on case studies that failed unethical methodology. to probe causation in any measurable way. The bias against experimentation began The word “science” was not entirely apt. to weaken in the 1970s when the emerging In the 1950s, the discipline was trans- field of political psychology attracted a new formed by the behavioral revolution, spear- constituency for interdisciplinary research. headed by advocates of a more social sci- Laboratory experiments gradually acquired entific, empirical approach. Even though the aura of legitimacy for a small band experimentation was the sine qua non of of scholars working at the intersection of research in the hard sciences and in psychol- the two disciplines.1 Most of these schol- ogy, the method remained a mere curiosity ars focused on the areas of political behav- among political scientists. For behavioralists ior, public opinion, and mass communication, interested in individual-level political behav- but there were also experimental forays into ior, survey research was the methodology the fields of international relations and public of choice on the grounds that experimenta- choice (Hermann and Hermann 1967;Riker tion could not be used to investigate real- 1967). Initially, these researchers faced signif- world politics (for more detailed accounts icant disincentives to applying experimental of the history of experimental methods in political science, see Bositis and Steinel 1987; 1 An important impetus to the development of politi- Kinder and Palfrey 1993; Green and Gerber cal psychology was provided by the Psychology and 2003 Politics Program at Yale University. Developed by ). The consensus view was that labo- Robert Lane, the program provided formal training ratory settings were too artificial and that in psychology to political science graduate students experimental subjects were too unrepresen- and also hosted postdoctoral fellows interested in pursuing interdisciplinary research. Later directors tative of any meaningful target population of this training program included John McConahay for experimental studies to be valid. Further- and Donald Kinder.

73 74 Shanto Iyengar methods – most important, research based on tus of experiments in political science per- experiments was unlikely to see the light of sisted throughout the 1970s. Observational day simply because there were no journals or methods, most notably, survey research, conference venues that took this kind of work dominated experimentation even among the seriously. practitioners of political psychology. One The first major breakthrough for political obvious explanation for the slow growth rate scientists interested in applying the experi- in experimental research was the absence mental method occurred with the founding of of necessary infrastructure. Experiments are the journal Experimental Study of Politics (ESP) typically space, resource, and labor intensive. in 1970. The brainchild of the late James Laboratories with sophisticated equipment or Dyson (then at Florida State University) and technology and trained staff were nonexis- Frank Scioli (then at Drew University and tent in political science departments, with one now at the National Science Foundation), notable exception, namely, the State Univer- ESP was founded as a boutique journal ded- sity of New York (SUNY) at Stony Brook. icated exclusively to experimental work. The When SUNY–Stony Brook was estab- coeditors and members of their editorial lished in the early 1960s, the political sci- board were committed behavioralists who ence department was given a mandate to were convinced that experiments could con- specialize in behavioral research and exper- tribute to more rigorous hypothesis testing imental methods. In 1978, the department and thereby to theory building in political moved into a new building with state-of- science (F. Scioli, personal communication, the-art experimental facilities, including lab- January 22, 2009). As stated by the editors, oratories for measuring psychophysiological the mission of the journal was to “provide an responses (modeled on the psychophysiol- outlet for the publication of materials deal- ogy labs at Harvard), cognitive or informa- ing with experimental research in the short- tion processing labs for tracking reaction est possible time, and thus to aid in rapid time, and an array of social psychological dissemination of new ideas and developments labs modeled on the lab run by the eminent in political research and theory” (F. Scioli, Columbia psychologist Stanley Schachter.3 personal communication, January 22, 2009). Once these labs were put to use by the ESP served as an important, albeit spe- several prominent behavioralists who joined cialized, outlet for political scientists inter- the Stony Brook political science faculty in ested in testing propositions about voting the early 1970s (e.g., Milton Lodge, Joseph behavior, presidential popularity, mass com- Tanenhaus, Bernard Tursky, John Wahlke), munication and campaigns, or group deci- the department would play a critical role sion making. The mere existence of a jour- in facilitating and legitimizing experimental nal dedicated to experimental research (with a research.4 masthead featuring established scholars from 2 3 The social psychology laboratories included rooms highly ranked departments) provided a cred- with transparent mirrors and advanced video and ible signal to graduate students and junior fac- sound editing systems. ulty (this author included) that it might just 4 The extent of the Stony Brook political science department’s commitment to interdisciplinary be possible to publish (rather than perish) and research was apparent in the department’s hiring of build a career in political science on the basis several newly minted social psychologists. The psy- of experimental research. chologists recruited out of graduate school – none of whom fully understood, at least during their job inter- Although ESP provided an important views, why a political science department would see “foot in the door,” the marginalized sta- fit to hire them – included John Herrstein, George Quattrone, Kathleen McGraw, and Victor Otatti. Of course, the psychologists were subjected to intense 2 Scholars who played important editorial roles at ESP questioning by the political science faculty over the included Marilyn Dantico (who took over as coeditor relevance and generalizability of their research. In of the journal when Scioli moved to the National Sci- one particularly memorable encounter, following a ence Foundation), Richard Brody, Gerald Wright, job talk on the beneficial impact of physical arousal Heinz Eulau, James Stimson, Steven Brown, and on information processing and judgment, an expert Norman Luttbeg. on voting behavior asked the candidate whether he Laboratory Experiments in Political Science 75

The unavailability of suitable laboratory generalizing the results to real-world settings. facilities was but one of several obstacles fac- Second, student-based and other volunteer ing the early experimentalists. An equally subject pools were considered unrepresenta- important challenge was the recruitment of tive of any broader target population of inter- experimental subjects. Unlike the field of psy- est (i.e., registered voters or individuals likely chology, where researchers could draw on to engage in political protest). To this day, a virtually unlimited captive pool of student the problem of external validity or question- subjects, experimentalists in political science able generalizability continues to impede the had to recruit volunteer (and typically unpaid) adoption of experimentation in political sci- subjects on their own initiative. Not only did ence. this add to the costs of conducting experi- In this chapter, I begin by describing the ments, but it also ensured that the resulting inherent strengths of the experiment as a samples would be far from typical. basis for causal inference, using recent exam- In the early 1980s, experimental methods ples from my own work in political com- were of growing interest to researchers in sev- munication. I argue that the downside of eral subfields of the discipline. Don Kinder experiments – the standard “too artificial” and I were fortunate enough to receive gen- critique – has been weakened by several devel- erous funding from the National Institutes opments, including the use of more realistic of Health and the National Science Founda- designs that move experiments outside a lab- tion for a series of experiments designed to oratory environment and the technological assess the effects of network news on public advances associated with the Internet. The opinion. These experiments, most of which online platform is itself now entirely realistic were administered in a dilapidated building (given the extensive daily use of the Inter- on the Yale campus, revealed that contrary to net by ordinary individuals); it also allows the conventional wisdom at the time, network researchers to overcome the previously pro- news exerted significant effects on the viewing found issue of sampling bias. All told, these audience. We reported the full set of experi- developments have gone a long way toward mental results in News That Matters (Iyengar alleviating concerns about the validity of and Kinder 1987). The fact that The Univer- experimental research – so much so that I sity of Chicago Press published a book based would argue that experiments now represent exclusively on experiments demonstrated that a dominant methodology for researchers in they could be harnessed to address questions several fields of political science. of political significance. That the book was generally well received demonstrated that a reliance on experimental methodology was 1. Causal Inference: The no longer stigmatized in political science. Strength of Experiments By the late 1980s, laboratory experimentation had become sufficiently recognized as The principal advantage of the experiment a legitimate methodology in political sci- over the survey or other observational meth- ence for mainstream journals to regularly ods – and the focus of the discussion that publish papers based on experiments (see follows – is the researcher’s ability to iso- Druckman et al. 2006). Despite the signifi- late and test the effects of specific compo- cant diffusion of the method, however, two nents of certain causal variables. Consider key concerns contributed to continued schol- the case of political campaigns. At the aggre- arly skepticism. First, experimental settings gate level, campaigns encompass a concate- were deemed lacking in mundane realism – nation of messages, channels, and sources, all the experience of participating in an experi- of which may influence the audience, often ment was sufficiently distinctive to preclude in inconsistent directions. The researcher’s task is to identify the potential causal mech- would suggest requiring voters to exercise prior to anisms and delineate the range of their rele- voting. vant attributes. Even at the relatively narrow 76 Shanto Iyengar level of campaign advertisements, for in- were exposed to a political advertisement stance, there are an infinite number of poten- were unable, some thirty minutes later, to recall tial causal forces, both verbal and visual. What having seen the advertisement (Ansolabehere was it about the infamous “Willie Horton” and Iyengar 1998). In a more recent exam- advertisement that is believed to have moved ple, Vavreck found that nearly half of a con- so many American voters away from Michael trol group not shown a public service message Dukakis during the 1988 presidential cam- responded either that they could not remem- paign? Was it, as widely alleged during the ber or that they had seen it (Vavreck 2007; campaign, that Horton was African Ameri- also see Prior 2003). Errors of memory also can (see Mendelberg 2001)? Or was it the compromise recall-based measures of expo- violent and brutal nature of his described sure to particular news stories (see Gunther behavior, the fact that he was a convict, or 1987) or news sources (Price and Zaller 1993). something else entirely? Experiments make it Of course, because the scale of the error in possible to isolate the attributes of messages self-reports tends to be systematic (respon- that move audiences, whether these are text- dents are prone to overstate their media expo- based or nonverbal cues. Surveys, in contrast, sure), survey-based estimates of the effects of can only provide indirect evidence on self- political campaigns are necessarily attenuated reported exposure to the causal variable in (Bartels 1993;Prior2003). question. An even more serious obstacle to causal Of course, experiments not only shed light inference in the survey context is that the on treatment effects, but they also enable indicators of the causal variable (self-reported researchers to test more elaborate hypotheses media exposure in most political communi- concerning moderator variables by assessing cation studies) are typically endogenous to interactions between the treatment factors a host of outcome variables researchers seek and relevant individual difference variables. to explain (e.g., candidate preference). Those In the case of persuasion, for instance, not all who claim to read newspapers or watch tele- individuals are equally susceptible to incom- vision news on a regular basis, for instance, ing messages (see Zaller 1992). In the case differ systematically (in ways that matter to of the 1988 campaign, perhaps Democrats their vote choice) from those who attend with a weak party affiliation and strong sense to the media less frequently. This problem of racial prejudice were especially likely to has become especially acute in the after- sour on Governor Dukakis in the aftermath math of the revolution in “new media.” In of exposure to the Horton advertisement. 1968, approximately seventy-five percent of In contrast with the experiment, the inher- the adult viewing audience watched one of the ent weaknesses of the survey design for iso- three network evening newscasts, but by 2008 lating the effects of causal variables have been the combined audience for network news was amply documented. In a widely cited paper, less than thirty-five percent of the viewing Hovland (1959) identified several problem- audience. In 2008, the only people watch- atic artifacts of survey-based studies of per- ing the news were those with a keen interest suasion, including unreliable measures of in politics, whereas almost everyone else had media exposure. Clearly, exposure is a nec- migrated to more entertaining, nonpolitical essary precondition for media influence, but programming alternatives (Prior 2007). self-reported exposure to media coverage is The endogeneity issue has multiple ramifi- hardly equivalent to actual exposure. People cations for political communication research. have notoriously weak memories for politi- Consider those instances where self-reported cal experiences (see, e.g., Pierce and Lovrich media exposure is correlated with political 1982; Bradburn, Rips, and Shevell 1987). In predispositions but actual exposure is not. the Ansolabehere and Iyengar experiments This is generally the case with televised on campaign advertising (which spanned the political advertising. Most voters encounter 1990, 1992, and 1994 election cycles), more political ads unintentionally, that is, in the than fifty percent of the participants who course of watching their preferred television Laboratory Experiments in Political Science 77 programs in which the commercial breaks to watch ads during campaigns (for a gen- contain a heavy dose of political messages. eral discussion of the issue, see Gaines and Thus, actual exposure is idiosyncratic (based Kuklinski 2008). Unlike advertisements, news on the viewer’s preference for particular coverage of political events can be avoided by television programs), whereas self-reported choice, meaning that exposure is limited to exposure is based on political predispositions. the politically engaged strata. Thus, as Hov- The divergence in the antecedents of land (1959) and others (Heckman and Smith self-reported exposure has predictable con- 1995) have pointed out, manipulational consequences for research on effects. In exper- trol actually weakens the ability to generalize iments that manipulated the tone of cam- to the real world, where exposure to poli- paign advertising, Ansolabehere and Iyengar tics is typically voluntary. In these cases, it (1995) found that actual exposure to negative is important that the researcher use designs messages demobilized voters (i.e., discour- that combine manipulation with self-selected aged intentions to vote). However, on the exposure. basis of self-reports, survey researchers con- One other important aspect of experimen- cluded that exposure to negative campaign tal design that contributes to strong causal advertising stimulated turnout (Wattenberg inference is the provision of procedures to and Brians 1999). But was it recalled expo- guard against the potential contaminating sure to negative advertising that prompted effects of “experimental demand” – cues in turnout, or was the greater interest in cam- the experimental setting or procedures that paigns among likely voters responsible for convey to participants what is expected of their higher level of recall? When recall of them (for the classic account of demand advertising in the same survey was treated effects, see Orne 1962). Demand effects rep- as endogenous to vote intention and the resent a major threat to internal validity: par- effects reestimated using appropriate two- ticipants are motivated to respond to subtle stage methods, the sign of the coefficient for cues in the experimental context suggesting recall was reversed: those who recalled nega- what is wanted of them rather than to the tive advertisements were less likely to express experimental manipulation itself. an intention to vote (see Ansolabehere, Iyen- The standard precautions against experi- gar, and Simon 1999).5 Unfortunately, most mental demand include disguising the true survey-based analyses fail to disentangle the purpose of the story by providing participants reciprocal effects of self-reported exposure with a plausible (but false) description,6 using to the campaign and partisan attitudes and relatively unobtrusive outcome measures, and behaviors. As this example suggests, in cases maximizing the “mundane realism” of the where actual exposure to the treatment is less experimental setting so that participants are selective than self-reported exposure, self- likely to mimic their behavior in real-world reports may prove especially biased. settings. (I return to the theme of realism in In other scenarios, however, the tables may Section 2.) be turned and the experimental researcher In the campaign advertising experiments may actually be at a disadvantage. Actual described, for instance, the researchers exposure to political messages in the real inserted manipulated political advertisements world is typically not analogous to random assignment. People who choose to partici- 6 Of course, the use of deception in experimental research necessitates full debriefing of participants pate in experiments on campaign advertising at the conclusion of the study. Typically, participants are likely to differ from those who choose are provided with a relatively detailed account of the experiment and are given the opportunity to receive any papers based on the study data. In recent years, 5 In a meta-analysis of political advertising research, experimental procedures have become highly regu- Lau et al. (1999) concluded that experimental studies lated by university review boards in order to maximize were not more likely to elicit evidence of significant the principle of informed consent and to preclude any effects. The meta-analysis, however, combines exper- lingering effects of deception. Most informed con- iments that use a variety of designs, most of which fail sent forms, for instance, alert participants to the use to isolate the negativity of advertising. of deception in experimental research. 78 Shanto Iyengar into the ad breaks of the first ten minutes and thus as either friends or foes of the of a local newscast. Study participants were environment. This manipulation was imple- diverted from the researchers’ intent by being mented by simply substituting the word “yes” misinformed that the study was about “selec- for the word “no.” In the positive condi- tive perception of television news.” The use of tions, the script began as follows: “When a design in which the participants answered federal bureaucrats asked for permission to the survey questions only after exposure to drill for oil off the coast of California, Pete the treatment further guarded against the Wilson/Dianne Feinstein said no. . . . ” In possibility that they might see through the the negative conditions, we substituted “said cover story and infer the true purpose of yes” for “said no.” An additional substitu- the study. tion was written into the end of the ad when In summary, the fundamental advantage of the announcer stated that the candidate in the experimental approach – and the reason question would either work to “preserve” or experimentation is the methodology of choice “destroy” California’s natural beauty. Given in the hard sciences – is the researcher’s abil- the consensual nature of the issue, negativity ity to isolate causal variables that constitute could be attributed to candidates who claimed 7 the basis for experimental manipulations. In their opponent was soft on polluters. the next section, I describe manipulations The results from these studies (which fea- designed to assess the effects of negative tured gubernatorial, mayoral, senatorial, and advertising campaigns, racial cues in televi- presidential candidates) indicated that partic- sion news coverage of crime, and the physical ipants exposed to negative rather than posi- similarity of candidates to voters. tive advertisements were less likely to say they intended to vote. The demobilizing effects of exposure to negative advertising were espe- Negativity in Campaign Advertising cially prominent among viewers who did not At the very least, establishing the effects of identify with either of the two political parties negativity in campaign advertising on vot- (see Ansolabehere and Iyengar 1995). ers’ attitudes requires varying the tone of a campaign advertisement while holding all other attributes of the advertisement con- Racial Cues in Local News stant. Despite the significant increase in Coverage of Crime scholarly attention to negative advertising, As any regular viewer of television will attest, few studies live up to this minimal thresh- crime is a frequent occurrence in broad- old of control (for representative examples of cast news. In response to market pressures, survey-based analyses, see Finkel and Geer television stations have adopted a formu- 1998 1999 ; Freedman and Goldstein ; Kahn laic approach to covering crime, an approach 1999 and Kenney .) designed to attract and maintain the high- In a series of experiments conducted by est degree of audience interest. This “crime Ansolabehere and Iyengar, the researchers script” suggests that crime is invariably vio- manipulated negativity by unobtrusively lent and those who perpetrate crime are dis- varying the text (soundtrack) of an advertise- proportionately nonwhite. Because the crime ment while preserving the visual backdrop script is encountered so frequently (several 1995 (Ansolabehere and Iyengar ). The neg- times each day in many cities) in the course ative version of the message typically placed of watching local news, it has attained the sta- the sponsoring candidate on the unpopular tus of common knowledge. Just as we know side of some salient policy issue. Thus, dur- full well what happens when one walks into ing the 1990 California gubernatorial campaign between Pete Wilson (Republican) and Dianne Feinstein (Democrat), the treatment 7 Of course, this approach assumes a one-sided distribution of policy preferences and that the tone manip- ads positioned the candidates either as oppo- ulation would be reversed for experimental partici- nents or proponents of offshore oil drilling pants who actually favored offshore drilling. Laboratory Experiments in Political Science 79

Figure 6.1. Race of Suspect Manipulation a restaurant, we also know – or at least think support for punitive policies (e.g., imposition we know – what happens when crime occurs of “three strikes and you’re out” remedies, (Gilliam and Iyengar 2000). treatment of juveniles as adults, support for In a series of recent experiments, the death penalty). Given the precision of the researchers have documented the effects of design, these differences in the responses of both elements of the crime script on audience the subjects exposed to the white or black attitudes (see Gilliam et al. 1996; Gilliam, perpetrators could only be attributed to the Valentino, and Beckman 2002). For illustra- perpetrator’s race (see Gilliam and Iyengar tive purposes, I focus here on the racial ele- 2000). ment. In essence, these studies were designed to manipulate the race/ethnicity of the prin- Facial Similarity as a Political Cue cipal suspect depicted in a news report while maintaining all other visual characteristics. A consistent ﬁnding in the political science The original stimulus consisted of a typical literature is that voters gravitate to candi- local news report, which included a close-up dates who most resemble them on questions still mug shot of the suspect. The picture was of political ideology, issue positions, and party digitized, adjusted to alter the perpetrator’s afﬁliation. But what about physical resem- skin color, and then reedited into the news blance; are voters also attracted to candidates report. As shown in Figure 6.1, beginning who look like them? with two different perpetrators (a white male Several lines of research suggest that phys- and a black male), the researchers were able ical similarity in general, and facial similarity to produce altered versions of each individual in particular, is a relevant criterion for choos- in which their race was reversed, but all other ing between candidates. Thus, frequency of features remained identical. Participants who exposure to any stimulus – including faces – watched the news report in which the suspect induces a preference for that stimulus over was believed to be nonwhite expressed greater other, less familiar stimuli (Zajonc 2001). 80 Shanto Iyengar

subject “George Bush” 60:40 Blend

subject “John Kerry” 60:40 Blend Figure 6.2. Facial Similarity Manipulation

Moreover, evolutionary psychologists argue presentation, participants had their own face that physical similarity is a kinship cue, and morphed with either Bush or Kerry at a ratio there is considerable evidence that humans of sixty percent of the candidate and forty are motivated to treat their kin preferentially percent of themselves.8 Figure 6.2 shows two (see, e.g., Burnstein, Crandall, and Kitayama of the morphs used in this study. 1994; Nelson 2001). The results of the face morphing study To isolate the effects of facial similarity revealed a significant interaction between on voting preferences, researchers obtained facial similarity and strength of the partic- digital photographs of 172 registered voters ipant’s party affiliation. Among strong par- selected at random from a national Inter- tisans, the similarity manipulation had no net panel (for details on the methodology, effect because these voters were already con- see Bailenson et al. 2009). Participants were vinced of their vote choice. But weak partisans asked to provide their photographs approx- and independents – whose voting preferences imately three weeks in advance of the 2004 were not as entrenched – moved in the direc- presidential election. One week before the tion of the more similar candidate (see Bailen- election, these same participants were asked son et al. 2009). Thus, the evidence suggests to participate in an online survey of political that nonverbal cues can influence voting, even attitudes that included a variety of questions about the presidential candidates (President George W. Bush and Senator John Kerry). 8 We settled on the 60:40 ratio after a pretest study The screens for these candidate questions indicated that this level of blending was insufficient for participants to detect traces of themselves in the included photographs of the two candidates morph, but sufficient to move evaluations of the tar- displayed side by side. Within this split panel get candidate. Laboratory Experiments in Political Science 81 in the most visible and contested of political at multiple levels: the realism of the experi- campaigns.9 mental setting, the representativeness of the In short, as these examples indicate, the participant pool, and the discrepancy between experiment provides unequivocal causal evi- experimental control and self-selected expo- dence because the researcher is able to iso- sure to media presentations. late the causal factor in question, manipulate its presence or absence, and hold other Mundane Realism potential causes constant. Any observed differences between experimental and control Because of the need for tightly controlled groups, therefore, can only be attributed to stimuli, the setting in which the typical lab- the factor that was manipulated. oratory experiment occurs is often quite Not only does the experiment provide dissimilar from the setting in which sub- the most convincing basis for causal infer- jects ordinarily experience the target phe- ence, but experimental studies are also inher- nomenon. Concern over the artificial proper- ently replicable. The same experimental ties of laboratory experiments has given rise design can be administered independently by to an increased use of designs in which the researchers in varying locales with different intervention is nonobtrusive and the settings stimulus materials and subject populations. more closely reflect ordinary life.4 Replication thus provides a measure of the One approach to increasing experimen- reliability or robustness of experimental find- tal realism is to rely on interventions with ings across time, space, and relatively minor which subjects are familiar. The Ansola- variations in study procedure. behere/Iyengar campaign experiments were Since the first published reports on the relatively realistic in the sense that they phenomenon of media priming – the ten- occurred during ongoing campaigns charac- dency of experimental participants to weigh terized by heavy levels of televised adver- issues that they have been exposed to in tising (see Ansolabehere and Iyengar 1995). experimental treatments more heavily in their The presence of a political advertisement political attitudes – the effect has been repli- in the local news (the vehicle used to con- cated repeatedly. Priming effects now apply vey the manipulation) was hardly unusual to evaluations of public officials and gov- or unexpected because candidates advertise ernmental institutions; to vote choices in a most heavily during news programs. The variety of electoral contests; and to stereo- advertisements featured real candidates – types, group identities, and any number of Democrats and Republicans, liberals and con- other attitudes. Moreover, the finding has servatives, males and females, incumbents and been observed across an impressive array of challengers – as the sponsors. The materi- political and media systems (for a recent als that comprised the experimental stim- review of priming research, see Roskos- uli were either selected from actual adver- Ewoldsen, Roskos-Ewoldsen, and Carpentier tisements used by the candidates during the 2005). campaign or produced to emulate typical campaign advertisements. In the case of the latter, the researchers spliced together 2. The Issue of Generalizability footage from actual advertisements or news reports, making the treatment ads representa- The problem of limited generalizability, long tive of the genre. (The need for control made the bane of experimental design, is manifested it necessary for the treatment ads to differ from actual political ads in several important 9 Facial similarity is necessarily confounded with familiarity – people are familiar with their own faces. attributes, including the absence of music and There is considerable evidence (see Zajonc 2001)that the appearance of the sponsoring candidate.) people prefer familiar to unfamiliar stimuli. An alter- Realism also depends on the physical set- native interpretation of these results, accordingly, is that participants were more inclined to support the ting in which the experiment is adminis- more familiar-looking candidate. tered. Although asking subjects to report to 82 Shanto Iyengar a location on a university campus may suit iarity with others in the study was sub- the researcher, it may make the experience ject to unknown variation. And producing of watching television equivalent to that of experimental ads that more closely emu- visiting the doctor. A more realistic strat- lated actual ads (e.g., ads with music in egy is to provide subjects with a milieu that the background and featuring the sponsor- closely matches the setting of their home tele- ing candidate) would necessarily have intro- vision viewing environment. The fact that duced a series of confounding variables asso- the advertising research lab was configured ciated with the appearance and voice of the to resemble a typical living or family room sponsor. Despite these trade-offs, however, setting (complete with reading matter and it is still possible to achieve a high degree refreshments) meant that participants did not of experimental control with stimuli that need to be glued to the television screen. closely resemble the naturally occurring phe- Instead, they could help themselves to cold nomenon of interest. drinks, browse through newspapers and magazines, or engage in small talk with fellow 10 Sampling Bias participants. A further step toward realism concerns the The most widely cited limitation of experi- power of the manipulation (also referred to ments concerns the composition of the sub- as “experimental realism”). Of course, the ject pool (Sears 1986). Typically, laboratory researcher would like the manipulation to experiments are administered on captive pop- have an effect. At the same time, it is impor- ulations – college students who must serve as tant that the required task or stimulus not guinea pigs in order to gain course credit. overwhelm the subject (as in the Milgram College sophomores may be a convenient obedience studies, where the task of adminis- subject population for academic researchers, tering an electric shock to a fellow participant but are they comparable to “real people?”11 proved overpowering and ethically suspect). In conventional experimental research, it is In the case of the campaign advertising exper- possible to broaden the participant pool, but iments, we resolved the experimental realism at considerable cost/effort. Locating exper- versus mundane realism trade-off by embed- imental facilities at public locations and ding the manipulation in a commercial break enticing a quasirepresentative sample to par- of a local newscast. For each treatment con- ticipate proves both cost and labor intensive. dition, the stimulus ad appeared with other Typical costs include rental fees for an exper- nonpolitical ads and subjects were led to imental facility in a public area (e.g., a shop- believe that the study was about “selective ping mall), recruitment of participants, and perception of news,” so they had no incentive training and compensation of research staff to pay particular attention to ads. Overall, the to administer the experiments. In our local manipulation was relatively small, amounting news experiments conducted in Los Ange- to thirty seconds of a fifteen-minute video- les in the summer and fall of 1999, the total tape. costs per subject amounted to approximately In general, there is a significant trade-off $45. Fortunately, and as I describe, technol- between experimental realism and manipula- ogy has both enlarged the pool of potential tional control. In the aforementioned adver- participants and reduced the per capita cost tising studies, the fact that subjects were of administering an experimental study. exposed to the treatments in the company Today, traditional experimental methods of others meant that their level of famil- can be rigorously and far more efficiently administered using an online platform. Using 10 In the early days of the campaign advertising research, the Internet as the experimental site pro- the experimental lab included a remote control device vides several advantages over conventional placed above the television set. This proved to be excessively realistic because some subjects chose to 11 For further discussion of the subject recruitment issue fast forward the videotape during the ad breaks. The and implications for external validity, see Druckman device was thus removed. and Kam’s chapter in this volume. Laboratory Experiments in Political Science 83 locales, including the ability to reach diverse in political interest or greater enthusiasm for populations without geographic limitations. online games among males. Diversity is important not only to enhance The second set of comparisons assesses generalizability, but also to mount more elab- the overlap between our self-selected online orate tests of mediator or moderator vari- samples and all voting-age adults (these com- ables. In experiments featuring racial cues, for parisons are based on representative samples 12 instance, it is imperative that the study partic- drawn by Knowledge Networks ). Here the ipants include a nontrivial number of minori- evidence points to a persisting digital divide ties. Moreover, with the ever-increasing use in the sense that major categories of the pop- of the Internet, not only are the samples more ulation remain underrepresented in online diverse, but also the setting in which partici- studies. In relation to the broader adult pop- pants encounter the manipulation (surfing the ulation, our experimental participants were Web on their own) is more realistic. significantly younger, more educated, more likely to be white males, and less apt to identify as a Democrat. “Drop-In” Samples Although these data make it clear that peo- The Political Communication Laboratory ple who participate in online media experi- (PCL) at Stanford University has been ments are not a microcosm of the adult administering experiments over the Internet population, the fundamental advantage of for nearly a decade. One of the lab’s more online over conventional field experiments popular online experiments is “whack-a-pol” cannot be overlooked. Conventional experi- (http://pcl.stanford.edu/exp/whack/polm), ments recruit subjects from particular locales, modeled on the well-known whack-a-mole whereas online experiments draw subjects arcade game. Ostensibly, the game provides from across the country. The Ansolabehere/ participants with the opportunity to “bash” Iyengar campaign advertising experiments, well-known political figures. for example, recruited subjects from a par- Since going live in 2001, more than 2,500 ticular area of southern California (greater visitors have played whack-a-pol. These Los Angeles). The online experiments, in “drop-in” subjects found the PCL site on contrast, attracted a sample of subjects from their own initiative. How does this group thirty American states and several countries. compare with a representative sample of adult Americans with home access to the Inter- Expanding the Pool of Online Participants net and a representative sample of voting- age adults? First, we gauged the degree One way to broaden the online subject pool is of divergence between drop-in participants by recruiting participants from better-known and typical Internet users. The results sug- and more frequently visited Web sites. News gested that participants in the online exper- sites that cater to political junkies, for exam- iments reasonably approximated the online ple, may be motivated to increase their circu- user population, at least with respect to lation by collaborating with scholars whose race/ethnicity, education, and party identi- research studies focus on controversial issues. fication. The clearest evidence of selection Whereas the researcher obtains data that may bias emerged with age and gender. The mean be used for scholarly purposes, the Web age of study participants was significantly site gains a form of interactivity through younger and participants were more likely to which the audience may be engaged. Play- be male. The sharp divergence in age may ing an arcade game or watching a brief video be attributed not only to the fact that our clip may pique participants’ interest, thus studies are launched from an academic server encouraging them to return to the site that is more likely to be encountered by college students, but also to the general “surfing” 12 The author is grateful to Mike Dennis, Execu- tive Vice President for Government and Academic proclivities of younger users. The gender gap Research, for providing data based on samples of regis more puzzling and may reflect differences istered voters drawn by Knowledge Networks. 84 Shanto Iyengar and boosting the news organization’s online with fully representative samples. Not sur- traffic. prisingly, both firms are located in the heart In recent years, PCL has partnered with of Silicon Valley. The first is Knowledge Net- www.washingtonpost.com to expand the works, based in Menlo Park, and the sec- reach of online experiments. Studies designed ond is Polimetrix (recently purchased by the by PCL – focusing on topics of interest UK polling company YouGov), based in Palo to people who read www.washingtonpost. Alto. com – are advertised on the Web site’s pol- Knowledge Networks has overcome the itics section. Readers who click on a link problem of selection bias inherent to online advertising the study in question are sent surveys (which reach only that proportion directly to the PCL site, where they com- of the population that is both online and plete the experiment and are then returned to inclined to participate in research studies) www.washingtonpost.com. The results from by recruiting a nationwide panel via stan- these experiments are then described in a dard telephone methods. This representative newspaper story and online column. In cases panel (including more than 150,000 Ameri- where the results were especially topical (e.g., cans between the ages of sixteen and eighty- a study of news preferences showing that five years) is provided free access to the Republicans avoided CNN and NPR in favor Internet via a WebTV. In exchange, panel of Fox News), a correspondent from www. members agree to participate (on a regular washingtonpost.com hosted an online “chat” basis) in research studies being conducted session to discuss the results and answer ques- by Knowledge Networks. The surveys are tions. administered over the panelist’s WebTV. To date, the www.washingtonpost.com– Thus, in theory, Knowledge Networks can PCL collaborative experiments have suc- deliver samples that meet the highest stan- ceeded in attracting relatively large samples, dards of probabilistic sampling. In practice, at least by the standards of experimental because their panelists have an obligation to research. Experiments on especially contro- participate, Knowledge Networks also pro- versial or newsworthy subjects attracted a vides relatively high response rates (Dennis, high volume of traffic (on some days exceed- Li, and Chatt 2004). ing 500). In other cases, the rate of participa- Polimetrix uses a novel matching approach tion slowed to a trickle, resulting in a longer to the sampling problem. In essence, they period of time to gather the data. extract a quasirepresentative sample from large panels of online volunteers. The process works as follows. First, Polimetrix assem- Sampling from Online Research Panels bles a very large pool of opt-in participants Even though drop-in online samples pro- by offering small incentives for study par- vide more diversity than the typical college ticipation (e.g., the chance of winning an sophomore sample, they are obviously biased iPod). As of November 2007, the number in several important respects. Participants of Polimetrix panelists exceeded 1.5 million from www.washingtonpost.com, for instance, Americans. To extract a representative sam- included very few conservatives or Republi- ple from this pool of self-selected panelists, cans. Fortunately, it is now possible to over- Polimetrix uses a two-step sampling proce- come issues of sampling bias – assuming the dure. First, they draw a conventional random researcher has access to funding – by admin- sample from the target population of inter- istering online experiments to representative est (i.e., registered voters). Second, for each samples. In this sense, the lack of generaliz- member of the target sample, Polimetrix sub- ability associated with experimental designs is stitutes a member of the opt-in panel who is largely overcome. similar to the corresponding member of the Two market research firms have pio- target sample on a set of demographic char- neered the use of web-based experiments acteristics such as gender, age, and education. Laboratory Experiments in Political Science 85

In this sense, the matched sample consists of placed in portals or sites that are known to respondents who represent the respondents attract underrepresented groups. Female sub- in the target sample. Rivers (2006) describes jects or African Americans, for instance, could the conditions under which the matched sam- be attracted by ads placed in sites tailored to ple approximates a true random sample. their interests. Most recently, the develop- The Polimetrix samples have achieved ment of online research panels has made it impressive rates of predictive validity, thus possible to administer experiments on broad bolstering the claims that matched samples cross-sections of the American population. emulate random samples.13 In the 2005 Cal- All told, these features of web experiments ifornia special election, Polimetrix accurately go a long way toward neutralizing the gener- predicted the public’s acceptance or rejection alizability advantage of surveys. of all seven propositions (a record matched Although web experiments are clearly a by only one other conventional polling orga- low-cost, effective alternative to conventional nization) with an average error rate compara- experiments, they are hardly applicable to all ble to what would be expected given random arenas of behavioral research. Most notably, sampling (Rivers and Bailey 2009). web-based experiments provide no insight into group dynamics or interpersonal influence. Web use is typically a solitary experi- 3. Conclusion ence, and web experiments are thus entirely inappropriate for research that requires plac- The standard comparison of experiments and ing individuals in some social or group milieu surveys favors the former on the grounds of (e.g., studies of opinion leadership or confor- precise causal inference and the latter on the mity to majority opinion). grounds of greater generalizability. As I sug- A further frontier for web experimentalists gest, however, traditional experimental meth- will be cross-national research. Today, exper- ods can be effectively and just as rigorously imental work in political science is typically replicated using online strategies. Web exper- reliant on American stimuli and American iments eliminate the need for elaborate lab subjects. The present lack of cross-national space and resources; all that is needed is a variation in the subject pool makes it impossi- room with a server. These experiments have ble to contextualize American findings14 and the advantage of reaching a participant pool also means that the researcher is unable to that is more far flung and diverse than the rule out a family of alternative explanations pool relied on by conventional experimental- for any observed treatment effects having to ists. Online techniques also permit a more do with subtle interactions between culture precise targeting of recruitment procedures and treatment (see Juster et al. 2001). Hap- so as to enhance participant diversity. Ban- pily, the rapidity with which public access to ner ads publicizing the study and the finan- the web has diffused on a global basis now cial incentives for study participants can be makes it possible to launch online experiments on a cross-national basis. Fully oper- 13 The fact that the Polimetrix online samples can be ational online opt-in research panels are matched according to a set of demographic charac- already available in many European nations, teristics does not imply that the samples are unbi- including Belgium, Britain, Denmark, Fin- ased. All sampling modes are characterized by different forms of bias and opt-in web panels are no land, Germany, the Netherlands, Norway, exception. In the United States, systematic compar- and Sweden. Efforts to establish and support isons of the Polimetrix online samples with random infrastructure for administering and archiv- digit dial (telephone) samples and face-to-face interviews indicate trivial differences between the tele- ing cross-national laboratory experiments are phone and online modes, but substantial divergences under way at several universities, including from the face-to-face mode (see Hill et al. 2007; Mal- hotra and Krosnick 2007). In general, online samples 14 Indeed, comparativists are fond of pointing out the appear biased in the direction of politically engaged inherently noncomparative and hence prescientific and attentive voters. nature of research in American politics. 86 Shanto Iyengar the Nuffield Centre for Experimental Social mental Research in Political Science.” Political Sciences and the Zurich Program in the Behavior 9: 263–84. Foundations of Human Behavior.15 Isus- Bradburn, Norman M., Lance J. Rips, and Stephen 1987 pect that by 2015, it will be possible to K. Shevell. . “Answering Autobiographical Questions: The Impact of Memory and Infer- deliver online experiments to national sam- 236 157 61 ples in most industrialized nations. Of course, ence in Surveys.” Science : – . Burnstein, Eugene, Christian Crandall, and Shi- given the importance of economic develop- nobu Kitayama. 1994. “Some Neo-Darwinian ment to web access, cross-national experi- Decision Rules for Altruism: Weighing Cues ments administered online – at least in the for Inclusive Fitness as a Function of the Bio- near term – will be limited to the “most sim- logical Importance of the Decision.” Journal ilar systems” design. of Personality and Social Psychology 67: 773– In closing, it is clear that information tech- 89. nology has removed the traditional barri- Dennis, J. Michael, Rick Li, and Cindy Chatt. ers to experimentation in political science, 2004. “Benchmarking Knowledge Networks’ including the need for lab space, convenient Web-Enabled Panel Survey of Selected GSS access to diverse subject pools, and skepticism Questions against GSS In-Person Interviews.” over the generalizability of findings. The web Knowledge Networks Technical Report. Re- trieved from www.knowledgenetworks.com/ makes it possible to administer realistic exper- ganp/docs/GSS02-DK-Rates-on-KN-Panel- imental designs on a worldwide scale with a v3.pdf (October 31, 2010). relatively modest budget. Given the advan- Druckman, James N., Donald P. Green, James tages of online experiments, I expect a bright H. Kuklinski, and Arthur Lupia. 2006.“The future for laboratory experiments in political Growth and Development of Experimental science. Research in Political Science.” American Politi- cal Science Review 100: 627–35. Finkel, Steven E., and John G. Geer. 1998.“A References Spot Check: Casting Doubt on the Demobi- lizing Effect of Attack Advertising.” American Journal of Political Science 42: 573–95. Ansolabehere, Stephen D., and Shanto Iyengar. Freedman, Paul, and Kenneth Goldstein. 1999. 1995. Going Negative: How Political Ads Shrink “Measuring Media Exposure and the Effects and Polarize the Electorate. New York: Free of Negative Campaign Ads.” American Journal Press. of Political Science 43: 1189–208. Ansolabehere, Stephen D., and Shanto Iyengar. Gaines, Brian J., and James H. Kuklinski. 2008. 1998. “Messages Forgotten: Misreporting in “A Case for Including Self-Selection alongside Surveys and the Bias towards Minimal Effects.” Randomization in the Assignment of Experi- Unpublished manuscript, University of Cali- mental Treatments.” Presented at the annual fornia, Los Angeles. meeting of the Midwestern Political Science Ansolabehere, Stephen D., Shanto Iyengar, and Association, Chicago. Adam Simon. 1999. “Replicating Experiments Gilliam, Franklin, Jr., and Shanto Iyengar. 2000. Using Aggregate and Survey Data.” American “Prime Suspects: The Influence of Local Tele- Political Science Review 93: 901–10. vision News on the Viewing Public.” American Bailenson, Jeremy, Shanto Iyengar, Nick Yee, Journal of Political Science 44: 560–73. and Nathan Collins. 2009. “Facial Similarity Gilliam, Franklin, Jr., Shanto Iyengar, Adam between Candidates and Voters Causes Influ- Simon, and Oliver Wright. 1996. “Crime in ence.” Public Opinion Quarterly 72: 935–61. Black and White: The Violent, Scary World of Bartels, Larry. 1993. “Messages Received: The Local News.” Harvard International Journal of Political Impact of Media Exposure.” American Press/Politics 1: 6–23. Political Science Review 87: 267–85. Gilliam, Franklin, Jr., Nicholas A. Valentino, and Bositis, David A., and Douglas Steinel. 1987.“A Matthew Beckman. 2002. “Where You Live Synoptic History and Typology of Experi- and What You Watch: The Impact of Racial 15 A useful compilation of online experimental labs Proximity and Local Television News on Atti- can be found at http://psych.hanover.edu/research/ tudes about Race and Crime.” Political Research exponnet.html. Quarterly 55: 755–80. Laboratory Experiments in Political Science 87

Green, Donald P., and Alan S. Gerber. 2003.“The ences about Political Attitudes and Behavior: Under-Provision of Experiments in Political Comparing the 2000 and 2004 ANES to Inter- and Social Science.” Annals of the American net Surveys with Non-Probability Samples.” Academy of Political and Social Science 589: 94– Political Analysis 15: 286–323. 112. Mendelberg, Tali. 2001. The Race Card: Cam- Gunther, Barrie. 1987. Poor Reception: Misunder- paign Strategy, Implicit Messages, and the Norm standing and Forgetting Broadcast News. Hills- of Equality. Princeton, NJ: Princeton Univer- dale, NJ: Lawrence Erlbaum. sity Press. Heckman, James J., and Jeffrey P. Smith. 1995. Nelson, Charles A. 2001. “The Development of “Assessing the Case for Social Experiments.” Neural Bases of Face Recognition.” Infant and Journal of Economic Perspectives 9: 85–110. Child Development 10: 3–18. Hermann, Charles F., and Margaret G. Hermann. Orne, Martin T. 1962. “On the Social Psychology 1967. “An Attempt to Simulate the Outbreak of of the Psychological Experiment: With Partic- World War I.” American Political Science Review ular Reference to Demand Characteristics and 61: 400–16. Their Implications.” American Psychologist 17: Hill, Seth J., James Lo, Lynn Vavreck, and 776–83. John Zaller. 2007. “The Opt-In Internet Pierce, John C., and Nicholas P. Lovrich. 1982. Panel: Survey Mode, Sampling Methodology “Survey Measurement of Political Participa- and the Implications for Political Research.” tion: Selective Effects of Recall in Petition Unpublished paper, University of California, Signing.” Social Science Quarterly 63: 164–71. Los Angeles. Retrieved from http://web.mit. Price, Vincent, and John R. Zaller. 1993. “Who edu/polisci/portl/cces/material/HillLoVavreck Gets the News? Alternative Measures of Zaller2007.pdf (October 31, 2010). News Reception and Their Implications for Hovland, Carl I. 1959. “Reconciling Conﬂicting Research.” Public Opinion Quarterly 57: 133–64. Results Derived from Experimental and Survey Prior, Markus. 2003. “Any Good News in Soft Studies of Attitude Change.” American Psychol- News? The Impact of Soft News Preference on ogist 14: 8–17. Political Knowledge.” Political Communication Iyengar, Shanto, and Donald R. Kinder. 1987. 20: 149–72. News That Matters: Television and American Prior, Markus. 2007. Post-Broadcast Democracy: Opinion. Chicago: The University of Chicago How Media Choice Increases Inequality in Political Press. Involvement and Polarizes Elections. New York: Juster, Thomas F., Richard Blundell, Richard V. Cambridge University Press. Burkhauser, Graziella Caselli, Linda P. Fried, Riker, William H. 1967. “Bargaining in a Three- Albert I. Hermalin, Robert L. Kahn, Arie Person Game.” American Political Science Review Kapteyn, Michael Marmot, Linda G. Martin, 61: 642–56. David Mechanic, James P. Smith, Beth J. Soldo, Rivers, Douglas. 2006. “Sample Matching: Repre- Robert Wallace, Robert J. Willis, David Wise, sentative Sampling from Internet Panels.” and Zeng Yi. 2001. Preparing for an Aging Palo Alto, CA: Polimetrix. Retrieved from World: The Case for Cross-National Research. http://web.mit.edu/polisci/portl/cces/material/ Washington, DC: National Academy Press. sample matching.pdf (October 31, 2010). Kahn, Kim F., and Patrick J. Kenney. 1999. Rivers, Douglas, and Delia Bailey. 2009.“Infer- “Do Negative Campaigns Mobilize or Suppress ences from Matched-Samples in the U.S. Turnout? Clarifying the Relationship between National Elections from 2004 to 2008.” Pro- Negativity and Participation.” American Politi- ceedings of the Survey Research Methods Section of cal Science Review 93: 877–90. the American Statistical Association, Joint Statisti- Kinder, Donald R., and Thomas R. Palfrey. 1993. cal Meeting 2009. Experimental Foundations of Political Science. Ann Roskos-Ewoldsen, David, Beverly Roskos- Arbor: University of Michigan Press. Ewoldsen, and Francesca R. Carpentier. 2005. Lau, Richard R., Lee Sigelman, Caroline Held- “Media Priming: A Synthesis.” In Media Effects: man, and Paul Babbitt. 1999. “The Effects Advances in Theory and Research, eds. Jennings of Negative Political Advertisements: A Meta- Bryant and Dolph Zillmann. Hillsdale, NJ: Analytic Assessment.” American Political Science Lawrence Erlbaum, 97–120. Review 93: 851–75. Sears, David O. 1986. “College Sophomores in Malhotra, Neil, and Jon A. Krosnick. 2007.“The the Laboratory: Inﬂuences of a Narrow Data Effect of Survey Mode and Sampling on Infer- Base on the Social Psychology View of Human 88 Shanto Iyengar

Nature.” Journal of Personality and Social Psy- or Mobilizer?” American Political Science Review chology 51: 515–30. 93: 891–900. Vavreck, Lynn. 2007. “The Exaggerated Effects Zajonc, Robert B. 2001. “Mere Exposure: A Gate- of Advertising on Turnout: The Dangers of way to the Subliminal.” Current Directions in Self-Reports.” Quarterly Journal of Political Sci- Psychological Science 10: 224–28. ence 2: 325–43. Zaller, John R. 1992. The Nature and Origins of Wattenberg, Martin P., and Craig L. Brians. 1999. Mass Opinion. New York: Cambridge Univer- “Negative Campaign Advertising: Demobilizer sity Press. CHAPTER 7

Experiments and Game Theory’s Value to Political Science

John H. Aldrich and Arthur Lupia

In recent decades, formal models have logically with basic premises about the peo- become common means of drawing impor- ple and places involved. tant inferences in political science. Best prac- Formal models in political science have tice formal models feature explicitly stated been both inﬂuential and controversial. premises, explicitly stated conclusions, and Although no one contends that explicitly proofs that are used to support claims about stated assumptions or attention to logical focal relationships between these premises consistency are anything other than good and conclusions. When best practices are fol- components of scientiﬁc practice, contro- lowed, transparency, replicability, and logi- versy often comes from the content of for- cal coherence are the hallmarks of the formal mal models themselves. Many formal models theoretic enterprise. contain descriptions of political perceptions, Formal models have affected a broad range opinions, and behaviors that are unrealistic. of scholarly debates in political science – Some scholars, therefore, conclude that for- from individual-level inquiries about why mal models, generally considered, are of lit- people vote as they do, to large-scale stud- tle value to political science. Yet, if we can ies of civil wars and international negoti- offer a set of premises that constitutes a suit- ations. The method’s contributions come able analogy for what key political actors when answers to normative and substan- want, know, and believe, then we can use for- tive questions require precise understandings mal models to clarify conditions under which about the conditions under which a given these actors will, and will not, take particular political outcome is, or is not, consistent actions. with a set of clearly stated assumptions about In this chapter, we address two questions relevant perceptions, motives, feelings, and that are particularly relevant to debates about contexts. Indeed, many formal modelers use formal models’ substantive relevance. We mathematics to sort intricate and detailed present each question in turn and, in so doing, statements about political cause and effect by explain how experiments affect the value of the extent to which they can be reconciled formal modeling to political science.

89 90 John H. Aldrich and Arthur Lupia

One question is, “Will people who are in orists who desire broadly applicable conclu- the situations you describe in your model sions can use these findings to better integrate act as you predict?” This question is about cultural factors into their explanations. the internal validity of a model. Experi- Indeed, a key similarity between what ments permit the creation of a specialized formal modelers and experimentalists do is setting in which a model’s premises can be that neither simply observes the environ- emulated, with the test being whether the ments they want to explain. Instead, both seek experiment’s subjects behave as the model to create settings that emulate the environ- predicts (akin to physicists studying the action ments. When used in tandem, formal mod- of falling objects in a laboratory-created vac- els can help experimentalists determine which uum and thus as free from air resistance as settings are most critical to a particular causal possible). Lupia and McCubbins (1998, chap- hypothesis and experimenters can inform ter 6), for example, devote an entire chapter formal modelers by evaluating their theo- of their book to pinpointing the corre- retical predictions’ performance in relevant spondence between the experimental settings environs. and the models that the experiments were Given the emphasis on logical precision designed to evaluate. Furthermore, if a mod- in formal modeling, it is also worth noting eler wants to claim that a particular factor that many model-related experiments follow is the unique cause of an important behav- practices in experimental economics, which ior, then he or she can design treatments includes paying subjects for their participa- that vary the presence of the presumably tion as a means of aligning their incentives unique causal factor. If the focal behavior is with those of analogous actors in the formal observed only when the presumed factor is models. As Palfrey (2007a) explains, present, then he or she will have greater evidence for his or her claim. Generally speak- [R]esearchers who were trained primarily as ing, this way of thinking about the role of [formal] theorists – but interested in learning experiments in formal theoretic political sci- whether the theories were reliable – turned to laboratory experiments to test their the- ence is akin to Roth’s (1995) description ories, because they felt that adequate field of experiments as a means of “speaking to data were unavailable. These experiments had 22 theorists” ( ). three key features. First, they required the A second question is, “Are your theoreti- construction of isolated (laboratory) environ- cal predictions representative of how people ments that operated under specific, tightly will act in more realistic circumstances?” This controlled, well-defined institutional rules. question speaks to the ecological validity of Second, incentives were created for the par- models and is akin to Roth’s (1995) descrip- ticipants in these environments in a way that tion of experiments as “whispering in the matched incentives that existed for the imag- ears of princes” (22). Experimental designs inary agents in theoretical models. Third, the that address these questions should incorpo- theoretical models to be studied had precise context-free implications about behavior in rate elements that audiences would see as any such environment so defined, and these essential to proffering a substantive expla- predictions were quantifiable and therefore nation, but that are not necessarily included directly testable in the laboratory. (915) in the model. Cross-national experiments are an example of designs that address such con- For formal modelers, these attributes of cerns. As Wilson and Eckel’s chapter in this experimentation are particularly important volume explains, for models whose predic- because a comparative advantage of formal tions are not culture specific, it is important theoretic approaches is precision in causal to evaluate whether experimental subjects in language. different regions or countries will really play In the rest of this chapter, we highlight given games in identical ways. When exper- ways in which experiments have affected the iments reveal cross-cultural differences, the- value of formal modeling in political science. Experiments and Game Theory’s Value to Political Science 91

Because the range of such activities is so 1. Cooperative Game Theory broad, we focus our attention on a type of and Experiments formal modeling called “game theory.” Game theory is a way of representing interper- Game theory is often divided into coopera- sonal interactions. Premises pertain to spe- tive and noncooperative game theory, and it cific attributes of individual actors and the is true that most results can be fit into one or contexts in which people interact. Conclu- the other division. However, both formal def- sions describe the aggregate consequences of initions are at the extreme points on a contin- what these actors do, or the properties of the uum, and thus virtually the entire continuum individuals themselves, that result from inter- fits in neither category very precisely. The actions among the actors. basic distinction is that in cooperative game The next two substantive sections of this theory, coalitions may be assumed to form, chapter pertain to the two main types of whereas in noncooperative game theory, any game theory, respectively. Section 1 focuses coalitions must be deduced from the model on experiments in the domain of coopera- itself rather than assumed a priori. In the tive game theory. Political scientists have used early days of game theory, which we can mark cooperative game theory to address key the- from the publication of Von Neumann and oretical and normative debates about prefer- Morgenstern (1944) to approximately the late ence aggregation and properties of common 1970s, when the Nash-Harsanyi-Selten revo- decision rules. As the first game-theoretic lution in noncooperative game theory carried experiments in political science tested results the day (and won them a Nobel prize in 1994), from cooperative game theory, many of the this now-critical distinction was much less standard protocols of experimental game the- important. Theoretical results, for example, ory were developed in this context. Section were not often identified as one or the other, 2 focuses on experiments in the domain of and theorists (including Nash [1997]) moved noncooperative game theory. Most game- back and forth across this division easily. theoretic treatments of political science top- This early blurring of the distinction ics today use some form of noncooperative between the two is relevant here because game theory. This type of theory can clar- experimental game theory began very early ify how actors pursue their goals when they in the history of game theory, and experi- and those around them have the ability to ments would at times include and intermingle perceive and adapt to important attributes results from cooperative and noncooperative of their environment. Influential models of game designs. This is perhaps most evident in this kind have clarified how institutions the truly vast experimental literature on pris- affect individual choices and collective out- oner’s dilemma (PD) games. The PD played comes, and how strategic uses of communi- an important role in game theory from the cation affect a range of important political beginning because it was immediately evi- outcomes. dent that the strong prediction of rational In the conclusion, we speak briefly about players both (or all) defecting with or with- how experiments may affect future relation- out communication being possible was sim- ships between formal modeling and political ply empirically false. Whether presented as science. We argue that, although the psycho- a teaching device in an introductory under- logical realism of many current models can be graduate course or tested in the most sophis- questioned, research agendas that integrate ticated experimental setting, people simply experimental and formal modeling pursuits do not follow the predictions of game the- provide a portal for more effective interdisci- ory (particularly noncooperative game the- plinary work and can improve the applicabil- ory) in this regard. Therefore, game theorists ity and relevance of formal models to a wide naturally turned to experimentation to study range of important substantive questions in play by actual players to seek theoretical political science. insight (see, e.g., Rapoport and Chammah 92 John H. Aldrich and Arthur Lupia

[1965] for a review of many early PD game cepts of game theory as it was known at the experiments). This case is thus very similar time. It is fair to say, from a game-theoretic to the experimental work on the “centipede perspective, that many of the new assump- game,” which is discussed in Section 2, in that tions needed to characterize coalition behav- both endeavors led scholars to question key ior were simply arbitrary. Third, many of assumptions and develop more effective mod- these new ways of characterizing coalition eling approaches. behaviors offered vague (as in many outcomes predicted) or unhelpful (as in nonex- istent) conclusions. Moreover, the diverse set Coalition Formation of solution concepts often had overlapping As noted, game theorists conducted experi- predictions, making it difficult to distinguish ments from the earliest days of game theory between them in observational data. (see, e.g., Kalish et al. [1954], on which John So, although the multiplicity of solution Nash was a coauthor). A common application concepts was part of an effort to clarify impor- was to the question of coalitions. Substan- tant attributes of coalition behavior, an aggre- tively, Riker (1962) argued for the central- gate consequence of such efforts was more ity of coalitions for understanding politics. confusion. Given his interest in the topic, Theoretically, cooperative game theory was and his substantive claim of the centrality unusually fecund, with a diverse set of n- of coalitions to politics, it is not surprising person games and many different solution that the first published attempt to simu- concepts whose purposes were to characterize late a game-theoretic context in a labora- the set of coalitions that might form. tory setting was by William Riker (1967; This set of solution concepts has three Riker and Zavoina 1970). These simulations notable attributes. First, one concept, called were important for several reasons. First, the core, had the normatively attractive prop- they established what became a standard pro- erty that outcomes within it did not stray too tocol for conducting game-theoretic exper- far from the preferences of any single player. iments. Preferences were typically induced In other words, the core was a set of outcomes by money. Communications between play- that were preferred by majorities of voters and ers were either controlled as carefully as pos- not easily overturned by attempts to manipu- sible by the experimenter or else were care- late voting agendas. Unfortunately, such core fully and, as far as technology permitted, fully outcomes have the annoying property of not recorded. Assumptions were built into the existing for a very large class of political research design to mimic the assumptions of contexts. As Miller’s chapter in this volume the solution concept being examined. Behav- details, the general nonexistence of the core iors were closely observed, with the observa- raises thorny normative questions about the tional emphasis being on whether the coali- meaning and legitimacy of majority decision tions chosen were consistent with the concept making. If there is no structure between the or concepts being tested. Lessons learned preferences of individuals and the choices that were also reported in the text. For exam- majority coalitions make, then it becomes ple, Riker discovered that students learned difficult to argue that preference-outcome that they could, in effect, deceive the experi- links (e.g., the will of the majority) legitimate menter by agreeing outside (and in advance) majority decision making. of the simulation to exchange their univer- To address these and other normative con- sity’s food cards as a way of reaching binding cerns, scholars sought other solution con- agreements. This outcome taught the lesson cepts that not only described coalitional that not only is the repeated use of the same choices in cases where the core did not exist, subjects potentially problematic, but also that but also retained some of the core’s attractive subjects – and therefore, at least potentially, properties. Second, theorists often developed people in real situations – will devise strate- these alternate concepts by adding assump- gies to make binding commitments even tions that did not flow from the basic con- when the situation precludes them formally. Experiments and Game Theory’s Value to Political Science 93

The proliferation of solution concepts, tive solution’s] predictions correspond for- such as those we describe, motivated many tuitously to some more general solution interesting and important extensions to the notion” (165). To see why this statement setting. With so many overlapping predic- is prescient, it is important to note that tions, observational data were often of little the experimental protocol was developed to help in selecting among competing accounts. allow researchers to evaluate various solu- For example, where Riker (1962) developed a tion concepts on the basis of coalition-level theory of minimal winning coalitions, virtu- outcomes. The solution concepts they eval- ally every other cooperative solution concept uated, however, derived their conclusions that focused on coalition formation was also about coalition-level outcomes from specific consistent with the observation of minimal assumptions regarding how players would winning coalitions. This overlap made it dif- actually bargain. McKelvey and Ordeshook ficult to distinguish the power of various solu- soon began to use their experimental proto- tion concepts. Riker’s research designs varied col to evaluate the status of these assump- the construction of subject preferences – pref- tions (i.e., the experiments allowed them to erences that one could induce with money – observe how subjects actually bargained with to distinguish competing causal mechanisms. one another). In another of their papers (Mc- These distinctions, in turn, clarified which Kelvey and Ordeshook 1983), they reported solution concepts were and were not viable in on experiments that once again produced the important cases. By this careful construction, results predicted by their competitive solu- Riker could distinguish through lab design tion. However, they also observed players what was difficult to distinguish in real-world bargaining in ways that appeared to vio- settings. late the assumptions of the competitive solu- McKelvey and Ordeshook’s research also tion. Such observations ultimately led them captured theorists’ attention; they developed to abandon the competitive solution research one of the last cooperative game-theoretic agenda for many years (Ordeshook 2007) and solution concepts, called the “competitive to devote their attention to other explana- solution” (McKelvey and Ordeshook 1978, tions of collective behavior (see Section 2 of 1979, 1980, 1983; McKelvey, Ordeshook, and this chapter, as well as Morton and Williams’ Winer 1978; Ordeshook 2007). They were chapter in this volume, for descriptions of that very interested in evaluating their concept work). experimentally vis-a-vis` other solution con- This idea of using the lab setting as a way of cepts. Their 1979 paper reports on a series sorting among solution concepts reached an of experiments that was able to establish apogee in Fiorina and Plott (1978), in which predictive differences among nine solution they develop sixteen different sets of theoret- concepts (some of which had noncoopera- ical predictions. Some are from cooperative tive game-theoretic elements). In particular, game theory and others from noncooperative they used the data to proffer statistical esti- game theory. Some focus on voting, whereas mates of likelihoods that revealed support others focus on agenda control. Some are not for two of the concepts (theirs and the min- even based in rational actor theories. The imax set of Ferejohn, Fiorina, and Packel beauty of their use of the lab setting (and [1980]), while effectively rejecting the other their own creativity) is that they are able to seven. design experiments that allow for compet- McKelvey and Ordeshook (1979) ended ing tests between many of the pairs of the- their paper with a conclusion that would turn ories, and sometimes tests that uniquely dis- out to be prescient. They write, “Finally, we criminate one account from virtually all oth- cannot reject the hypothesis that [the com- ers. In addition to differentiating outcomes petitive solution] succeeds here for the same in this way, they examined the effect of vary- reason that [other solutions] receive sup- ing treatments – in particular, whether the port in earlier experiments entailing trans- payoffs were relatively high or low to the ferable utility – namely that [the competi- players and whether there was open exchange 94 John H. Aldrich and Arthur Lupia of communication or no communication at all draw logically transparent conclusions about among the players. how individuals adapt and react to the antici- They found that when there is a core (a pated strategic moves of others. It uses the majority-preferred outcome that is not easily Nash equilibrium concept, or well-known undone by agenda manipulation), it is almost refinements of the concept, as a criterion for always chosen. High payoffs yielded out- identifying behavioral predictions. In recent comes even closer to those predicted points decades, noncooperative games using the than lower payoffs, and, to a much lesser extensive form have been formal modelers’ extent, communication facilitated that out- primary instrument in attempting to make come. Comparing these outcomes to those contributions to political science. The exten- where no core exists allows the authors to sive form outlines, in order, the decisions to conclude more sharply that it is indeed the be reached, actor by actor, from the opening existence of a core that drives the results. In move to the final outcome. It thus offers a rich contrast, the absence of a core also yielded perspective for analyzing strategic decision an apparent structure to the set of coalition- making, as well as the roles of beliefs and com- level outcomes rather than an apparently munication in decision making. These games unpredictable set of results, as might have have informed our discipline’s attempts to been expected by some readings of Mc- clarify the relationship among political insti- Kelvey’s (1976) and Schofield’s (1978)the- tutions, individual choices, and collective out- oretical results about the unlimited range of comes. They have also been the means by possible outcomes of majority decision mak- which scholars have examined positive and ing in the absence of a core. Rather (as, normative implications of the strategic use indeed, McKelvey argued in the original 1976 of information (see Austen-Smith and Lupia article), there seems to be structure to what [2007] for a recent review). As the field coalitions do, even in the absence of a core. evolves, these games increasingly serve as a Hence, the correspondence between individ- portal through which implications of substan- ual intentions and coalition-level outcomes in tive premises from fields such as economics the absence of a core is not totally chaotic and psychology can become better under- or unstable, and this correspondence pro- stood in political contexts. vides a basis for making normatively appeal- Key moments in the evolution of such ing claims about the legitimacy of majority understandings have been experimental eval- rule (see Frohlich and Oppenheimer [1992] uations of these games. In this section, we for a broader development of links among review three examples of where the com- models, experiments, and important norma- bination of noncooperative game-theoretic tive considerations). However, these struc- models and experiments has produced new tures, although observed by Fiorina and Plott insights about important social scientific mat- (1978), were not the result of any theory then ters. The examples are voter competence, jury established. Hence, one of the lasting con- decision making, and the centipede game. tributions of the work by Fiorina and Plott, as is true of the other work described in this Voter Competence section, is that it set the stage for other experiments on political decision making, such as A conventional wisdom about mass politics those discussed in Section 2 and by Morton is that candidates and their handlers seek and Williams’ chapter in this volume. to manipulate a gullible public and that the public makes inferior decisions as a result (see, e.g., Converse 1964). In recent decades, 2. Noncooperative Game Theory scholars have used formal models and exper- and Experiments iments in tandem to examine when seemingly uninformed voters do – and do not – Noncooperative game theory is a method of make inferior decisions. In this section, we formal modeling that allows researchers to review two examples of such work. In each Experiments and Game Theory’s Value to Political Science 95 case, scholars use formal models to under- McKelvey and Ordeshook (1990) evaluate stand whether claims about the manipulabil- key aspects of their theoretical work experi- ity of voters are, and are not, consistent with mentally. As Palfrey (2007a, bracketed caveat clearly stated assumptions about voters’ and inserted by authors) reports, candidates’ incentives and knowledge. Exper- iments then clarify the extent to which sub- Perhaps the most striking experiment . . . used a single policy dimension, but candidates had jects will act in accordance with focal model no information about voters and only a few predictions when they are placed in decision- of the voters in the experiments knew where making environments that are similar to the the candidates located. The key information ones described in the models. transmission devices explored were polls and McKelvey and Ordeshook (1990) focus on interest group endorsements. In a theoreti- a spatial voting model in which two can- cal model of information aggregation, adapted didates compete for votes by taking policy from the rational expectations theory of mar- positions on a unidimensional policy space. kets, they proved that this information alone Voters have spatial preferences, which is to [along with the assumption that voters know say that they have an ideal point that repre- approximately where they stand relative to the sents the policy outcome they most prefer. A rest of the electorate on a left-right scale] is sufficient to reveal enough to voters that even voter in these models obtains higher utility uninformed voters behave optimally – i.e., as when the candidate whose policy preference if they were fully informed. (923) is closest to his or her ideal point is elected. If the game were one of complete infor- Of course, uninformed voters in the Mc- mation, then the outcome would be that Kelvey-Ordeshook (1990) model do not cast both candidates adopt the median voter’s informed votes in all circumstances. The ideal point as their policy preference and the caveat inserted into the Palfrey quote high- median voter’s ideal point becomes the pol- light key assumptions that contribute to the icy outcome (Black 1948). The focal research stated result. However, it is important to question for McKelvey and Ordeshook remember that the conventional wisdom at (1990) is how such outcomes change when the time was that uninformed voters could voters know less. To address these questions, seldom, if ever, cast competent votes – where they develop a model with informed and competence refers to whether a voter casts uninformed voters. Informed voters know the same vote that he or she would have the policy positions of two candidates. Unin- cast if he or she possessed full informa- formed voters do not, but they can observe tion about all matters in the model that are poll results or interest group endorsements. pertinent to his or her choice (e.g., candi- McKelvey and Ordeshook examine when date policy positions). The breadth of condi- uninformed voters can use the polls and tions under which McKelvey and Ordeshook endorsements to cast the same votes they proved that 1) uninformed voters vote com- would have cast if completely informed. petently, and 2) election outcomes are identi- In the model’s equilibrium, voters make cal to what they would have been if all voters inferences about candidate locations by using were informed prompted a reconsideration of poll results to learn how informed voters are the conditions under which limited informa- voting. Uninformed voters come to correctly tion made voters incompetent. infer the candidates’ positions from insights Lupia and McCubbins (1998) pursue these such as “if that many voters are voting for the conditions further. They examine multiple [rightist candidate], he can’t be too liberal.” ways in which voters can be uninformed and McKelvey and Ordeshook (1990) prove that incorporate focal insights from the psycho- the greater the percentage of informed voters logical study of persuasion. By using formal represented in such polls, the more quickly models and experiments, they could clarify uninformed voters come to the correct con- how conditional relationships among psycho- clusion about which candidate is closest to logical, institutional, and other factors affect their interests. competence and persuasion in ways that the 96 John H. Aldrich and Arthur Lupia dominant approach to studying voter compe- sible to conduct an analysis of the conditions tence – conventional survey-based analyses – under which a range of factors has differential had not. and conditional effects on whether persuasion The starting point for Lupia and Mc- occurs. Lupia and McCubbins (1998)dojust Cubbins (1998) is that citizens must make that, examining the logical consequences of decisions about things that they cannot mixing a range of assumptions about beliefs experience directly. For voters, the task is and incentives to generate precise conclu- to choose candidates whose future actions in sions about the conditions under which 1) one office cannot be experienced in advance of the person can persuade another, and 2) persua- election. Relying on others for information sive attempts make voters competent – that in such circumstances can be an efficient way is, help them choose as they would if fully to acquire knowledge. However, many peo- informed. ple who provide political information (e.g., Their models and experiments also show campaign organizations) do so out of self- that any attribute causes persuasion only interest, and some may have an incentive to if it informs a receiver’s perceptions of a mislead. For voters who rely on others for speaker’s knowledge or interests. Otherwise, information, competence depends on whom the attribute cannot (and experimentally does they choose to believe. If they believe people not) affect persuasion, even if it actually who provide accurate information and ignore affects the speaker’s choice of words. Exper- people who do otherwise, then they are more iments on this topic clarified how environ- likely to be competent. mental, contextual, and institutional variables A key move in the development of the (e.g., those that make certain kinds of state- Lupia-McCubbins (1998)modelistofollow ments costly for a speaker to utter) make the arguments of empirical scholars of vot- learning from others easier in some cases and ing behavior and public opinion who linked difficult in others. this question to the social psychological study These and other subsequent experiments of persuasion. O’Keefe (1990) defines per- demonstrate that the knowledge threshold for suasion as “a successful intentional effort at voting competently is lower than the norma- influencing another’s mental state through tive and survey-based literatures at the time communication in a circumstance in which had conjectured (see Boudreau and Lupia’s the persuadee has some measure of free- chapter in this volume for more examples of dom” (17). Seen in this way, the outcomes of such experiments). Instead of being required many political interactions hinge on who can to have detailed information about the utility persuade whom. Social psychologists have consequences of all electoral alternatives, it generated important data on the successes can be sufficient for the voter to know enough and failures of persuasive attempts (see, e.g., to make good choices about whom to believe. McGuire 1985; Petty and Cacioppo 1986). So, when information about endorsers is eas- Although psychological studies distinguish ier to acquire than information about policies, factors that can be antecedents of persuasion voters who appear to be uninformed can cast from factors that cannot, they are typically the same votes that they would have cast if formulated in a way that limits their appli- they had known more. In sum, there appears cability to questions of voting behavior. The to be logic to how uninformed voters use typical social psychological study of persua- information. Formal models have provided a sion is a laboratory experiment that examines basis for discovering it, and experimentation how a single variation in a single factor corre- offers one important way for testing it. sponds to a single attribute of persuasiveness. Such studies are designed to answer ques- Jury Decision Making tions about the conditions under which some attributes will be more important than others Experiments have also clarified implications in affecting the persuasive power of a particu- of a visible game-theoretic claim about jury lar presentation. In a formal model, it is pos- decision making. Many courts require a Experiments and Game Theory’s Value to Political Science 97 unanimous vote of a jury in order to con- that were otherwise in the type of deci- vict a defendant. A common rationale for this sion environment described by Feddersen requirement is that unanimity minimizes the and Pesendorfer (1998). Guernaschelli et al. probability of convicting the innocent. report that where “Feddersen and Pesendor- Feddersen and Pesendorfer (1998) iden- fer (1998) imply that large unanimous juries tify an equilibrium in which unanimity will convict innocent defendants with fairly produces more false convictions than was high probability ...this did not happen in previously believed. The logic underlying our experiment” (416). In fact, and con- their result is as follows. Suppose that all trary to Feddersen and Pesendorfer’s claims, jurors are motivated to reach the correct this occurred less frequently as jury size verdict. Suppose further that it is common increased. knowledge that every juror receives a sig- Experimental results such as these imply nal about a defendant’s status (i.e., courtroom that the frequency at which unanimity rule testimony and/or jury room deliberation) that convicts the innocent requires additional is true with a known probability. knowledge of how jurors think. These exper- In this case, a juror is either not pivotal (i.e., iments helped motivate subsequent model- his or her vote cannot affect the outcome) or ing that further clarified when strategic vot- is pivotal (i.e., his or her vote does affect the ing of the kind identified by Feddersen and outcome). Under unanimity, if at least one Pesendorfer (1998) causes unanimity require- other juror is voting to acquit, then a juror is ments to produce false convictions. With not pivotal, and the defendant will be found a model whose assumptions are built from not guilty regardless of how this juror decides. empirical studies of juries by Pennington and Likewise, a juror is pivotal under unanimity Hastie (1990, 1993) and psychological experi- rule only if every other juror is voting to con- ments on need for cognition by Cacioppo and vict. Hence, a juror can infer that either his or Petty (1982), Lupia, Levine, and Zharinova her vote makes no difference to the outcome (2010) prove that it is not strategic voting per or that all other jurors are voting to convict. se that generates Feddersen and Pesendor- Feddersen and Pesendorfer (1998) examine fer’s (1998) high rate of false convictions. how such reasoning affects the jurors’ assess- Instead, driving the increase in false convic- ments of the defendant’s guilt. They identify tions is the assumption that all jurors con- conditions in which the weight of each juror’s jecture that all other jurors are thinking in conjecture about what other jurors are doing the same manner as they are. Lupia et al. leads every juror to vote to convict – even if (2010) show that strategic voting under dif- every single juror, acting solely on the basis ferent, and more empirically common, beliefs of the signal he or she received, would have can cause far fewer false convictions. Collec- voted innocent. False convictions come from tively, experiments and subsequent models such calculations and are further fueled by show that using more realistic assumptions jury size (as n increases, so does the informa- about jurors generates equilibria with many tional power of the conjecture that “if I am fewer false convictions. pivotal, then it must be the case that every More generally, we believe that the pair- other juror is voting to convict.”) Fedder- ing of formal models and experiments can sen and Pesendorfer use these results to call be valuable in improving political scientists’ into question claims about unanimity’s con- efforts to pursue psychological explanations victions of the innocent. of behavior. Models can help scholars deter- A number of scholars raised ques- mine whether claims being made about cit- tions about whether making more realistic izen psychology must be true given a set of assumptions about jurors could yield dif- clearly stated assumptions, or whether the ferent results. Some scholars pursued the claim is possibly true given those founda- question experimentally. Guernaschelli, Mc- tions. These types of questions are now being Kelvey, and Palfrey (2000) examine student asked with increasing directness (e.g., Lupia juries of different sizes (n = 3 and n = 6) and Menning 2009) and are serving as the 98 John H. Aldrich and Arthur Lupia foundations for several exciting new research It is easy to imagine that both players agendas, such as Dickson’s chapter in this vol- could earn very high payoffs by passing for ume describes. a while to let the object grow in value. The game, however, has a unique Nash equilibrium (Rosenthal 1981), and it does not involve Contributions to Other Fields such behavior. Instead, it predicts that play- Political scientists have also used combina- ers will take at the first opportunity. A num- tions of game theory and experiments to make ber of scholars raised questions about the contributions whose relevance extends well applicability of this prediction. Two political beyond political science. Eckel and Wilson’s scientists, McKelvey and Palfrey (1992), ran discussion of trust (see their chapter in this experiments to address these questions. As volume) and Coleman and Ostrom’s discus- Palfrey (2007b) describes their experimental sion of collective action (see their chapter efforts, in this volume) provide prominent exam- [W]e designed and conducted an experiment, ples. Other such examples are experiments 1992 1995 1998 not to test any particular theory (as both of by McKelvey and Palfrey ( , , ). us had been accustomed to doing), but sim- Their experimental and theoretical efforts ply to find out what would happen. How- provide a focal moment in the emergence of ever, after looking at the data, there was a behavioral economics. problem. Everything happened! Some players Behavioral economics is a movement passed all the time, some grabbed the big pile that seeks to derive economically relevant at their first opportunity, and others seemed conclusions from premises with increased to be unpredictable, almost random. But there psychological realism. The evolution and were clear patterns in the average behav- gradual acceptance in economics of a behavior, the main pattern being that the proba- ioral approach was motivated by an impor- bility of taking increased as the piles grew. (426) tant set of experiments. These experiments revealed systematic divergence between the Their efforts to explain such behavior evolved predictions of several well-known game- into the development of a new equilib- theoretic models and the behavior of labo- rium concept, quantal response equilibrium ratory subjects in arguably similar decision (QRE). QRE is a variant of the Nash equi- contexts. librium concept that allows modelers to One such model is called the centipede account for a much wider set of assump- game. In a centipede game, two players decide tions about what players believe about one how to divide an object of value (say, $10). another than did traditional concepts. As One player can take a very unequal share of applied to the centipede game, the concept the object for him- or herself (say, “I get $7 allowed players to adjust their strategies to and you get $3”), or the player can pass on varying beliefs that the other players would that opportunity. If he or she takes the larger (or would not) play the strategies named share, the game ends and players are paid in the game’s unique Nash equilibrium. accordingly. If he or she passes, the object This concept allowed McKelvey and Palfrey doubles in value and the other player can (1998) to explain patterns of play in the cen- take the larger share (say, “I get $14 and tipede game far more effectively than other you get $6”). An important part of the game approaches. is that payoffs are arranged so that a player Both McKelvey and Palfrey’s (1992, 1995, gets slightly more from taking in the cur- 1998) experimental documentation of the rent round ($7) than he or she will if the problems with the applicability of the cen- other player takes in the next round ($6). The tipede game’s Nash equilibrium and their game continues for as long as players pass. In use of these results to develop an alter- every subsequent round, the object continues nate equilibrium concept now serve as mod- to double in value after each pass and players els for why a more behavioral approach to alternate in their ability to take. economic topics is needed and how more Experiments and Game Theory’s Value to Political Science 99 realistic psychological content can begin to comes. Coleman and Ostrom’s chapter in be incorporated. this volume reviews how experiments have helped scholars from multiple disciplines better understand the prerequisites for effective 3. Conclusion collective action. Diermeier’s chapter in this volume high- An important attribute of formal models is lights experimental evaluations of the Baron- that they allow scholars to analyze, with Ferejohn model of coalition bargaining. He precision and transparency, complex condi- describes how his early experiments con- tional relationships among multiple factors. sistently demonstrate that the person with As such, scholars can use formal models to the ability to propose coalition agreements evaluate the conditions under which various to other actors consistently takes less power kinds of causal relationships are logically con- than the model predicts. Drawing from psy- sistent with clearly stated assumptions about chology, he argues that other factors rele- actors and institutions. The distinguishing vant to sustained human interaction could characteristic of formal models is their abil- induce a bargainer to offer potential partners ity to facilitate constructive and precise con- more than the absolute minimum amounts versations about “what is related to what” in that they would accept. His later experi- domains of political choice. ments incorporate these factors and show In some cases, however, scholars raise rea- promise as a foundation of more effective sonable questions about whether the logic explanations of coalition behavior. of a particular formal model is relevant to a Wilson and Eckel’s chapter in this volume particular set of real-world circumstances. At shows how experiments have clarified many such moments, empirical demonstrations can questions about the relevance and applicabil- be valuable. They can demonstrate that the ity of formal models of trust. They begin model does in fact explain relevant behaviors by describing the Nash equilibrium of the well, or they can show that the model requires “investment game.” It is a game where players serious revision. can benefit by trusting one another to con- In many cases, however, nature does not tribute some of their resources to a common provide the kinds of data scholars would need pool. But the unique Nash equilibrium of this to answer such questions. Moreover, if the particular game is that players will not trust claims in question pertain to how certain one another enough to realize these gains. actors would react under a wide range of cur- They then review a series of experiments that rently hypothetical circumstances, or if the examine many psychological and biological controversy pertains to whether a particular factors relevant to trust in a range of cultural claim is true given a different set of under- and institutional settings around the world. lying counterfactuals, then there may be no Through these efforts, theories and exper- observational approach that will provide suf- imental designs build off one another, and ficient data. In such cases, experiments can what results is clarification about when we help us evaluate the relevance and applica- can expect trust in a set of critical social rela- bility of focal model attributes to important tionships. political phenomena. Collectively, these chapters reveal both the This chapter describes a few instances in challenges inherent in using formal mod- which experiments played an important role els and experiments to provide substantive in the development and evaluation of game- insight into political science and the ways in theoretic political science. Several chapters which experiments help formal modelers and in this volume review other interesting exam- scholars with more substantive interests com- ples. Morton and Williams’ chapter in this municate more effectively. Given the increas- volume, for example, details how clever ing number of political scientists who are experimental agendas clarify how institu- interested in experiments, we believe that the tional rules affect electoral behavior and out- examples described in this chapter are merely 100 John H. Aldrich and Arthur Lupia the tip of the iceberg relative to the rela- Games.” In Decision Processes, eds. Robert M. tionship among formal models, experiments, Thrall, Clyde H. Coombs, and Robert L. Davis. and political science. For as long as scholars New York: John Wiley, 301–327. who are knowledgeable about political con- Lupia, Arthur, Adam Seth Levine, and Natasha 2010 texts want models to be closer to facts or Zharinova. . “Should Political Scientists built from premises with more psychological Use the Self-Confirming Equilibrium Con- cept? Benefits, Costs and an Application to Jury or sociological realism, there will be demand Theorems.” Political Analysis 18: 103–23. for bridges between the logic of the models Lupia, Arthur, and Mathew D. McCubbins. 1998. and the world in which we live. Experiments The Democratic Dilemma: Can Citizens Learn are uniquely positioned to serve as the foun- What They Need to Know? New York: Cam- dations of those bridges. bridge University Press. Lupia, Arthur, and Jesse O. Menning. 2009. “When Can Politicians Scare Citizens into References Supporting Bad Policies?” American Journal of Political Science 53: 90–106. Austen-Smith, David, and Arthur Lupia. 2007. McGuire, William J. 1985. “Attitudes and Attitude “Information in Elections.” In Positive Changes Change.” In Handbook of Social Psychology. vol. in Political Science: The Legacy of Richard Mc- II, 3rd ed., eds. Gardner Lindzey and Elliot Kelvey’s Most Influential Writings, eds. John H. Aronson. New York: Random House, 233–346. Aldrich, James E. Alt, and Arthur Lupia. Ann McKelvey, Richard D. 1976. “Intransitives in Arbor: University of Michigan Press, 295–314. Multidimensional Voting Models and Some Black, Duncan. 1948. “On the Rationale of Group Implications for Agenda Control.” Journal of Decision-Making.” Journal of Political Economy Economic Theory 18: 472–82. 56: 23–34. McKelvey, Richard D., and Peter C. Ordeshook. Cacioppo, John T., and Richard E. Petty. 1982. 1978. “Competitive Coalition Theory.” In “The Need for Cognition.” Journal of Personal- Game Theory and Political Science, ed. Peter C. ity and Social Psychology 42: 116–31. Ordeshook. New York: New York University Converse, Philip E. 1964. “The Nature of Belief Press, 1–37. Systems in Mass Publics.” In Ideology and its McKelvey, Richard D., and Peter C. Ordeshook. Discontents, ed. David E. Apter. New York: The 1979. “An Experimental Test of Several The- Free Press of Glencoe, 206–261. ories of Committee Decision Making under Feddersen, Timothy, and Wolfgang Pesendorfer. Majority Rule.” In Applied Game Theory, eds. 1998. “Convicting the Innocent: The Inferior- Steven J. Brams, A. Schotter, and G. Schwodi- ity of Unanimous Jury Verdicts under Strategic auer. Wurzburg, Germany: Springer-Verlag, Voting.” American Political Science Review 92: 152–167. 23–35. McKelvey, Richard D., and Peter C. Ordeshook. Ferejohn, John A., Morris P. Fiorina, and Edward 1980. “Vote Trading: An Experimental Study.” Packel. 1980. “Non-Equilibrium Solutions for Public Choice 35: 151–84. Legislative Systems.” Behavioral Science 25: McKelvey, Richard D., and Peter C. Ordeshook. 140–48. 1983. “Some Experimental Results That Fail Fiorina, Morris P., and Charles R. Plott. 1978. to Support the Competitive Solution.” Public “Committee Decisions under Majority Rule: Choice 40: 281–91. An Experimental Study.” American Political Sci- McKelvey, Richard D., and Peter C. Ordeshook. ence Review 72: 575–98. 1990. “A Decade of Experimental Research on Frohlich, Norman, and Joe A. Oppenheimer. Spatial Models of Elections and Committees.” 1992. Choosing Justice: An Experimental Approach In Advances in the Spatial Theory of Voting, eds. to Ethical Theory. Berkeley: University of Cali- Melvin J. Hinich and James Enelow. Cam- fornia Press. bridge: Cambridge University Press, 99–144. Guernaschelli, Serena, Richard D. McKelvey, and McKelvey, Richard D., Peter C. Ordeshook, and Thomas R. Palfrey. 2000. “An Experimental Mark D. Winer. 1978. “The Competitive Solu- Study of Jury Decision Rules.” American Politi- tion for N-Person Games without Transfer- cal Science Review 94: 407–23. able Utility, with an Application to Committee Kalish, G. K., J. W. Milnor, J. Nash, and E. D. Games.” American Political Science Review 72: Nehrig. 1954. “Some Experimental n-Person 599–615. Experiments and Game Theory’s Value to Political Science 101

McKelvey, Richard D., and Thomas R. Palfrey. Pennington, Nancy, and Reid Hastie. 1993. “Rea- 1992. “An Experimental Study of the Cen- soning in Explanation-Based Decision Mak- tipede Game.” Econometrica 60: 803–36. ing.” Cognition 49: 123–63. McKelvey, Richard D., and Thomas R. Palfrey. Petty, Richard E., and John T. Cacioppo. 1986. 1995. “Quantal Response Equilibria for Nor- Communication and Persuasion: Central and Peri- mal Form Games.” Games and Economic Behav- pheral Routes to Attitude Change. New York: ior 10: 6–38. Springer-Verlag. McKelvey, Richard D., and Thomas R. Palfrey. Rapoport, Anatol, and Albert M. Chammah. 1965. 1998. “Quantal Response Equilibria for Exten- Prisoner’s Dilemma: A Study in Conflict and Coop- sive Form Games.” Experimental Economics 1: eration. Ann Arbor: University of Michigan 9–41. Press. Nash, John F. 1997. Essays on Game Theory.New Riker, William H. 1962. The Theory of Political York: Edward Elgar. Coalitions. New Haven, CT: Yale University O’Keefe, Daniel J. 1990. Persuasion: Theory and Press. Research. Newbury Park, CA: Sage. Riker, William H. 1967. “Bargaining in a Three- Ordeshook, Peter C. 2007. “The Competitive Person Game.” American Political Science Review Solution Revisited.” In Positive Changes in Polit- 61: 642–56. ical Science: The Legacy of Richard D. McKelvey’s Riker, William H., and William James Zavoina. Most Influential Writings, eds. John H. Aldrich, 1970. “Rational Behavior in Politics: Evidence James E. Alt, and Arthur Lupia. Ann Arbor: from a Three-Person Game.” American Political University of Michigan Press, 165–86. Science Review 64: 48–60. Palfrey, Thomas R. 2007a. “Laboratory Exper- Rosenthal, Robert. 1981. “Games of Perfect Infor- iments.” In The Oxford Handbook of Political mation, Predatory Pricing, and the Chain Economy, eds. Barry R. Weingast and Donald Store.” Journal of Economic Theory 25: 92– Wittman. New York: Oxford University Press, 100. 915–936. Roth, Alvin E. 1995. “Introduction to Experimen- Palfrey. Thomas R. 2007b. “McKelvey and Quan- tal Economics.” In The Handbook of Experimen- tal Response Equilibrium.” In Positive Changes tal Economics, eds. John H. Kagel and Alvin in Political Science: The Legacy of Richard D. Mc- E. Roth. Princeton, NJ: Princeton University Kelvey’s Most Influential Writings, eds. John H. Press, 3–110. Aldrich, James E. Alt, and Arthur Lupia. Ann Schofield, Norman. 1978. “Instability of Simple Arbor: University of Michigan Press, 425–40. Dynamic Games.” Review of Economic Studies Pennington, Nancy, and Reid Hastie. 1990. “Prac- 45: 575–94. tical Implications of Psychological Research on Von Neumann, John, and Oscar Morgenstern. Juror and Jury Decision Making.” Personality 1944. Theory of Games and Economic Behavior. and Social Psychology Bulletin 16: 90–105. Princeton, NJ: Princeton University Press. CHAPTER 8

The Logic and Design of the Survey Experiment

An Autobiography of a Methodological Innovation

Paul M. Sniderman

The title promises a chapter about meth- The modern survey experiment is the ods, so a confession is in order. Here, as biggest change in survey research in a half everywhere, my concerns are substantive, not century. There is some interest in how it came methodological. Still, what one wants to learn about, I am told. So I begin by telling how and how one ought to go about learning I got the idea of computer-assisted survey it are intertwined. So, I propose to bring experiments. I excuse this personal note partly out the logic of the survey experiment by because the editors requested it but, more presenting a classification of survey experi- importantly, because it allows me to acknowl- ment designs. Specifically, I distinguish three edge publicly the contributions of others. designs: manipulative, permissive, and facilitative. The distinctions among the designs turn on the hypotheses being tested, not the 1. Logic of Discovery operations performed, and, above all, on the role of predispositions. The first design aims Experiments have been part of public opin- to get people to do what they are not pre- ion surveys for many years, but they took disposed to do; the second to allow them to the form of the so-called split ballot. The do what they are predisposed to do, with- questionnaire would be printed in two ver- out encouraging them; and the third to pro- sions, and test questions would appear in vide them with a relevant reason to do what each that were identical in every respect but they already are predisposed to do. Against one. A more procrustean design is difficult to the background of this threefold classifica- imagine. So, it is the more compelling testi- tion, I want to comment briefly on some mony to the ingenuity of researchers that they issues of causal inference and external valid- still managed to learn a good deal.1 All the ity and then conclude by offering my own same, what they learned, although of advan- view on the reasons for the explosive growth tage for the applied side of public opinion in survey experiments in the study of public opinion. 1 Schuman and Presser (1981) is the seminal work.

102 The Logic and Design of the Survey Experiment 103 research, was of less value for the academic not as children parachuted in from California side. Indeed, if one wanted to be waspish, then for a brief stay, with their grandmother plac- one could argue that this first generation of ing vats of candy by their bedsides, but as a survey experiments did more harm than good. family living together. It seemed like a good By appearing to show that even trivial changes idea, and once again I learned the danger of in question wording could produce profound good ideas. Our children were heartbroken at changes in responses, they contributed to a returning to California. There was an upside, zeitgeist that presumed that citizens did not however. Living with one’s in-laws, however really have genuine attitudes and beliefs. And, welcoming they are, is an out-of-equilibrium by the sheer repetitiveness of the split bal- experience. I mention this only because it says lot design, they reinforced in the minds of something about the social psychology of dis- several generations of subsequent researchers covery. I do not believe that I would have that survey experiments had to fit the straight had the breakthrough idea about computer- jacket of the two – and only two – conditions. assisted survey experiments were it not for the Partly because the split ballot was pro- sharp and long break with everyday routine. crustean, and partly because my interest is Among other things, it allowed the past to substantive rather than methodological, the catch up with the present. idea that survey experiments could be a use- As a child, I went to a progressive sum- ful tool did not enter my head. The idea came mer camp. After a day of games on land and to me by a more circuitous route. The cof- water, we would be treated to a late-afternoon fee machine at the Survey Research Center lecture in the rec hall on issues of social (SRC) at the University of California, Berke- importance. One of the lessons that we were ley, was located on the second floor. Because taught was that discrimination and prejudice there was only one machine, whoever wanted are quite different things. Prejudice is how coffee had to go there. One day, in 1983,a others feel about us (i.e., Jews), whereas dis- cup of coffee was just what I wanted. The crimination is how others treat us. Although line was long and directly behind me was prejudice is a bad thing, how others feel about Merrill Shanks. He was pumped up, so I asked us is not nearly as important as how they treat him what was happening. Merrill is basket- us. Covenants against Jews buying property ball tall; I, less so, it would be fair to say. It in “protected” areas, bans on membership must have been quite a sight, Merrill tower- in clubs, and quotas on university admission ing over me, gesticulating with excitement, were the norm then.3 But between then and explaining that he had succeeded in writing now, the memory of the lecture on the dif- a general-purpose, computer-assisted interference between prejudice and discrimination viewing program and illustrating with (both would regularly recur, and I would just as rou- physical and mental) gusto the measure of the tinely be struck by the frustrating irony that breakthrough.2 I was thrilled for my friend’s I had enlisted in a vocation, survey research, achievement, even if quite uninterested in the that could study prejudice (attitudes) but not achievement itself. Computer-assisted inter- discrimination (action). That persisting frus- viewing passed right through – and out of – tration, I believe, was behind the idea that my mind. struck me on my walk with such force. A year later, we took our children to spend Here was the idea. The first computer- a year in Toronto, living at my in-laws’ house, assisted interviewing program was purpose so that they would know their grandparents, built for question sequencing. Depending on and their grandparents would know them – the answer that a respondent gave to the first question in a series, the interviewer’s screen 2 It was an exceptional achievement. ISR at Michigan and NORC in Chicago, the two heavyweight cham- 3 My father and father-in-law were among the first pions of academic survey research, gave years and a Jews permitted to attend the University of Toronto treasure chest of man-hours attempting to match the Medical School. My wife was a member of the first programming achievement of Merrill and his col- class of the University of Toronto Medical School in leagues, only to fail. which the Jewish quota was lifted. 104 Paul M. Sniderman would automatically light up with the next years, the main thing we knew about each appropriate question. In turn, depending on other is that we shared an interest in the the answer that she gave to the second ques- analysis of racial attitudes (see Apostle et al. tion, the screen would light up with the next 1983). Blanche DuBois relied on the kind- appropriate question, and so on. The depth of ness of strangers. I have relied on their cre- Shanks’ achievement, though, was that he had ativity and character. Tom was the one who transformed the opinion questionnaire into a made computer-assisted randomized exper- computer program that could be put to many iments work. Every study that I have done uses. His was question sequencing, whereas since, we have done together, regardless of mine was randomized experiments. whether his name appeared on the project. I saw that, with one change – inserting Childhood memories, disruption of rou- a random operator to read a computer clock – tines, social science as a collaborative enter- computer-assisted interviewing could pro- prise, technology as door opening, research vide a platform for randomized experiments. centers as institutionalized sources of ecolog- Depending on the value of the random oper- ical serendipity – those are the themes of the ator, questions could be programmed to first part of my story on the logic of discovery. appear on an interviewer’s monitor, varying The theme of the second part of my story is the wording, formatting, and order. The pro- a variation on Robert Merton’s (1973)clas- cedure would be effortless for the interviewer. sic characterization of the communist – his She need only ask the form of the question word – character of science.5 that appeared on her monitor. And it would Tom and I had a monopoly position. be invisible to the respondent. The fact that Rather than take advantage of Merrill’s there were multiple versions of the question, breakthrough, the Institute for Social Rese- randomly administered, would be invisible to arch (ISR) at the University of Michigan the interviewees because they would only be and the National Opinion Research Center asked one.4 (NORC) at The University of Chicago Voila! There was the broad answer to attempted – for years – to write their own the summer camp lecture on the distinction computer-assisted program. It was a bad deci- between prejudice and discrimination. Ask sion for them because they failed. An ideal a randomly selected set of respondents how outcome for us, you might think. Only studies much help the government should give to a done through the Berkeley SRC could exploit white American who lost her job in finding the flexibility of computer-assisted interview- another. Ask the others exactly the same ques- ing in the design of randomized experiments, tion, except assume that it is a black American which meant that we would have no compe- who has been laid off. If more white Ameri- tition in conducting survey experiments for cans back a claim to government assistance if years into the future. the beneficiary is white, then we are capturing Merton was right about the communist not only how they feel about black Americans character of science, however. We would but also how they treat them. succeed, but we would do so alone. And if That was the idea – and I remember the we succeeded alone, we would fail. If other street that I was on and the house that I researchers could not play in our sandbox, was looking at when I had it. And abso- then they would find another sandbox to play lutely nothing would have come of it but for in. Our work would always be at the margins. Tom Piazza. Although Tom and I had seen The properly communist character of sur- each other around the halls of the SRC for vey research showed up in a second way.

4 The contrast is with the then-common practice of asking a series of items, varying the beneficiary of a 5 When referring to communism, Merton (1973) policy (i.e., would you favor the program if it bene- meant the principle of common ownership of scien- fited a white American?, if it benefited a black Amer- tific discoveries. We do not have a right to the means ican?). I am also presuming, when I speak of the pro- to make scientific discoveries, but we do have a right cedure being invisible to the respondent, the artful to share in them. And those who make them have a writing of an item. corresponding duty to allow us to share in them. The Logic and Design of the Survey Experiment 105

Public opinion surveys are expensive, so not out having to raise the money.8 But how to many researchers got a chance to do them. identify who should have the chance? A large Then Warren Miller effected the biggest- part of the motivation is that very few had ever structural change in the study of pub- had an opportunity to distinguish themselves lic opinion and elections. Consistent with through the design of original studies. My Merton’s doctrine of communism, Miller solution: I shamelessly solicited invitations to made the data of the flagship voting studies give talks at any university that would have nationally shared scientific property. In my me in order to identify a pool of possible par- judgment, there cannot be enough parades ticipants. I then invited them to write a pro- in his honor. It was quantitative analysis posal, on the understanding that their idea that shot ahead, however, not the design was theirs alone, but that the responsibility of surveys.6 The American National Elec- for making the case to the National Science tion Studies offered the opportunity for some Foundation (NSF) was mine. My sales pitch innovation in measurement. But its overrid- was, “We will do thirteen studies for the price ing obligation was to time series. Continuity of one.” That was the birth of the Multi- of design was the primary value; innovation Investigator Project. It is Karen Garret, in design was secondary. the director of the project, who deserves the I had had my chance to come to bat in credit for making the studies a success. designing a study, actually two studies.7 The I have one more personal note to add. The first article that we succeeded in publishing Multi-Investigator Project ran two waves. using randomized experiments gave me the The day that I received the grant from NSF idea. The article was built on the analysis of for the second wave, I made a decision. two experiments. Yet each experiment was I should give up the project. Gatekeepers only a question, admittedly a question that should be changed, I had always believed, came in many forms, but at the end of the and that applied to me, too. Diana Mutz day, only a question, which is to say that the and Arthur Lupia were the obvious choices. experiment took only about thirty seconds As the heads of Time-sharing Experiments to administer. It then came to me that an in the Social Sciences (TESS), they trans- interview of standard length could be used formed the Multi-Investigator Project. To as a platform for multiple investigators. Each get some order of the magnitude of the dif- would have time for two to four experiments; ference between the two platforms, think of each would have access to a common pool the Multi-Investigator Project as a stagecoach of right hand–side variables; each would be and TESS as a Mercedes-Benz truck. Add the a principal investigator. If their experiments support of the NSF, particularly through the were a success, they would be a success. If not, Political Science Program, the creativity of they would have had a chance to swing at the researchers, and the radical lowering of costs ball. to entry through cooperative election stud- This idea of a shared platform for inde- ies, and survey experiments have become a pendent studies was the second-best design standard tool in the study of public opinion idea on my score card. It made it possible and voting surveys. There is not a medal big for investigators, in the early stages of their enough to award Lupia and Mutz that would careers, to do original survey research with- do justice to their achievements. What is good fortune? Seeing an idea of 6 For an overview of how much progress was made on how many fronts, see Bartels and Brady (1993). yours travel the full arc, from being viewed 7 The first was the Bay Area Survey with Thomas at the outset as ridiculous to becoming in the Piazza, which led to Sniderman and Piazza (1993). end commonplace,9 and my sense of the idea The second was The Charter of Rights Study, which led to Sniderman et al. (1996). The third was the National Race and Politics (RAP) Study, which led 8 One of the benefits I did not anticipate was that, even to, among many other publications, Sniderman and if their first try had not succeeded, they had a leg up Carmines (1997) and Hurwitz and Peffley (1998). in writing a proposal for a full-scale study. The RAP was the trial run for the Multi-Investigator 9 My first proposal to the NSF to do survey experi- Project, and involved eight coprincipal investigators. ments was judged by two of the reviewers to be a 106 Paul M. Sniderman has itself traveled an arc. Originally, I saw it The first generation of “framing” experi- as a tool to do one job. Gradually, I came to ments is a poster child example of a manip- view it as a tool to do another. ulative design (e.g., Zaller 1992; Nelson and Kinder 1996). In one condition, a policy was framed in a way to evoke a positive response; 2. A Design Classification in the other, the same policy was framed to evoke a negative response. And, would To bring out the explanatory roles of survey you believe, the policy enjoyed more sup- experiments in the study of public opinion, I port in the positive framing condition and distinguish among three design templates for evoked more opposition in the negative one? survey designs: manipulative, permissive, and 10 The substantive conclusion that was drawn facilitative. was that the public was a marionette, and its strings could be pulled for or against a policy Manipulative Designs by controlling the frame. But this is to tell a Standardly, the distinction between obser- story about politics with the politics left out. vational and experimental designs parallels The parties and candidates battle over how the distinction between those that are rep- policies should be framed, just as they battle resenting and intervening (Hacking 1983). over the positions that citizens should take on Interventions or manipulations are the nat- them.13 So Theriault and I (Sniderman and ural way to think of the treatment condi- Theriault 2004) carried out a pair of experi- tion in an experiment. How does one test a ments that replicated the positive and negative vaccine?11 By intervening on a random basis, conditions of the first generation of framing administering a vaccine to some patients and experiments, but we added a third condition a placebo to others, and noting the differ- in which both frames were presented and a ence in outcome between the two. Moreover, fourth in which neither appeared. The first the equation of intervention and manipula- two conditions replicated the findings of the tion seemed all the more natural against the first generation of framing experiments. But background understanding of public opinion the third led to a quite different conclusion. a generation ago. Knowing and caring little Confronted with both frames in the experi- about politics, the average citizen arranged ment (as they typically would be in real life, if her opinions higgledy-piggledy (the lack of not simultaneously, then in close succession), constraint problem), even supposing that she rather than being confused and thrown off had formed some in the first place (the nonat- the tracks, respondents are better able to pick titudes problem), the reductio of this concep- the policy alternative closest to their general tion of public opinion being the claim that view of the matter. Druckman (e.g., Druck- “most” people lacked attitudes on “most” man 2001a, 2001b, 2001c, 2004; Chong and issues, preferring instead “to make it up as Druckman 2007a, 2007b; Druckman et al. they go along.”12 What, then, was the role 2010) pried this small opening into a semi- of survey experiments? To demonstrate how nal series of studies on framing. In the areas easily one could get respondents to do what in which I have research expertise, I am hard- they were not predisposed to do. pressed to think of another who has, step by step, progressively deepened our understand- farcical undertaking, one of whom took eight pages ing of a focal problem. to make sure that his opinion of the project was clear. 10 The classification hinges on the aims of experiments. Survey experiments employing a manip- Because I know the hypotheses that experiments I ulative design can be of value. But my own have designed were designed to test, I (over)illustrate the principles, using examples of experiments that my colleagues and I have conducted. 13 The idea of dual frames – or, to use Chong and 11 See Freedman, Pisani, and Purves (1998), who offer Druckman’s (2007a) term, competitive frames – came the Salk vaccine test as a paradigm example of ran- to me while watching a Democratic campaign ad on domized experiment. television framing an issue to its advantage, immedi- 12 For a detailed critique of this view of public opinion, ately followed by a Republican ad framing the same see Sniderman, Tetlock, and Elms (2001). issue to its advantage. The Logic and Design of the Survey Experiment 107 approach to the logic, and therefore the but the data analyst can determine ex post the design of survey experiments, travels in the proportion of respondents making a partic- opposite direction. To overstate, my premise ular response (see Kuklinski et al. 1997). To is that you can get people to do in a survey give a hypersimplified description of the pro- experiment mainly what they already are will- cedure, in the baseline condition, the inter- ing to do – which is fortunate because this is viewer begins by saying, “I am going to read what we want to learn after all. This premise you a list of some things that make some peo- is the rationale for the next two designs: per- ple angry. I want you to tell me how many missive and facilitative. make you angry. Don’t tell me which items make you angry. Just how many.” The interviewer then reads a list of, say, four items. Permissive Designs In the test condition, everything is exactly A manipulative design aims to get respon- the same, except that the list now has one dents to do what they are not predisposed more item, say, affirmative action for blacks. to do. In contrast, a permissive design aims to To determine the proportion of respondents allow respondents to do what they are pre- angry over affirmative action, it is only neces- disposed to do without encouraging them to sary to subtract the mean angry responses in do it. The strategy is to remove, rather than the baseline condition from the mean angry apply, pressure to favor one response alter- responses in the test condition, and then mul- native over another. Think of this as exper- tiply by 100. Characteristics of respondents imental design in the service of unobtrusive that increase (or decrease) the hit rate can be measurement (Webb et al. 1996). identified iteratively. A showpiece example of a permissive This type of design I baptize permissive design in survey experiments is the List Exp- because it allows respondents to respond eriment.14 The measurement problem is this: without encouraging, inducing, or exerting Can one create a set of circumstances in pressure on them to do so. So it is with the which a person being interviewed can express List Experiment. Why do some respondents a potentially objectionable sentiment with- respond with a higher number in the treat- out the interviewer being aware that she has ment condition than in the baseline condi- expressed it?15 Kuklinski’s creative insight: to tion? Because they are predisposed to do so. devise a question format that leads respon- They are angry over affirmative action and are dents to infer, correctly, that the interviewer being given the opportunity to express their cannot tell which responses they have made, anger – believing (correctly) that the interviewer has no way of knowing that they have 14 Here is one of the few times when I know for certain where an idea came from. I myself was a witness done so – without realizing that a data ana- at the creation of List Experiment. During a plan- lyst could deduce the proportion expressing ning session for the 1990 Race and Politics Study at anger ex post. the Circle 7 Ranch, I took Jim Kuklinski for a Jeep ride in the meadow. Suddenly, by the front gate, he How should we conceive of the logic of stood up, exclaimed the equivalent of “Eureka,” and a permissive design such as the List Experi- outlined the design of the List Experiment. I men- ment? Baseline and test conditions were the tion this for two reasons: 1) to put on record that Kuklinski devised the List Experiment, easily the words I used to refer to the two conditions most widely used survey experiment design, and 2)to in our hypersimplified example of the List offer an historical example of the creativity of multi- Experiment.16 The baseline condition corre- investigator studies – the National Race and Politics Study had nine coprincipal investigators and con- sponds to the natural understanding of the tributed more innovations than any previous study because of its power for innovation. 16 Again, by way of underlining the decisive difference 15 There is another possibility, and a more likely one between the straight jacket or the split ballot design in my view. They do not want to say openly that and the plasticity of the computer-assisted interview- affirmative action makes them angry because doing ing, I would underline that the actual design of List so conflicts with their sense of self and their politi- Experiments tends to involve a number of test con- cal principles; that is, it violates a principle or image ditions, allowing for the comparison and contrast of, of themselves that they value (see Sniderman and say, responses to African Americans becoming neigh- Carmines 1997). bors and asking for affirmative action. 108 Paul M. Sniderman control condition. But in what sense is the designs aim to allow respondents to do what “test” condition a “treatment” condition? It they are predisposed to do without encour- entails exposure to a stimulus, in this case, aging them. Manipulative designs aim to get affirmative action. Affirmative action is a people to do what they are not predisposed to provocative stimulus, one could argue. But do. Like permissive designs but unlike manip- to make this argument would be to miss the ulative ones, facilitative designs do not involve point. If a person is indifferent to affirmative the use of coercive or impelling force. Unlike action, sympathetic to it, or simply ignorant permissive and manipulative designs, facilita- of it, then the mere mention of affirmative tive designs involve a directional force in the action will not evoke an angry response. To form of a relevant reason to do what people evoke an angry response, it is necessary that are already predisposed to do. she already be angry about it. I have become persuaded that this notion A second example of a permissive design of a relevant reason is a tip-off to a primary comes from a celebrated series of studies use of survey experiments for the study of on risk aversion by Tversky and Kahneman. public opinion. Let me illustrate what I mean They demonstrated that people have strik- by the notion of a relevant reason with an ingly different preferences on two logically experiment designed by Laura Stoker (1998). equivalent choices, depending on whether the The aim of this experiment is to determine choice is framed in terms of gains or losses. the connection between support for a policy Their Asian Flu Experiment is a paradig- and the justification provided for it. Stoker matic example. People are far more likely to picks affirmative action in its most provoca- favor exactly the same course of action if the tive form – mandatory job quotas. choice alternatives are posed in terms of lives This in-your-face formulation policy saved as opposed to lives lost. This result, frame should trigger the emotional logic that labeled “risk aversion,” is highly robust. With Converse (1964) argued underlies “reason- an ingenious design, Druckman (2001a) caring” about racial policies in general. How ried out an experiment that had two arms: one feels about blacks, he hypothesized, is the one matched the Kahneman-Tversky design, key to understanding why whites tend to line whereas the other added credible advice, in up on one or the other side of racial policies the form of endorsements of a course of action across the board. Feel negatively about blacks, by political parties. The key finding: partisans and you will oppose policies to help them; take their cue from party endorsements, so feel positively, and you will support them. much so that the gain–loss framing effect vir- Stoker’s (1998) experiment opens a new door tually disappears. I want to make two points on policy reasoning, though. It investigates with this example. First, framing effects are the persuasive weight of two different rea- robustly found between choices that are log- sons for mandatory quotas. Stoker’s results ically equivalent, depending on whether the show that one reason, the underrepresenta- choices are framed in terms of gains or losses, tion of blacks, counts as no reason at all – that in the absence of other information to exploit. is, there is no difference between deploying Second, the observed effect is not a function it as a justification and not deploying a justi- of an experimental intervention in the form fication at all. In contrast, the other reason, a of an application of pressure on a respondent finding of discrimination, counts as a relevant to react in a particular direction. It is instead a reason indeed – that is, it markedly increases matter of allowing people to respond as they support for affirmative action even framed in are predisposed without encouraging them to its most provocative form. Stoker’s discovery do so. is not the common-sense idea that policy justifications can make a difference. It is rather the differentiation of justifications that makes Facilitative Designs a difference. There is a world of difference The third type of design for survey exper- between declaiming that fairness matters and iments I christen facilitative. Permissive specifying what counts as fairness. The Logic and Design of the Survey Experiment 109

As a second example of facilitation, con- any position). Twice as many report changing sider the counter-argument technique. The their minds in response to a content- counter-argument technique was introduced laden, rather than a content-free, counter- in Sniderman and Piazza (1993) and explored argument. There is, in short, a difference further in Sniderman et al. (1996). Gibson between getting an argument and getting made it a central technique in the survey argued with. The second point is that the bulk researchers’ toolkit, deploying it in a remark- of those changing in the face of a content- ably ambitious series of survey settings.17 laden counter-argument had taken a position The first generation of counter-arguments at odds with their general view of the mat- only comprises a quasi-experiment, however. ter. What work, then, was the content-laden The counter-argument presented to recon- counter-argument doing? Most who change sider support for a policy is (naturally enough) their initial position in response to a content- different from the one presented to recon- laden counter-argument had good reason sider opposition to it, hence the relevance to change. The side of the issue they had of the second generation of the counter- initially chosen was inconsistent with their argument experiments (Jackman and Snider- general view of the matter.19 They were man 2006). Respondents take a position on rethinking their initial position by dint of a an issue and are then presented with a rea- reason that, from their point of view, should son to reconsider. What should count as a count as a reason to reconsider their posi- reason to reconsider, one may reasonably tion. In reconsidering, they were not chang- ask, and what more exactly are people doing ing their mind; rather, they were correcting when they are reconsidering their initial posi- a misstep. What, then, was the experimen- tion? Two content-laden counter-arguments tal intervention accomplishing? It was facil- are administered. One presents a substantive itating their reconsideration of the position reason for respondents who have supported they had taken in light of a consideration that more government help to renounce this posi- counted as a relevant reason for reconsider- tion, whereas the other provides a substan- ation, given their own general view of the tive reason for respondents who have opposed matter. it to renounce their position. In addition, a content-free counter-argument – that is, an objection to the position that respondents 3. Experimental Treatments and have taken that has the form of an argument Political Predispositions but not the specific substantive content of one18 – is also administered. Thus, In a pioneering analysis of the logic of survey half of the respondents initially support- experiments, Gaines, Kuklinski, and Quirk ing the policy get a content-laden counter- (2007) bring to the foreground a neglected argument; half get a content-free one. consideration. Respondents do not enter Ditto for respondents initially opposing the public opinion interviews as blank slates. policy. They bring with them the effects of previ- There are two points I would make. The ous experiences. Gaines et al. refer to the first is that respondents at all levels of polit- enduring effects of previous experience as ical sophistication discriminate between a pre-treatment. In their view, understanding genuine reason (i.e., an argument that pro- how pre-treatments condition experimental vides a substantive argument to reconsider) responses is a precondition of understand- and a pseudo reason (i.e., an argument that ing the logic of survey experiments. This is merely points to the uncertainty of taking a dead-on-target insight. In my view, it is an understatement. The purpose of survey 17 For an especially fascinating example, see Gibson and Gouws (2003). 19 Treatment and control groups were thus identically 18 The wording of the content-free counter-argument positioned. Analysis searching for asymmetric effects in this study is, “However, if one thinks of all the conditional on being pro or con the policy failed to problems this is going to create. . . . ” detect any. 110 Paul M. Sniderman experiments in the study of public opinion surely is a reasonable expectation that African is precisely to understand pretreatment – or, Americans will take into account the con- as I think of it, previous conditioning. tinuing burden of discrimination. The question then is, how small does the difference in scores between the two young men need The Null Hypothesis to be in order to be regarded as negligible for What is the null hypothesis in a survey African Americans to give the nod to the black experiment? It sounds odd to ask this, I candidate on other grounds – for example, the acknowledge. Textbooks drill into us a uni- fact that they have to overcome obstacles that form understanding of the null, that is, the whites do not. We worked to establish feet- absence of a difference between responses in in-cement expectations, recruiting a sample the treatment and control conditions. From of experts to pick the point at which a major- this it follows that the purpose of an exper- ity of African Americans would favor the black imental treatment is to produce a difference candidate. Seventy-five percent of our experts in the treatment condition, and if it fails to do picked a difference of just ten points to be so this, the experiment has failed. So it is com- small as to wave away against the historic and monly – and wrongly – supposed.20 continuing injustices done to blacks. And 100 To bring out the logic of the problem, I percent of them predicted that a difference of enlist the SAT Experiment (Sniderman and only five points would be judged as insignif- Piazza 2002). African Americans have their icant. In fact, even when the difference in own culture, it is claimed (Dawson 2001). scores is smallest, the overwhelming num- Although there is a positive sense in which ber of African Americans picked the white this claim may be true, there is also a neg- candidate.23 ative sense in which it is false. The val- The hypothesis is that African Americans ues of the American culture are as much share the core values of the American culture. the values of African Americans as of white If this is true, then they should overwhelm- Americans.21 ingly favor the candidate with the higher To test this hypothesis of shared values, exam score, even if that always means favor- respondents, all of whom are black, are told ing a white candidate over a black candidate. of two young men, one black and the other The null hypothesis, then, is that responses white. Only one of the two can be admit- in the treatment and the control conditions ted. The young white man’s college entrance should not differ. In fact, whether the dif- exam score is always 80; the young black ference between candidates’ SAT scores was man’s exam score is (randomly) 55, 60, 65, large or small, they were equally likely to 70, and 75.22 Respondents are asked which of favor the high scorer – even though the high the two young men should be admitted, if the scorer in the experiment was always the white college can admit only one. student. It is difficult for us to conceive of a Our hypothesis was that African Ameri- more compelling demonstration of the com- cans share the core values of the common mitment of African Americans to the value culture. So far as they do, they should choose of achievement.24 Nor, at a lower rhetorical the young white man because he always has register, to imagine a better example of an the higher exam score. On the other hand, it absence of a treatment effect being evidence for a substantive hypothesis. 20 This is a costly view. Among other things, it produces a publication bias of experiments being regarded as succeeding when they produce differences and failing when they do not. 23 As a test of social desirability, we examined sepa- 21 This is an example of a descriptive as opposed to a rately respondents interviewed by black interviewers, causal hypothesis, although it should not be assigned and they were even more likely to hew to the value of second-class status on this account. The former is achievement than those interviewed by white inter- capable of being as enlightening as the latter, and viewers. better grounded by far. 24 I am curious how many want to bet that white Ameri- 22 Their social class (in the form of their father’s occu- cans would show a similar measure of commitment to pation) is also randomly varied. the value of achievement in an equivalent situation. The Logic and Design of the Survey Experiment 111

Interactions Piazza and I solved the puzzle. It was not blacks in general that evoked an especially God made the world additive, a simplifying supportive response from political conserva- assumption I recommend for theoretical self- tives: it was hard-working blacks distinctively. discipline. But like all simplifying assump- And why did conservatives respond to a hard- tions, it oversimplifies. Consider one of the working black? Precisely because, for them, a first survey experiments that Tom Piazza and hard-working black was the exception, so they I conducted, the Laid-Off Worker Experi- wanted to make an exception for them by hav- ment (Sniderman and Piazza 1993). ing the government help them find another The question that the Laid-Off Worker job. So we argued in our initial study, and so 25 Experiment was designed to investigate was we cross-validated in a follow-up. whether political conservatives discriminate From this experience, I draw two method- against African Americans. Are they as will- ological lessons. We designed the experi- ing to honor a claim for government assis- ment to test the hypothesis that conservatives tance made by a white American as they are racially discriminate (and would have had a one made by a black American? But fram- blessed-on-all-sides career had the Laid-Off ing the question broadly obscures the real Worker Experiment done the job that we question, we reasoned. Supposing that being believed it would do). The result was noth- black made a difference to conservatives, what ing like we anticipated. And that is the first is it about being black that makes a differ- methodological lesson. Surprise is a cogni- ence? Three stigmatizing characterizations tive emotion. And just because the design of of blacks stood out: “lazy” blacks, unmarried experiments requires a definition of expec- black mothers, and young black (stereotyp- tations, experiments can surprise in a way ically aggressive) males. Accordingly, in the that observational analysis cannot. Hypothe- Laid-Off Worker Experiment, respondents ses precede experiments rather than the other are told about a person who has lost her job way around, which is the reason that each is and asked how much help the government designed the way it is. The second method- should give her in finding another. Naturally, ological point I would make is that the expres- the race of the person who has been laid off sion “split-half” should be banished. The randomly varied. But so, too, did the gender, presumption that survey experiments can age, marital-parental status, and work history have only two conditions has handcuffed sur- (dependable vs. undependable). vey experimenters. Complexity is not a value When the data were analyzed, what should in and of itself. To say that an experiment pop up but the finding that political conser- has the right design is to say that it is set vatives are more, not less, likely to favor gov- up in the right way to answer the ques- ernment assistance for a black worker who tion it is designed to answer. And computer- has lost her job than a white worker. “Pop assisted surveys are a breakthrough, among up” is not a scientific term, I recognize. But it other respects, because of the plasticity of the would be a scam to imply that we anticipated designs that they permit. that conservatives would go all out for out- of-work blacks. We had a reasonable expectation that conservatives would be harder on Survey Experiments and Counterfactual blacks than on whites. We never expected Conditionals: Majorities and to find that they would respond with more Counter-Majorities under the Same sympathy and more support for a black who Equilibrium Conditions had lost her job than for a white who sim- I argue that the principal business of sur- ilarly found herself on the street. Nor had vey experiments is to reveal what people are anyone else. The result would discredit the already predisposed to do. Ironically, this whole idea of using randomized experiments in public opinion, I feared. Days of frenzied 25 See the Helping Hand Experiment (Sniderman and analysis followed. On the fourth day, Tom Carmines 1997). 112 Paul M. Sniderman means that they can put us in a position to ditions goes further than the standard inter- explore possible worlds, an example of which pretation of Riker’s (1996) heresthetics. His will make clear what I have in mind. claim is that bringing about a new winning With Ted Carmines, I investigated a coalition requires bringing a new dimension hypothesis about the potential for a break- of cleavage to the fore. Thanks to experi- through in public support for policies to assist ments opening up the exploration of possible blacks. Researchers of symbolic racism main- worlds, one can see how a new winning coali- tain that racial prejudice has a death grip tion can be brought about without bringing a on the American mind. In their view, for new dimension of cleavage to the fore. the grip of racism to weaken, nothing less The second point has to do with the pol- than a change in the hearts and minds of itics of race. Many race specialists in politi- white Americans was necessary. In contrast, cal science have nailed their flag to the claim we believed that there was a political open- that in order to establish a new majority on ing. Revive the moral universalism of the civil the issue of race, a change in the politics of rights movement, we reasoned, and a win- race requires a change in the core values of ning coalition of whites and blacks could be Americans. In contrast, our claim was that it brought into existence. was not necessary to change the hearts and To test this conjecture, we carried out minds of white Americans in order to change a pair of experiments, the Regardless of the politics of race. A counter-majority ready Race Experiment and the Color Blind Exper- to support a politics of race that was morally iment (Sniderman and Carmines 1997).26 universalistic was in existence and already in Both experiments showed that support for position. It would be brought to the surface policies that would help blacks is markedly when a politician was ambitious and clever higher if the arguments made on their behalf enough to mobilize it. It would go too far are morally universalistic, rather than racially to say that our analysis predicted the Obama particularistic. To be sure, conservatives are victory.27 It does not go too far to say that no more likely to support the policy when that it is the only analysis of race and Ameri- a universalistic appeal is made on its behalf can politics that is consistent with it. than when a particularistic one is. But then again, why should they? They are being asked to support a liberal policy. Consistent with 4. A Final View our hypothesis, moderates are markedly more likely to support the policy in the face of The experimental method has made inroads a universalistic, rather than a particularis- on many fronts in political science, but why tic, appeal. Still more telling, so, too, are have survey experiments met with earlier and liberals. broader acceptance? Part of the answer to this This result illustrates a general point about question is straightforward. Survey experi- politics and a specific one about racial politics. ments (and, when I say survey experiments, The general point is this: in politics, more I include the whole family of interview- than one winning coalition can exist under ing modes, from face-to-face to telephone the same equilibrium conditions. There is the to web-based modes) have a lower hurdle majority that one observes, conditional on to jump in meeting requirements of exter- the available political alternatives. But there nal validity. Lower does not mean low, I are the counter-majorities that one would would hastily add. A second part of the answer observe, conditional on different alternatives for the explosive growth in survey experi- or different reasons for choosing between ments is similarly straightforward. Research the same alternatives. This claim of multiple majorities under the same equilibrium con- 27 We had in mind an ambitious and gifted politician such as President Clinton, but, alas, Monica Lewinsky prevented a test of our hypothesis. It never 26 Designing experiments in pairs provides invaluable entered our heads that the country had so progressed opportunities for replication in the same study. that an African American could do so. The Logic and Design of the Survey Experiment 113 areas flourish in inverse proportion to bar- Druckman, James N. 2001b. “On the Limits of riers to entry. With the introduction of the Framing Effects.” Journal of Politics 63: 1041– Multi-Investigator Project studies and then 66. 2001 the enormous advance of TESS as a plat- Druckman, James N. c. “The Implications of Framing Effects for Citizen Competence.” form for survey experiments, the marginal 23 225 56 cost of conducting survey experiments plum- Political Behavior : – . Druckman, James N. 2004. “Political Preference meted. Cooperative election studies, provid- Formation.” American Political Science Review ing teams of investigators the time to carry 98: 671–86. out autonomously designed studies, have Druckman, James N., Cari Lynn Hennessy, Kristi become the third stage of this cost revolution. St. Charles, and Jonathan Weber. 2010.“Com- The importance of these two factors peting Rhetoric over Time: Frames versus should not be underestimated, but a third Cues.” Journal of Politics 72: 136–48. factor is even more important, in my opin- Freedman, David A., Roger Pisani, and Roger Pur- ion. When it comes to survey experiments as ves. 1998. Statistics. New York: Norton, 3–11. a method for the study of politics, the “what” Gaines, Brian J., James H. Kuklinski, and Paul J. Quirk. 2007. “The Logic of the Survey Exper- that is being studied has driven the “how” it 15 1 20 is studied, rather than the other way around. iment Reexamined.” Political Analysis : – . Gibson, James L., and Amanda Gouws. 2003. It is the power of the ideas of generations of Overcoming Intolerance in South Africa: Experi- researchers in the study of public opinion and ments in Democratic Persuasion. New York: Cam- voting, incorporating theoretical frameworks bridge University Press. from the social psychological to the rational, Hacking, Ian. 1983. Representing and Intervening: that has provided the propulsive force in the Introductory Topics in the Philosophy of Natu- use of survey experiments in the study of mass ral Science. New York: Cambridge University politics. Press. Hurwitz, Jon, and Mark Peffley. 1998. Perception and Prejudice: Race and Politics in the United References States. New Haven, CT: Yale University Press. Jackman, Simon, and Paul M. Sniderman. 2006. Apostle, Richard A., Charles Y. Glock, Thomas “The Limits of Deliberative Discussion: A Piazza, and Marijane Suelze. 1983. The Anatomy Model of Everyday Political Arguments.” Jour- of Racial Attitudes. Berkeley: University of Cal- nal of Politics 68: 272–83. ifornia Press. Kuklinski, James H., Paul M. Sniderman, Kathleen Bartels, Larry, and Henry E. Brady. 1993.“The Knight, Thomas Piazza, Philiip E. Tetlock, State of Quantitative Political Methodology.” Gordon R. Lawrence, and Barbara Mellers. In The State of the Discipline II, ed. Ada Finifter. 1997. “Racial Attitudes Toward Affirmative Washington, DC: American Political Science Action.” American Journal of Political Science 41: Association, 121–62. 402–19. Chong, Dennis, and James N. Druckman. 2007a. Merton, Robert K. 1973. The Sociology of Science. “Framing Public Opinion in Competitive Chicago: The University of Chicago Press. Democracies.” American Political Science Review Nelson, Thomas E., and Donald R. Kinder. 1996. 101: 637–55. “Issue Frames and Group-Centrism in Amer- Chong, Dennis, and James N. Druckman. 2007b. ican Public Opinion.” Journal of Politics 58: “Framing Theory.” Annual Review of Political 1055–78. Science 10: 103–26. Riker, William H. 1996. The Strategy of Rhetoric. Converse, Philip E. 1964. “The Nature of Belief New Haven, CT: Yale University Press. Systems in Mass Publics.” In Ideology and Dis- Schuman, Howard, and Stanley Presser. 1981. content, ed. David E. Apter. New York: Free Questions and Answers in Attitude Surveys.New Press, 206–61. York: Academic Press. Dawson, Michael C. 2001. Black Visions. Chicago: Sniderman, Paul M., and Edward G. Carmines. The University of Chicago Press. 1997. Reaching beyond Race. Cambridge, MA: Druckman, James N. 2001a. “Using Credible Harvard University Press. Advice to Overcome Framing Effects.” Journal Sniderman, Paul M., Joseph F. Fletcher, Peter of Law, Economics, & Organization 17: 62–82. Russell, and Philip E. Tetlock. 1996. The Clash 114 Paul M. Sniderman

of Rights: Liberty, Equality, and Legitimacy in Plu- Logic of Issue Framing.” In Studies in Public ralist Democracies. New Haven, CT: Yale Uni- Opinion, eds. Willem Saris and Paul M. Sni- versity Press. derman. Princeton, NJ: Princeton University Sniderman, Paul M., and Thomas Piazza. 1993. Press, 133–65. The Scar of Race. Cambridge, MA: Harvard Uni- Stoker, Laura. 1998. “Understanding Whites’ versity Press. Resistance to Afﬁrmative Action: The Role Sniderman, Paul M., and Thomas Piazza. 2002. of Principled Commitments and Racial Prej- Black Pride and Black Prejudice.Princeton,NJ: udice.” In Perception and Prejudice: Race and Pol- Princeton University Press. itics in the United States, eds. Jon Hurwitz and Sniderman, Paul M., Philip E. Tetlock, and Laurel Mark Pefﬂey. New Haven, CT: Yale University Elms. 2001. “Public Opinion and Democratic Press, 135–70. Politics: The Problem of Non-attitudes and Webb, Eugene, J., Donald T. Campbell, Richard Social Construction of Political Judgment.” In D. Schwartz, and Lee Sechrest. 1996. Unobtru- Citizens and Politics: Perspectives from Political sive Measures: Nonreactive Research in the Social Psychology, ed. James H. Kuklinski. New York: Sciences. New York: Rand McNally. Cambridge University Press, 254–84. Zaller, John R. 1992. The Nature and Origins of Sniderman, Paul M., and Sean M. Theriault. 2004. Mass Opinion. New York: Cambridge Univer- “The Structure of Political Argument and the sity Press. CHAPTER 9

Field Experiments in Political Science

Alan S. Gerber

After a period of almost total absence in of the common methodological deficiencies political science, field experimentation has identified in earlier observational research on become a common research design. In this campaign effects and voter participation. Sec- chapter, I discuss some of the reasons for tion 4 reviews the range of applications of the increasing use of field experiments. field experimentation. In Section 5, I answer Several chapters in this volume provide com- several frequently asked questions about the prehensive introductions to specific experi- limitations and weaknesses of field experimental techniques and detailed reviews of mentation. In Section 6, I briefly discuss some the now extensive field experimental liter- challenges that field experimentation faces atures in multiple areas. This chapter does as it becomes a more frequently employed not duplicate these contributions, but instead methodological approach in political science. provides background, arguments, opinions, This includes a discussion of the external and speculations. I begin by defining field validity of field experimental results and con- experiments in Section 1. In Section 2,Idis- sideration of how difficulties related to repli- cuss the intellectual context for the emer- cation and bias in experimental reporting gence of field experimentation in political might affect the development of field experi- science, beginning with the recent revival ment literatures. of field experimentation in studies of voter 3 turnout. In Section , I describe the sta- 1. Definition tistical properties of field experiments and explain how the approach addresses many In social science experiments, units of observation are randomly assigned to groups, This review draws on previous literature reviews I have and treatment effects are measured by com- authored or coauthored, including Gerber and Green 1 (2008); Davenport, Gerber, and Green (2010); Ger- paring outcomes across groups. Random ber (in press); and Gerber (2004). The author thanks Jamie Druckman, John Bullock, David Doherty, Conor 1 The discussion in this section draws on Gerber and Dowling, and Eric Oliver for helpful comments. Green (2008).

115 116 Alan S. Gerber assignment permits unbiased comparisons Congress through an experiment in which because randomization produces groups congressional staffers made scheduling deci- that, prior to the experimental interven- sions after being told whether the meeting is tion, differ with respect to both observ- sought by a political action committee rep- able and unobservable attributes only due resentative or a constituent. The natural field to chance. Field experiments seek to com- experiment, which is the design often referred bine the internal validity of randomized to in political science as a “field experiment,” experiments with increased external valid- is the same as the framed field experiment, ity, or generalizability, gained through except that it involves subjects who naturally conducting the experiment in real-world undertake the task of interest in its natural settings. Field experiments aim to repro- environment and who are unaware they are duce the environment in which the phe- participating in a study. Research in which nomenon of interest naturally occurs and political campaigns randomly assign house- thereby enhance the external validity of the holds to receive campaign mailings to test experiment. the effect of alternative communications on Experiments have many dimensions, voter turnout is one example of a natural field including type of subjects, experimental experiment. environment, treatments, outcome measure- This chapter focuses on natural field ments, and subject awareness of the experi- experiments. Although the degree of natu- ment. The degree to which each dimension ralism in field experiments is the distinc- parallels the real-world phenomenon of inter- tive strength of the method, it is important est may vary, leading to a blurring of the dis- to keep in mind that the goal of most exper- tinction between what is and is not a field imental interventions is to estimate a causal experiment. The economists Harrison and effect, not to achieve realism. It might appear List (2004) propose a system for classifying from the classification system that the move- studies according to their varying degrees ment from conventional lab experiment to of naturalism. Per their taxonomy, the least natural field experiment is similar to “the naturalistic experimental study is the con- ascent of man,” but that is incorrect. The ventional lab experiment. This familiar study importance of naturalism along the vari- involves an abstract task, such as playing a ous dimensions of the experimental design standard game (e.g., prisoner’s dilemma, dic- will depend on the research objectives and tator game), and employs the typical student whether there is concern about the assump- subject pool. The artefactual field experiment tions required for generalization. Consider is a conventional laboratory study with non- the issue of experimental subjects. If the standard subjects. That is, the study features researcher aims to capture basic psycholog- subjects that more closely parallel the real- ical processes that may be safely assumed world actors of interest to the researcher to be invariant across populations, experi- rather than a subject pool drawn from the mental contexts, or subject awareness of the university community. Examples of this work experiment, then nothing is lost by using a include Habyarimana and colleagues (2007), conventional lab experiment. That said, who investigate ethnic cooperation through understanding behavior of typical popula- an exploration of the degree of altruism dis- tions in natural environments is frequently played in dictator games. This study was con- the ultimate goal of social science research, ducted in Africa and drew its subjects from and it is a considerable, if not impossible, various ethnic groups in Kampala, Uganda. challenge to even recognize the full set of The framed field experiment isthesameas threats to external validity present in arti- the artefactual field experiment, except that ficial contexts, let alone to adjust the mea- the task is more naturalistic. An example is sured experimental effects and uncertainty to Chin, Bond, and Geva’s (2000) study of the account for these threats (Gerber, Green, and effect of money on access to members of Kaplan 2004). Field Experiments in Political Science 117

2. Intellectual Context for in the Vietnam War on veteran’s earnings, Emergence of Field Experiments where the draft lottery draw altered the likelihood of military service, and Angrist and General Intellectual Environment Krueger’s (1991) use of birthdates and mini- The success of randomized clinical trials mum age requirements for school attendance in medicine provided a general impetus for to estimate the effect of educational attain- exploring the application of similar methods ment on wages. to social science questions. The first large- scale randomized experiment in medicine – Development of Field Experimentation in the landmark study of the effectiveness of Political Science streptomycin in treating tuberculosis (Med- ical Research Council 1948) – appeared The earliest field experiments in political shortly after World War II. In the years since science were performed by Harold Gosnell then, the use of randomized trials in clini- (1927), who investigated the effect of get out cal research has grown to where this method the vote (GOTV) mailings in the 1924 pres- now plays a central role in the evaluation of idential election and 1925 Chicago mayoral 3 medical treatments.2 The prominence of ran- election. In the 1950s, Eldersveld (1956) domized trials in medicine led to widespread conducted a randomized field experiment familiarity with the method and an appre- to measure the effects of mail, phone, and ciation of the benefits of using random canvassing on voter turnout in Ann Arbor, assignment to measure the effectiveness of Michigan. These pioneering experiments had interventions. only a limited effect on the trajectory of With some important exceptions, such as subsequent political science research. Field the negative income tax experiments of the experimentation was a novelty and, when late 1960s and 1970s, there were relatively few considered at all, was dismissed as impractical social science field experiments prior to the or of limited application (Gerber and Green 1990s. The increased use of field experimen- 2008). In fact, the method was so rarely used tation in the social sciences emerged from that there were no field experiments pub- an intellectual climate of growing concern lished in any major political science journals about the validity of the key assumptions sup- in the 1990s. porting observational research designs and The recent revival of field experiments growing emphasis on research designs in in political science began with a series of which exogeneity assumptions were more experimental studies of campaign activity plausible. By the late 1980s, the extreme (Gerber and Green 2000; Green and Ger- difficulty in estimating causal effects from ber 2004). The particular focus of the lat- standard observational data was increasingly est generation of field experimentation on appreciated in the social sciences, especially measuring the effect of political communi- in economics (e.g., LaLonde 1986). In the cations can be traced to persistent method- field of labor economics, leading researchers ological and substantive concerns regarding began searching for natural experiments to the prior work in this area. To provide the overcome the difficulties posed by unob- intellectual context for the revival of field servable factors that might bias regression experimentation in political science, I briefly estimates. The result was a surge in studies review the literature on campaign effects at that investigated naturally occurring randomizations or near-randomized applications of a 3 Gosnell assembled a collection of matched pairs of streets and selected one of the pairs to get the treat- “treatment.” Examples of this work include ment, but it is not entirely clear whether Gosnell Angrist’s (1990) study of the effect of serving used random assignment to decide which pair was to be treated. Given this ambiguity, it might be more 2 For a comparison of medical research and social sci- appropriate to use a term other than “experiment” ence research, see Gerber, Doherty, and Dowling to describe the Gosnell studies, perhaps “controlled (2009). intervention.” 118 Alan S. Gerber the time of the Gerber and Green (2000) Table 9.1: Approximate Cost of Adding New Haven experiment. This prior literature One Vote to Candidate Vote Margin includes some of the best empirical political science studies of their time. However, Incumbent Challenger although the research designs used to study campaign spending effects and voter mobi- Jacobson (1985)$250/vote $16/vote 1988 20 17 lization were often ingenious, these extensive Green and Krasno ( )$/vote $ /vote Levitt (1994)$488/vote $146/vote literatures suffered from important method- 2000 61 32 ological weaknesses and conflicting findings. Erikson and Palfrey ( )$ /vote $ /vote Many of the methodological difficulties are Notes: 2008 dollars. Calculations are based on successfully addressed through the use of field 190,000 votes cast in a typical House district. For experimentation. House elections, this implies that a 1% boost in Consider first the work on the effect of the incumbent’s share of the vote increases the campaign spending on election outcomes incumbent’s vote margin by 3,800 votes. circa 1998, the date of the first modern voter Source: Adapted from Gerber (2004). mobilization experiment (Gerber and Green 2000). This literature did not examine the candidate spending between the initial con- effects of specific campaign activities, but test and rematch, a strategy that serves to dif- rather the relationship between overall Fed- ference away difficult to measure district- or eral Election Commission reported spend- candidate-level variables that might be lurking levels and candidate vote shares.4 There ing in the error term. were three main approaches to estimating Unfortunately, the alternative research the effect of campaign spending on candidate designs produce dramatically different 9 1 vote shares. In the earliest work, Jacobson results. Table . reports, in dollar per vote and others (e.g., Jacobson 1978, 1985, 1990, terms, the cost per additional vote implied 1998; Abramowitz 1988) estimated spending by the alternative approaches. The dollar effects using ordinary least squares regres- figures listed are the cost of changing the vote 6 9 1 sions of vote shares on incumbent and chal- margin by one vote. Table . illustrates lenger spending levels. This strategy assumes the dramatic differences in the implications that spending levels are independent of omit- of the alternative models and underscores ted variables that also affect vote share. Con- how crucial modeling assumptions are in cern that this assumption was incorrect was this line of research. Depending on the heightened by the frequently observed neg- research design, it is estimated to cost as 500 20 ative correlation between incumbent spend- much as $ or as little as $ to improve the 2004 ing and incumbent vote share. In response vote margin by a single vote (Gerber ). to this potential difficulty, there were two However, it is not clear which estimates are main alternative strategies. First, some schol- most reliable because each methodological approach relies on assumptions that are ars proposed instrumental variables for candi- 7 date spending levels (e.g., Green and Krasno vulnerable to serious critiques. The striking 1988 1996 ; Ansolabehere and Snyder ; Gerber 6 1998 5 1994 If a campaign activity causes a supporter who would ). Second, Levitt ( ) examined the otherwise have stayed home on Election Day to vote, performance of pairs of candidates who faced then this changes the vote margin by one vote. If a each other more than once. The change in campaign activity causes a voter to switch candidates, then this changes the vote margin by two votes. For vote shares between the initial contest and further details about these calculations, see Gerber rematch were compared to the changes in (2004). 7 The ordinary least squares estimate relies on the questionable assumption that spending levels are 4 There were some exceptions (e.g., Ansolabehere and uncorrelated with omitted variables. The instrumen- Gerber 1994). tal variables approach relies on untestable assump- 5 A closely related approach was taken by Erikson and tions about the validity of the instruments. Levitt’s Palfrey (2000), who use a theoretical model to deduce study uses a small nonrandom subset of election con- conditions under which candidate spending levels tests, so there is a risk that these elections are atypical. could be treated as exogenous. Furthermore, restricting the sample to repeat elec- Field Experiments in Political Science 119

Table 9.2. Voter Mobilization Experiments Prior to 1998 New Haven Experiment

n of Subjects (Including Effects on Study Date Election Place Control Group) Treatment Turnout∗

Gosnell (1927) 1924 Presidential Chicago 3,969 registered Mail +1% voters Gosnell (1927) 1925 Mayoral Chicago 3,676 registered Mail +9% voters Eldersveld (1956) 1953 Municipal Ann Arbor, 41 registered voters Canvass +42% MI 43 registered voters Mail +26% Eldersveld (1956) 1954 Municipal Ann Arbor, 276 registered voters Canvass +20% MI 268 registered voters Mail +4% 220 registered voters Phone +18% Miller et al. (1981) 1980 Primary Carbondale, 79 registered voters Canvass +21% IL 80 registered voters Mail +19% 81 registered voters Phone +15% Adams and Smith 1979 Special city Washington, 2,650 registered Phone +9% (1980) council DC voters

∗ These are the effects reported in the tables of these research reports. They have not been adjusted for contact rates. Notes: In Eldersveld’s 1953 experiment, subjects were those who opposed or had no opinion about charter reform. In 1954, subjects were those who had voted in national but not local elections. This table includes only studies that use random experimental design (or near random, in the case of Gosnell [1927]). Source: Adapted from Gerber, Green, and Nickerson (2001). diversity of results in the campaign spending vote. Indeed, as the campaign spending lit- literature, and the sensitivity of the results erature progressed, a parallel and indepen- to statistical assumptions, suggested the dent literature on the effects of campaign potential usefulness of a fresh approach to mobilization on voter turnout was develop- measuring the effect of campaign activity on ing. What did these observational and exper- voter behavior. imental studies say about the effectiveness of One feature of the campaign spending lit- voter mobilization efforts? erature is that it typically uses overall cam- As previously mentioned, at the time of the paign spending as the independent variable. 1998 New Haven study (Gerber and Green Overall campaign spending is an amalgama- 2000), there was already a small field exper- tion of spending for particular purposes, and imental literature on the effect of campaigns so the effectiveness of overall spending is on voter turnout. Table 9.2 summarizes the determined by the effectiveness of particu- field experiment literature prior to the 1998 lar campaign activities, such as voter mobi- New Haven experiment. By far, the largest lization efforts. This suggests the value of previous study was from Gosnell (1927), who obtaining a reasonable dollar per vote esti- measured the effect of nonpartisan mail on mate for the cost of inducing a supporter to voter turnout in Chicago. In this pioneering research, 8,000 voters were divided by tions may reduce, but not fully eliminate, the biases street into treatment (GOTV mailings) and due to omitted variables because changes in spend- control group. Three decades later, Elder- ing levels between the initial election and the rematch sveld conducted a randomized intervention (which is held at least two, and sometimes more, years later) may be correlated with unobservable changes during a local charter reform vote to mea- in variables correlated with vote share changes. sure the effectiveness of alternative campaign 120 Alan S. Gerber tactics. He later analyzed the effect of a drive lization effects is challenging. Although the to mobilize apathetic voters in an Ann Arbor internal validity of such studies is impressive, municipal election (Eldersveld and Dodge the magnitude of the laboratory effects may 1954; Eldersveld 1956). These experiments not provide a clear indication of the magni- measured the turnout effects of a variety of tude of treatment effects in naturalistic con- different modes of communications. In the texts. More generally, although it is often following years, only a handful of scholars remarked that a laboratory experiment will performed similar research. Miller, Bositis, reliably indicate the direction but not the mag- and Baer (1981) examined the effects of a nitude of the effect that would be observed in letter sent to residents of a precinct in Car- a natural setting, to my knowledge this has bondale, Illinois, prior to the 1980 general not been demonstrated, and it is not obvi- 8 election. Adams and Smith (1980) conducted ously correct in general or specific cases. an investigation of the effect of a single thirty- Furthermore, despite the careful efforts of second persuasion call on turnout and candi- researchers such as Ansolabehere and Iyengar date choice in a special election for a Wash- to simulate a typical living room for conduct- ington, DC, city council seat. In sum, prior to ing the experiment, the natural environment 1998, only a few field experiments on mobi- differs from the laboratory environment in lization – spread across a range of politi- many obvious and possibly important ways, cal contexts and over many decades – had including the subject’s awareness of being been conducted. Nevertheless, these studies monitored. formed a literature that might be taken to Complementing the experimental evi- support several tentative conclusions. First, dence, there is a large amount of obser- the effects of voter contacts appeared to be vational research on campaigns and voter extremely large. Treatment effects of twenty turnout. As of 2000, the most influential percentage points or more are common in work on turnout was survey-based analyses of these papers. Thus, we might conclude that the causes of participation. Rosenstone and voters can be mobilized quite easily, and Hansen’s (1993) book is a good example of because mobilizing supporters is a key task, by the state of the art circa 1998 (see also Verba, implication even modest campaign resource Schlozman, and Brady 1995). This careful disparities will play an important role in elec- study is an excellent resource that is consulted tion results. Second, there is no evidence that and cited by nearly everyone who writes about the effect of contacts had changed over the turnout (according to Google Scholar, as of many years that separated these studies – the June 30, 2010, more than 1,500 times), and effectiveness of mailings in the 1980swas the style of analysis employed is still common as great as what had been found in earlier in current research. Rosenstone and Hansen decades. use the American National Election Studies In addition to these early field experi- (ANES) to measure the effect of campaign ments, another important line of work on contacts on various measures of political par- campaign effects used laboratory experiments ticipation. They report the contribution of to investigate how political communications many different causes of participation in pres- affect voter turnout. A leading example is the idential and midterm years (see tables 5.1 and study by Ansolabehere and Iyengar (1996), 5.2 in Rosenstone and Hansen 1993) based which finds that exposure to negative cam- on estimates from a pooled cross-sectional paign advertisements embedded in mock news broadcasts reduced subjects’ reported 8 For instance, the natural environment will provide intention to vote, with effects particularly the subject more behavioral latitude, which might reverse the lab findings. For example, exposure to large among independent voters. As with field negative campaigning might create an aversion to experiments, these laboratory studies use ran- political engagement in a lab context, but negative dom assignment. However, integrating the information may pique curiosity about the advertising claims, which, outside the lab, could lead to results of these important and innovative lab- increased information search and gossiping about oratory studies into estimates of voter mobi- politics, and in turn greater interest in the campaign. Field Experiments in Political Science 121 analysis of ANES survey data. The estimated bility that those who report campaign contact effect of campaign contact on reported voter are different from those who do not report turnout is approximately an eight to ten per- contact in ways that are not adequately cap- centage point boost in turnout probability. tured by the available control variables. The This sizable turnout effect from campaign Gerber and Green (2000) study and subse- contact is of similar magnitude to many of quent field experiments were in many ways those reported in the field experiments from an attempt to address this selection bias and the 1920s through the 1980s. The sample size other possible weaknesses of the earlier work. used in the Rosenstone and Hansen (1993) study is impressive (N is 11,310 for presidential years, 5,124 for mid-term elections), 3. How Do Experiments Address giving the estimation results the appearance Problems in Prior Research? of great precision. However, there are several methodological and substantive reasons why In this section, I present a framework for ana- the findings might be viewed as unreliable. lyzing causal effects and apply the framework First, the results from the survey-based voter to describe how field experiments eliminate mobilization research appear to be in tension some of the possible bias in observational with at least some of the aggregate campaign studies. For concreteness, I use the Rosen- spending results. If voters can be easily mobi- stone and Hansen (1993) study as a run- lized by a party contact, then it is difficult to ning example. In their participation study, understand why a campaign would have to some respondents are contacted by cam- spend so much to gain a single vote (Table paigns and others are not. In the language 9.1). Rather, modest amounts of spending of experiments, some subjects are “treated” should yield large returns. This tension could (contacted) and others are “untreated” (not perhaps be resolved if there are large differ- contacted). The key challenge in estimat- ences between average and marginal returns ing the causal effect of the treatment is that to mobilization expenditures or if campaign the analyst must somehow use the available spending is highly inefficient. Nevertheless, data to construct an estimate of a counterfac- taking the survey evidence and the early field tual: what outcome would have been observed experiments on voter mobilization seriously, for the treated subjects had they not been if a campaign contact in a presidential year treated? The idea that for each subject there boosts turnout by eight percentage points is a “potential outcome” in both the treated and a large share of partisans in the ANES and the untreated state is expressed using the report not being contacted, then it is hard notational system termed the “Rubin causal to simultaneously believe both the mobiliza- model” after Rubin (1978, 1990).10 To focus tion estimates and the findings (summarized on the main ideas, I initially ignore covariates. in Table 9.1), suggesting that campaigns must For each individual i,letYi0 be the outcome spend many hundreds of dollars per vote.9 if i does not receive the treatment (in this More important, the survey work on turnout example, contact by the mobilization effort) effects is vulnerable to a number of method- and Yi1 be the outcome if i receives the treat- ological criticisms. The key problem in the ment. The treatment effect for individual i is survey-based observational work is the possi- defined as

τ = − . 9 Compounding the confusion, the results presented i Y i1 Y i0 (1) in the early field experimental literature may overstate the mobilization effects. The pattern of results in those studies suggests the possibility that the The treatment effect for individual i is the effect sizes are exaggerated due to publication- difference between the outcomes for i in two related biases. There is a strong negative relationship between estimates and sample size, a pattern consis- possible, although mutually exclusive, states tent with inflated reports due to file drawer problems or publication based on achieving conventional levels 10 This section draws heavily on, and extends the dis- of statistical significance (Gerber et al. 2001). cussion in, Gerber and Green (2008). 122 Alan S. Gerber of the world – one in which i receives the treatment effect for the treated plus a selec- treatment and another in which i does not. tion bias term. The selection bias is due to the Moving from a single individual to the aver- difference in the outcomes in the untreated age for a set of individuals, the average treat- state for those treated and those untreated. ment effect for the treated (ATT) is defined This selection bias problem is a critical issue as addressed by experimental methods. Random assignment forms groups without reference = τ | = = | = AT T E ( i Ti 1) E (Y i1 Ti 1) to either observed or unobserved attributes − | = , of the subjects and, consequently, creates E (Y i0 Ti 1) (2) groups of individuals that are similar prior to = application of the treatment. When groups where E stands for a group average and Ti 1 | = are formed through random assignment, the when a person is treated. In words, Yi1 Ti 1 is the post-treatment outcome among those group randomly labeled the control group | = has the same expected average outcome in who are treated, and Yi0 Ti 1 is the outcome for i that would have been observed if the untreated state as the set of subjects des- those who are treated had not been treated. ignated at random to receive the treatment. Equation (2) suggests why it is difficult to esti- The randomly assigned control group can mate a causal effect. Because each individual therefore be used to produce an unbiased esti- is either treated or not, for each individual we mate of what the outcome would have been observe either Y1 or Y0. However, to calculate for the treated subjects, had the treated sub- Equation (2) requires both quantities for each jects remained untreated, thereby avoiding treated individual. In a dataset, the values of selection bias. Y1 are observed for those who are treated, The critical assumption for observational but the causal effect of the treatment cannot work to produce unbiased treatment effect be measured without an estimate of what the estimates is that, controlling for covari- average Y would have been for these individ- ates (whether through regression or through | = 1 = | = 0 uals had they not been treated. Experimen- matching), E(Yi0 Ti ) E(Yi0 Ti )(i.e., tal and observational research designs employ apart from their exposure to the treatment, different strategies for producing estimates of the treated and untreated group outcomes are this counterfactual. Observational data anal- on average the same in the untreated state). ysis forms a comparison group using those Subject to sampling variability, this will be who remain untreated. This approach gener- true by design when treatment and control ates selection bias in the event that the out- groups are formed at random. In contrast, comes in the untreated state for those who are observational research uses the observables untreated are different from the outcomes in to adjust the observed outcomes and thereby the untreated state for those who are treated. produce a proxy for the treated subject’s In other words, selection bias occurs if the potential outcomes in the untreated state. differences between those who are and are If this effort is successful, then there is no not treated extend beyond exposure to the selection bias. Unfortunately, without a clear treatment. Stated formally, the observational rationale based on detailed knowledge of why comparison of the treated and the untreated some observations are selected for treatment compares: and others are not, this assumption is rarely convincing. Consider the case of estimat- | = − | = ing the effect of campaign contact on voter E (Y i1 Ti 1) E (Y i0 Ti 0) turnout. First, there are likely to be impor- = [E (Y 1|T = 1) − E (Y 0|T = 1)] i i i i tant omitted variables correlated with cam- + | = 1 − | = 0 [E (Y i0 Ti ) E (Y i0 Ti )] paign contact that are not explained by the = ATT + Selection bias. (3) included variables. Campaigns are strategic and commonly use voter files to plan which A comparison of the average outcomes for the households to contact. A key variable in many treated and the untreated equals the average campaign targeting plans is the household’s Field Experiments in Political Science 123 history of participation, and households that ment error could lead to exaggeration of do not vote tend to be ignored. The set treatment effects. In the case of survey-based of control variables available in the ANES voter mobilization research, there is empir- data, or other survey datasets, does not com- ical support for concern that misreporting monly include vote history or other variables of treatment status leads to overestimation that might be available to the campaign for of treatment effects. Research has demon- its strategic planning. Second, past turnout strated both large amounts of misreporting is highly correlated with current turnout. and also a positive correlation between mis- | = Therefore, E(Yi0 Ti 1) may be substan- reporting having been contacted and misre- | = tially higher than E(Yi0 Ti 0). Moreover, porting having voted (Vavreck 2007; Gerber although it may be possible to make a reason- and Doherty 2009). able guess at the direction of selection bias, There are some further difficulties with analysts rarely have a clear notion of the mag- survey-based observational research that are nitude of selection bias in particular applica- addressed by field experiments. In addition tions, so it is uncertain how estimates may be to the uncertainty regarding who was corrected.11 assigned the treatment, it is sometimes In addition to selection bias, field exper- unclear what the treatment was because sur- iments address a number of other common vey measures are sometimes not sufficiently methodological difficulties in observational precise. For example, the ANES item used work, many of these concerns related to for campaign contact in the Rosenstone and measurement. In field experiments, the ana- Hansen study asks respondents: “Did anyone lyst controls the treatment assignment, so from one of the political parties call you up there is no error in measuring who is tar- or come around and talk to you about the geted for treatment. Although observational campaign?” This question ignores nonparti- studies could, in principle, also measure the san contact; conflates different modes of com- treatment assignment accurately, in practice munication, grouping together face-to-face analysis is frequently based on survey data, canvassing, volunteer calls, and commercial which relies on self-reports. Again, consider calls (while omitting important activities such the case of the voter mobilization work. Con- as campaign mailings); and does not measure tact is self-reported (and, for the most part, so the frequency or timing of contact. is the outcome, voter turnout). When there In addition to the biases discussed thus is misreporting, the collection of individu- far, another potential source of difference als who report receiving the treatment are between the observational and experimental in fact a mix of treated and untreated individ- estimates is that those who are treated outside uals. By placing untreated individuals in the the experimental context may not be the same treated group and treated individuals in the people who are treated in an experiment. If untreated group, random misclassification those who are more likely to be treated in the will tend to attenuate the estimated treatment real world (perhaps because they are likely to effects. In the extreme case, where the survey be targeted by political campaigns) have espe- report of contact is unrelated to actual treat- cially large (or small) treatment effects, then ment status or individual characteristics, the an experiment that studies a random sam- difference in outcomes for those reporting ple of registered voters will underestimate (or treatment and those not reporting treatment overestimate) the ATT of what may often be will vanish. In contrast, systematic measure- the true population of interest – those individuals most likely to be treated in typical cam- 11 This uncertainty is not contained in the reported paigns. A partial corrective for this is weight- standard errors, and, unlike sampling variability, it remains undiminished as the sample size increases ing the result to form population proportions (Gerber, Green, and Kaplan 2004). The conven- similar to the treated population in natural tional measures of coefficient uncertainty in observa- settings, although this would fail to account tional research thereby underestimate the true level of uncertainty, especially in cases where the sample for differences in treatment effects between size is large. those who are actually treated in real-world 124 Alan S. Gerber settings and those who “look” like them but gies, such as e-mail and text messaging (Dale are not treated. and Strauss 2007). Field experiments have Finally, although this discussion has also measured the effect of novel approaches focused on the advantages of randomized to mobilization, such as Election Day parties experiments over observational studies, field at the polling place (Addonizio, Green, and experimentation also has some advantages Glaser 2007). over conventional laboratory experimenta- The results of these studies are compiled tion in estimating campaign effects. Briefly, in a quadrennial review of the literature, Get field experiments of campaign communi- Out the Vote!, the latest version of which was cations typically study the population of published in 2008 (Green and Gerber 2008). registered voters (rather than a student pop- A detailed review of the literature over the ulation or other volunteers), measure behav- past ten years is also contained in Nickerson ior in the natural context (versus a university and Michelson’s chapter in this volume. A laboratory or a “simulated” natural environ- meta-analysis of the results of dozens of can- ment; also, subjects are typically unaware of vassing, mail, and phone studies shows that the field experiment), and typically estimate the results from the initial New Haven study the effect of treatments on the actual turnout have held up fairly well. Canvassing has a (rather than on a surrogate measure such as much larger effect than do the less personal stated vote intention or political interest). modes of communication, such as phone and mail. The marginal effect of brief commercial calls, such as those studied in New Haven, 4. Development and Diffusion and nonpartisan mailings appears to be less of Field Experiments in than one percentage point, whereas canvass- Political Science ing boosts turnout by about seven percentage points in a typical electoral context.12 The details of the 1998 New Haven study In recent years, field experimentation has are reported in Gerber and Green (2000). moved well beyond the measurement of Since this study, which assessed the mobiliza- voter mobilization strategies and has now tion effects of nonpartisan canvassing, phone been applied to a broad array of questions. calls, and mailings, more than 100 field exper- Although the first papers were almost entirely iments have measured the effects of political by American politics specialists, comparative communications on voter turnout. The num- politics and international relations scholars ber of such studies is growing quickly (more are now producing some of the most excit- than linearly), and dozens of researchers ing work. Moreover, the breadth of topics have conducted voter mobilization experi- in American politics that researchers have ments. Some studies essentially replicate the addressed using field experiments has grown New Haven study and consider the effect immensely. A sense of the range of applica- of face-to-face canvassing, phone, or mail tions can be gained by considering the top- in new political contexts, including other ics addressed in a sampling of recent studies countries (e.g., Guan and Green 2006; Ger- using field experiments: ber and Yamada 2008; John and Brannan r 2008). Other work looks at new modes of Effect of partisanship on political attitudes: communication or variations on the sim- Gerber, Huber, and Washington (2010) ple programs used in New Haven, includ- study the effect of mailings informing ing analysis of the effect of phone calls or unaffiliated, registered voters of the need contacts by communicators matched to the to affiliate with a party to participate in ethnicity of the household (e.g., Michelson the upcoming closed primary. They find 2003), repeat phone calls (Michelson, Garcıa´ that the mailings increase formal party 2009 Bedolla, and McConnell ), television and 12 2008 For details, see appendixes A, B, and C of Get Out radio (Panagopoulos and Green ; Ger- the Vote: How to Increase Voter Turnout (Green and ber, Gimpel et al. in press), and new technolo- Gerber 2008). Field Experiments in Political Science 125 r affiliation and, using a post-treatment sur- Effect of Election Day institutions on election vey, they find a shift in partisan identifica- administration: Hyde (2010) studies the effect tion, as well as a shift in political attitudes. of election monitors on vote fraud levels. r r Influence of the media on politics: Gerber, Effect of lobbying on legislative behavior: Karlan, and Bergan (2009) randomly pro- Bergan (2009) examines the effect of a vide Washington Post and Washington Times lobbying effort on a bill in the New Hamp- newspaper subscriptions to respondents shire legislature. An e-mail from an inter- prior to a gubernatorial election to exam- est group causes a statistically significant ine the effect of media slant on voting increase in roll call voting for the spon- behavior. They find that the newspapers sor’s measure. r increase voter participation and also shift Effect of constituency opinion on legislator voter preference toward the Democratic behavior: Butler and Nickerson (2009) candidate. examine the effect of constituency opinion r Effect of interpersonal influence: Nickerson on legislative voting. They find that mail- (2008) analyzes the effect of a canvassing ing legislators polling information about effort on household members who are not an upcoming legislative measure results in directly contacted by the canvasser. He changes in the pattern of roll call support finds that spouses and roommates of those for the measure. r who are contacted are also more likely to Effect of voter knowledge on legislative behav- vote following the canvassing treatment. ior: Humphreys and Weinstein (2007) r Effect of mass media campaigns: Gerber, examine the effect of legislative per- Gimpel, et al. (in press) analyze the effect formance report cards on representa- of a multimillion-dollar partisan television tives’ attendance records in Uganda. They advertising campaign. Using tracking polls find that showing legislators’ attendance to measure voter preferences each day, records to constituents results in higher they find a strong but short-lived boost in rates of parliamentary attendance. r the sponsor’s vote share. Effect of social pressure on political participa- r Effect of candidate name recognition: tion: Gerber, Green, and Larimer (2008) Panagopoulos and Green (2008) measure investigate the effect of alternative mail- the effect of radio ads that boost name ings, which exert varying degrees of social recognition in low salience elections. They pressure. They find that a preelection find that ads that provide equal time to mailing that lists the recipient’s own vot- both the incumbent’s name and chal- ing record and a mailing that lists the vot- lenger’s name have the effect of boosting ing record of the recipient and his or her the (relatively unknown) challenger’s vote neighbors caused a dramatic increase in performance. turnout. r r Effect of partisan political campaigns: Media and interethnic tension/prejudice Wantchekon (2003) compares broad pol- reduction: Paluck and Green (2009) con- icy versus narrow clientelistic campaign duct a field experiment in postgeno- messages in a 2001 Benin election. Ger- cide Rwanda. They randomly assign some ber (2004) reports the results of a 1999 communities to a condition where they are partisan campaign. provided with a radio program designed to r Effect of political institutions and policy out- encourage people to be less deferential to comes and legitimacy:Olken(2010) com- authorities. The findings demonstrate that pares the performance of alternative insti- listening to the program makes listeners tutions for the selection of a public good more willing to express dissent. in Indonesia. He finds that, although more participatory institutions do not Mickelson and Nickerson’s chapter and change the set of projects approved, partic- Wantchekon’s chapter in this volume provide ipants are more satisfied with the decision- excellent discussions of recent field experi- making process. ments many further examples. In addition to 126 Alan S. Gerber addressing important substantive questions, well. Third, field experimentation has spread field experiments can make methodological from initial application in American politics contributions, such as assessing the perfor- to comparative politics and international rela- mance of standard observational estimation tions. Fourth, field experiments are now used methods. In this line of research, data from to study both common real-world phenom- experimental studies are reanalyzed using ena (e.g., campaign television commercials observational techniques. The performance or the effect of election monitors), as well of the observational estimation method as novel interventions for which there are is evaluated by comparing the estimation no observational counterparts (unusual mail- results from an application of the obser- ings or legislative report cards in developing vational method with the unbiased exper- countries). For these novel interventions, of imental estimates. Arceneaux, Gerber, and course, no observational study is possible. Green (2006) conduct one such comparison by assessing the performance of regression and matching estimators in measuring 5. Frequently Asked Questions the effects of experimental voter mobilization phone calls. The study compares the experi- Field experiments are not a panacea, and mental estimates of the effect of a phone call there are often substantial challenges in the (based on a comparison of treatment and con- implementation, analysis, and interpretation trol group) and the estimates that would have of findings. For an informative recent dis- been obtained had the experimental dataset cussion of some of the limitations of field been analyzed using observational techniques experiments, see Humphreys and Weinstein (based on a comparison of those whom the (2009) and especially Deaton (2009); for a researchers were able to successfully contact reply to Deaton, see Imbens (2009). Rather by phone and those not contacted). They find than compile and evaluate a comprehensive that exact matching and regression analysis list of potential concerns and limitations, I overestimate the effectiveness of phone calls, provide in this section a somewhat infor- producing treatment effect estimates several mal account of how I address some of the times larger than the experimental estimates. questions I am frequently asked about field Reviewing these contributions, both subexperiments.13 The issue of the external valid- stantive and methodological, a collection that ity of field experiments is left for Section 7. is only part of the vast body of recent work, Some field experiments have high levels of shows the depth and range of research in noncompliance due to the inability to treat political science using field experimentation. all of those assigned to the treatment group The earliest studies have now been replicated (low contact rates). Other methods, such as many times, and new studies are branching lab experiments, seem to have perfect com- into exciting and surprising areas. I doubt that pliance. Does this mean field experiments are ten years ago anyone could have predicted biased? the creativity of recent studies and the range Given that one-sided noncompliance (i.e., of experimental manipulations. From essen- the control group remains untreated, but tially zero studies just more than a decade ago, some of those assigned to the treatment group field experimentation is now a huge enter- are not treated) is by far the most comprise. I draw several conclusions about recent mon situation in political science field exper- developments. First, voter mobilization is still iments, the answer addresses this case. If the studied, but the research focus has shifted researcher is willing to make some important from simply measuring the effectiveness of technical assumptions (see Angrist, Imbens, campaign communications to broader theo- and Rubin [1996] for a formal statement retical issues such as social influence, norm of the result) when there is failure to treat compliance, collective action, and interper- in a random experiment, a consistent (large sonal influence. Second, there has been a 13 For more detailed treatment of these and other issues move from studying only political behav- regarding field experiments, see Gerber and Green ior to the study of political institutions as (2011). Field Experiments in Political Science 127

Hash-marked area is the change in Y due to treatment group assignment

Y1(1) Y2(1)

Y3(1) Y2(0) Y2(0)

Y1(0) Y1(0)

Y3(0) Y3(0)

p1 p1 +p 2 1 p1 p1 +p 2 1 Panel A: Treatment Group Panel B: Control Group

Figure 9.1. Graphical Representation of Treatment Effects with Noncompliance = 1 + 1 + 1 − − 0 Note: Y T p1Y 1( ) p2Y 2( ) ( p1 p2)Y 3( ) = 0 + 0 + 1 − − 0 Y C p1Y 1( ) p2Y 2( ) ( p1 p2)Y 3( ) = = 0 = 1 Yi(X) Potential outcome for type i when treated status is X (X is untreated, X is treated). The Y axis measures the outcome, whereas the X axis measures the proportion of the subjects of each type. sample unbiased) estimate of the average of type i when treated (X = 1) or untreated treatment effect on those treated can be esti- (X = 0). Individuals are arrayed by group, mated by differencing the mean outcome for with the X axis marking the population pro- those assigned to the treatment and control portion of each type and the Y axis indicat- groups and dividing this difference by the ing average outcome levels for subjects in proportion of the treatment group that is each group. Panel A depicts the subjects when actually treated. they are assigned to the treatment group, and The consequences of failure to treat panel B shows the subjects when assigned to are illustrated in Figure 9.1, which depicts the control group. (Alternatively, Figure 9.1 the population analogues for the quantities can be thought of as depicting the poten- that are produced by an experiment with tial outcomes for a large population sample, noncompliance.14 Figure 9.1 provides some with some subjects randomly assigned to the important intuitions about the properties and treatment group and others to the control limitations of the treatment effect estimate group. In this case, the independence of treat- when some portion of the treatment group ment group assignment and potential out- is not treated. It depicts a pool of subjects comes ensures that for a large sample, the in which there are three types of people (a proportions of each type of person are the person’s type is not directly observable to same for the treatment and the control group, the experimenter), and in which each type as are the Yi(X) levels.) has different values of Yi(0) and Yi(1), where Panel A shows the case where two of the Yi(X) is the potential outcome for a subject three types of people are actually treated when assigned to the treatment group and 14 Figure 9.1 and subsequent discussion incorporates several important assumptions. Writing the poten- one type is not successfully treated when tial outcomes as a function of the individual’s own assigned to the treatment group (in this exam- treatment assignment and compliance rather than the ple, type 1 and type 2 are called “compliers” treatment assignment and compliance of all subjects 3 employs the stable unit treatment value assumption. and type people are called “noncompliers”). Depicting the potential outcomes as independent of The height of each of the three columns rep- treatment group assignment given the actual treat- resents the average outcome for each group, ment or not of the subjects employs the exclusion restriction. See Angrist et al. (1996). and their widths represent the proportion of 128 Alan S. Gerber the subject population of that type. Consider on the research objectives and whether treat- a simple comparison of the average outcome ment effects vary across individuals. Some- when subjects are assigned to the treatment times the treatment effect among those who group versus the control group (a.k.a. the are treated is what the researcher is inter- intent-to-treat [ITT] effect). The geometric ested in, in which case failure to treat some analogue to this estimate is to calculate the types of subjects is a feature of the experiment, difference in the total area of the shaded rect- not a deficiency. For example, if a campaign angles for both treatment and control assign- is interested in the returns from a particular ment. Visually, it is clear that the difference type of canvassing sweep through a neigh- between the total area in panel A and the borhood, the campaign wants to know the total area in panel B is an area created by response of the people whom the effort will the change in Y in panel A due to the applica- likely reach, not the hypothetical responses tion of the treatment to groups 1 and 2 (the of people who do not open the door to can- striped rectangles). Algebraically, the differ- vassers or who have moved away. ence between the treatment group average If treatment effects are homogeneous, and the control group average, the ITT, is then the CACE and the ATE are the same, equal to [Y1(1)–Y1(0)] p1 + [Y2(1)–Y2(0)] regardless of the contact rate. Demonstrat- p2. Dividing this quantity by the share of the ing that those who are treated in an experi- treatment group actually treated, (p1 + p2), ment have pre-treatment observables that dif- produces the ATT.15 This is also called the fer from the overall population mean is not complier average causal effect (CACE), high- sufficient to show that the CACE is differ- lighting the fact that the difference between ent from the ATE, because what matters is the average outcomes when the group is the treatment effect for compliers versus non- assigned to the treatment versus the control compliers (see Equation [1]), not the covari- condition is produced by the changing treat- ates or the level of Yi(0). Figure 9.1 could ment status and subsequent difference in out- be adjusted (by making the size of the gap comes for the subset of the subjects who are between Yi(0) and Yi(1) equal for all groups) compliers. so that all groups have different Yi(0) but the As Figure 9.1 suggests, one consequence same values of Yi(1)–Yi(0). Furthermore, of failure to treat all of those assigned to the higher contact rates may be helpful at reduc- treatment group is that the average treatment ing any gap between CACE and ATE. As Fig- effect is estimated for the treated, not the ure 9.1 illustrates, if the type 3 (untreated) entire subject population. The average treat- share of the population approaches zero (the ment effect (ATE) for the entire subject pool column narrows), then the treatment effect equals [Y1(1)–Y1(0)] p1 + [Y2(1)–Y2(0)] p2 + for this type would have to be very different [Y3(1)–Y3(0)] p3. Because the final term in the from the other subjects in order to produce ATE expression is not observed, an implica- enough “area” for this to lead to a large differ- tion of noncompliance is that the researcher ence between the ATE and CACE. Although is only able to directly estimate treatment raising the share of the treatment group that is effects for the subset of the population that successfully treated typically reduces the dif- one is able to treat. The implications of mea- ference between ATE and CACE, in a patho- suring the ATT rather than the ATE depend logical case, if the marginal treated individual has a more atypical treatment effect than the 15 This estimand is also known as the complier aver- average of those “easily” treated, then the gap age causal effect (CACE) because it is the treatment between CACE and ATE may grow as the effect for the subset of the population who are “com- proportion treated increases. The ATE and pliers.” Compliers are subjects who are treated when assigned to the treatment group and remain untreated CACE gap can be investigated empirically when assigned to the control group. When there is by observing treatment effects under light two-sided noncompliance, some subjects are treated and intensive efforts to treat. This approach whether assigned to the treatment or control group, and consequently the ATT and CACE are not the parallels the strategy of investigating the same. effects of survey nonresponse by using extra Field Experiments in Political Science 129 effort to interview and determining whether treatment status (e.g., the contact rate is very there are differences in the lower and higher low in a mobilization experiment) and the response rate samples (Pew Research Center sample size is small, group assignment is a 1998). weak instrument and estimates may be mean- Although the issue of partial treatment of ingfully biased. The danger of substantial bias the target population is very conspicuous in from weak instruments is diagnosed when many field experiments, it is also a common assignment to the treatment group does not problem in laboratory experiments. Designs produce a strong statistically significant effect such as typical laboratory experiments that on treatment status. See Angrist and Pis- put off randomization until compliance is chke (2009) for an extended discussion of this assured will achieve a 100-percent treatment issue. rate, but this does not “solve” the problem of measuring the treatment effect for a Do Field Experiments Assume population (ATE) versus those who are treat- Homogeneous Treatment Effects? able (CACE). The estimand for a laboratory experiment is the ATE for the particular The answer is “no.” See Figure 9.1, which group of people who show up for the exper- depicts a population in which the compliers iment. Unless this is also the ATE for the are divided into two subpopulations with dif- broader target population, failure to treat has ferent treatment effects. The ITT and the entered at the subject recruitment stage. CACE both estimate the average treatment One final note is that nothing in this effects, which may vary across individuals. answer should be taken as asserting that a low contact rate does not matter. The dis- Are Field Experiments Ethical? cussion has focused on estimands, but there are several important difficulties in estimat- All activities, including research, raise ethical ing the CACE when there are low levels questions. For example, it is surprising to read of compliance. First, noncompliance affects that certain physics experiments currently the precision of the experimental estimates. being conducted are understood by theoreti- Intuitively, when there is nearly 100 percent cians to have a measurable (although very failure to treat, it would be odd if mean- small) probability of condensing the planet ingful experimental estimates of the CACE Earth into a sphere 100 meters in diameter could be produced because the amount of (Posner 2004). I am not aware of any field noise produced by random differences in Y experiments in political science that pose a due to sampling variability in the treatment remotely similar level of threat. A full treat- and control groups would presumably swamp ment of the subject of research ethics is well any of the difference between the treatment beyond the scope of a brief response and not and control groups that was generated by my area of expertise, but I will make several the treatment effect. Indeed, a low contact points that I believe are sometimes neglected. rate will lead to larger standard errors and First, advocates of randomized trials in may leave the experimenter unable to pro- medicine turn the standard ethical ques- duce useful estimates of the treatment effect tions around and argue that those who treat for the compliers. Further, estimating the patients in the absence of well-controlled CACE by dividing the observed difference in studies should reflect on the ethics of using treatment and control group outcomes by the unproven methods and not performing the share of the treatment group that is treated experiments necessary to determine whether can be represented as a two-stage estimator. the interventions they employ actually work. The first stage regression estimates observed They argue that many established prac- treatment status (treated or not treated) as a tices and policies are often merely society- function of whether the subject is assigned to wide experiments (and, as such, poorly the treatment group. When treatment group designed experiments that lack a control assignment produces only a small change in group but somehow sidestep ethical scrutiny 130 Alan S. Gerber and bureaucratic review). They recount the Does the Fact That Field Experiments Do tragedies that followed when practices were Not Control for Background Activity Cause adopted without the support of experimental Bias? evidence (Chalmers 2003). Taking this a step Background activity affects the interpretation further, recent work has begun to quantify of the experimental results but does not cause the lives lost due to delays imposed by institu- bias. Background conditions affect Y (0) and tional review boards (Whitney and Schneider i Y (1), but subjects can be assigned and unbi- 2010). i ased treatment effects can be estimated in the Second, questions are occasionally raised usual fashion. That is not to say that back- as to whether an experimental intervention ground conditions do not matter, because might change a social outcome, such as they may affect Y(0) and Y(1) and there- affecting an election outcome by increasing fore the treatment effect Y(1)–Y(0). If the turnout. Setting aside the issue of whether treatment effect varies with background con- changing an election outcome through ditions, then background factors affect the increased participation or a more informed generalizability of the results; the treatment electorate (the most common mechanism for effect that is estimated should be thought of this hypothetical event, given current politi- as conditional on the background conditions. cal science field experiments) is problematic or praiseworthy, in the highly unlikely event that an experiment did alter an election result, Are Field Experiments Too Expensive to Be this would only occur for the small subset Used in My Research? of elections where the outcome would have Field experiments tend to be expensive, but been tied or nearly tied in the absence of there are ways to reduce the cost, sometimes the experiment. In this case, there are count- dramatically. Many recent field experiments less other mundane and essentially arbitrary were performed in cooperation with orga- contributions to the outcome with electoral nizations that are interested in evaluating a consequences that are orders of magnitude program or communications effort. Fortu- larger than the typical experimental interven- nately, a growing proportion of foundations tion. A partial list includes ballot order (Miller are requiring (and paying for) rigorous eval- and Krosnick 1998); place of voting (Berger, uation of the programs they support, which Meredith, and Wheeler 2008); number of should provide a steady flow of projects look- polling places (Brady and McNulty 2004); ing for partners to assist in experimental eval- use of optical scan versus punch card ballots uations. (Ansolabehere and Stewart 2005); droughts, floods, or recent shark attacks (Achen and Bartels 2004); rain on Election Day (Knack What about Treatment “Spillover” Effects? 1994); and a win by the local football team Spillover effects occur when those who are on the weekend prior to the election (Healy, treated in turn alter their behavior in a way Malhotra, and Mo 2009). That numerous 16 that affects other subjects. Spillover is a trivial or even ridiculous factors might swing potentially serious issue in field experiments. an election seems at first galling, but note It is also fair to note that spillover is typi- that these factors only matter when the elec- cally not a problem in the controlled envi- torate is very evenly divided. In this special ronment of laboratory experiments because case, however, regardless of the election out- contact among subjects can be observed and come, an approximately equal number of cit- regulated. In most applications, the presence izens will be pleased and disappointed with of spillover effects attenuate estimated treat- the result. As long as there is no regular bias ment effects by causing the control group in which side gets the benefit of chance, there to be partially treated. If the researcher is may be little reason for concern. Perhaps this is why we do not bankrupt the treasury to 16 See Sinclair’s chapter in this volume for further dis- make sure our elections are entirely error free. cussion. Field Experiments in Political Science 131 concerned about mitigating the danger from ciate the enormous importance of obtaining spillover effects, then reducing the density convincing causal estimates. Throughout the of treatment is one option; this will likely history of science, new measurement tech- reduce the share of the control group affected nologies (e.g., the microscope, spectogra- by spillover. Another perspective is to con- phy) and reliable causal estimates (controlled sider spillover effects as worth measuring in experiments) have been the crucial impetus to their own right; some experiments have been productive theorizing. There are also practi- designed to measure spillover (Nickerson cal costs to ignoring solid empirical demon- 2008). It is sometimes forgotten that spillover strations because of concerns about theoreti- is also an issue in observational research. In cal mechanisms. Taking an example from the survey-based observational studies of party history of medicine, consider the prescient contact and candidate choice, for example, findings of Semmelweis, who conducted a only those who report direct party contact pioneering experiment in the 1860s demon- are coded as contacted. If those who are con- strating that washing hands in a disinfectant tacted in turn mobilize those who are not significantly reduced death from postpartum contacted, then this will introduce a down- infection. Critics, however, claimed that his ward bias into the observational estimate of theory of how the intervention worked was the causal effect of party contact, which is flawed and incomplete (Loudon 2000). This based on comparison of those coded treated justified critique of Semmelweis’ theoreti- and those coded untreated. cal arguments was taken as a license by the medical community to ignore his accurate empirical conclusions, resulting in count- 6. Further Issues less unnecessary deaths over the next several decades. In this section, I sound some notes of caution Without gainsaying the value of measure- regarding the development of field experiment, what is lost if there is no clearly articu- ments in political science. I first discuss the lated theoretical context? There are implica- issue of how to interpret the results from field tions for both external and internal validity. experiments. Field experiments to date have First, consider external validity. There is no often focused on producing accurate mea- theoretical basis for the external validity of surement rather than illuminating broader field experiments comparable to the statisti- theoretical issues. However, in the absence cal basis for claims of internal validity, and it of some theoretical context, it may be dif- is often very plausible that treatment effects ficult to judge what exactly is being mea- might vary across contexts. The degree of sured. Second, I discuss some problems with uncertainty assigned when applying a treat- the development of literatures based on field ment effect produced in one context (place, experiments. These issues relate to the diffi- people, time, treatment details) to another culty of replication and the potential sources context is typically based on reasonable con- of bias in experimental reporting, especially jecture. Appeals to reasonableness are, in the when projects are undertaken with nonaca- absence of evidence or clear theoretical guid- demic partners. ance, disturbingly similar to the justifications for the assumptions on which observational UnbiasedEstimates...ofWhat? approaches often rest. Field experimentation is a measurement tech- To be concrete, consider the case of can- nique. Many researchers who use field exper- vassing to mobilize voters where the treat- iments are content to report treatment effects ment effect is the effect of the intervention on from an intervention and leave it at that, an the subject’s turnout. In the most rudimen- empiricism that has led some observers to tary framework, the size of this effect might dismiss field experiments as mere program depend on how the intervention affects his evaluations without broader theoretical con- or her beliefs about the costs and benefits tribution. This line of criticism fails to appre- of voting in the upcoming election. Beliefs 132 Alan S. Gerber about the costs and benefits of participation ilar vein, interventions may fail on repeti- may depend in turn on, among other things, tion if they work in part due to their nov- how the intervention affects subject knowl- elty. Alternatively, if the effect of canvass- edge about or the salience of the upcoming ing works through social reciprocity, where election, expectations about the closeness of the canvasser exerts effort and the subject the election, beliefs about the importance of exchanges a pledge of reciprocal effort to the election, and the perceived social desir- vote, then the voter’s experience at the polls ability of voting. The intervention’s effect on may not alter the effectiveness of the inter- these variables might depend on the political vention. That is, the estimate of canvassing context, such as the political history or polit- effects in today’s election might apply well ical norms of the place in which the experi- to subsequent interventions. What matters is ment occurs. Additional factors affecting the the voter’s perception that the canvasser has size of the treatment effect might include exerted effort – perhaps canvassing in a snow- which subset of the population is success- storm would be especially effective. This dis- fully treated and how near the subjects are to cussion of the effect of canvassing suggests the the threshold of participation. The treatment value of delineating and adjudicating among effect estimated by a given experiment might the various possible mechanisms. More gen- conceivably be a function of variables related erally, being more explicit about the theo- to any and all of these considerations.17 retical basis for the observed result might Understanding the mechanism by which inspire some caution and provide guidance the treatment is working may be critical for when generalizing findings. accurate predictions about how the treatment Reflecting on the theoretical context can will perform outside the initial experimental also assist in establishing the internal validity context. Consider the challenge of extrapo- of the experiment. For example, it might be lating the effectiveness of face-to-face can- useful to reflect on how the strategic incen- vassing. Alternative theories have very differ- tives of political actors can alter treatment ent implications. One way this intervention effect estimates. Continuing with the exam- may increase participation is if contact by a ple of a canvassing experiment, suppose some canvasser increases the subject’s perception local organization is active in a place where of the importance of the election (changing a canvassing experiment is (independently) the subject’s beliefs about the benefits of par- being conducted. Consider how the canvass- ticipation). However, a subject’s beliefs may ing intervention might affect the behavior of be less affected by canvassing in a place where such an independent group that expends a canvassing is routine than in a place where it fixed amount of effort making calls to peo- occurs only under the most extreme political ple and asking them if they intend to vote. conditions. Turning to the long-term effec- Suppose that the group operates according to tiveness of canvassing, if canvassing works the rule: if the voter says he or she will vote, by causing subjects to update their percep- then there is no further attempt to encourage tions of the importance of voting, the link him or her, whereas if the voter says he or she between canvassing and turnout effects may will not vote, then the group expends sub- not be stable. If, following an intensive can- stantial time and effort to encourage the sub- vassing effort, the election turns out to be ject to vote. If the canvassing treatment took a landslide or the ballot has no important place prior to the independent group’s efforts contests, a voter might ignore subsequent and it was effective, this will result in a share canvassing appeals as uninformative. In a sim- of their limited mobilization resources being diverted from the treatment group to the con- 17 The most striking difference across contexts demon- trol group (who are less likely to say they strated to date is that mobilization effects appear plan to vote), depressing the estimated treat- strongest for those voters predicted to be about fifty ment effect. Less subtly, if an experimental percent likely to vote, a result that follows theoretically from a latent variable model (Arceneaux and canvassing effort is observed, then this might Nickerson 2009). alter the behavior of other campaigns. More Field Experiments in Political Science 133 common violations of the requirement that should move as a result of an experimental treatment group assignment of one subject report. Though it will always be a challenge not affect potential outcomes of other sub- to know how to combine experimental find- jects may occur if a treated subject communi- ings given the diversity of treatments, con- cates directly with other subjects. The impor- texts, subjects, and so on, under ideal cir- tance of these effects may vary with context cumstances, updating is a mundane matter and treatment. For example, if the treatment of adjusting priors using the new reported is highly novel, then it is much more likely effect sizes and standard errors. However, the that subjects will remark on the treatment to uncharted path from execution of the experi- housemates or friends. ment to publication adds an additional source Finally, careful consideration of the com- of uncertainty because both the direction and plete set of behavioral changes that might fol- magnitude of any bias incorporated through 18 low an intervention may also suggest new out- this process are unknown. Although any come measures. Theorizing about how the given experiment is unbiased, the experimen- intervention alters the incentives and capa- tal literature may nevertheless be biased if bilities of subjects may affect which outcome the literature is not a representative sam- measures are monitored. It is common to ple of studies. This issue is especially vex- measure the effect of a voter mobilization ing in the case of proprietary research. A intervention on voter turnout. However, it significant amount of experimental research is unclear how and whether political partic- on campaign effects is now being conducted ipation in elections is related to other forms by private organizations such as campaigns, of political involvement. If citizens believe unions, or interest groups. This type of work that they have fulfilled their civic responsi- has the potential to be of immense benefit bility by voting, then voting may be a sub- because the results are of theoretical and prac- stitute for attending a Parent-Teacher Asso- tical interest, the studies numerous and con- ciation meeting or contributing to the Red ducted in varying contexts, and the cost of the Cross. Alternately, the anticipation of voting research is borne by the sponsoring organiza- may lead to enhanced confidence in politi- tion. This benefit might not be realized, how- cal competence and a stronger civic identity, ever, if only a biased subset of experiments are which may then inspire other forms of polit- deemed fit for public release. ical and community involvement or informa- It is often suggested that the scholarly pub- tion acquisition. lication process may be biased in favor of publicizing arresting results. However, anomalous reports have only a limited effect when Publication Process: Publication Bias, there is a substantial body of theory and fre- Proprietary Research, Replication quent replication. The case of the “discov- One of the virtues of observational research ery” of cold fusion illustrates how theory and is that it is based on public data. The ANES replication work to correct error. When it data are well known and relatively transpar- was announced that nuclear fusion could be ent. People would notice if, for some rea- achieved at relatively low temperatures using son, the ANES data were not released. In equipment not far beyond that found in a contrast, experimental data are produced by well-equipped high school lab, some believed the effort of unsupervised scholars, who then that this technology would be the solution must decide whether to write up results and to the world’s energy problems. Physicists present the findings to the scholarly commu- were quite skeptical about this claim from the nity. These are very different situations, and it outset because theoretical models suggested is unclear what factors affect which results are it was not very plausible. Well-established shared when sharing depends on the choices models of how atoms interact under pressure of researchers and journal editors. The process by which experimental results are dis- 18 For a related point, where the target is observational closed or not affects how much one’s priors research, see Gerber et al. (2004). 134 Alan S. Gerber imply that the distance between the atoms 7. Conclusion in the cold fusion experiments would be bil- lions of times greater than what is necessary After a generation in which nonexperimental to cause the fusion effects claimed. survey research dominated the study of politi- Physics theory also made another contri- cal behavior, we may now be entering the age bution to the study of cold fusion. The famous of experimentation. There was not a single cold fusion experiment result was that the field experiment published in a major political experimental cell produced heat in excess of science journal in the 1990s, whereas scholars the heat input to the cell. Extra heat was, in have published dozens of such papers in the fact, the bottom line measurement of greatest past decade. The widespread adoption of field practical importance because it suggested that experimentation is a striking development. cold fusion could be an energy source. The- The results accumulated to date have had a oretical work on fusion, however, pointed to substantial effect on what we know about pol- a number of other outputs that could be used itics and have altered both the methods used to determine if fusion was really occurring. to study key topics and the questions that These included gamma rays, neutrons, and are being asked. Furthermore, field experi- tritium. These fusion by-products are easier mentation in political science has moved well to measure than is excess heat, which requires beyond the initial studies of voter mobiliza- a careful accounting of all heat inputs and out- tion to consider the effects of campaign com- puts to accurately calculate the net change. munications on candidate choice; the political It was the absence of these by-product mea- effects of television, newspapers, and radio; surements (in both the original and replica- the effects of deliberation; tests of social psy- tion studies), in addition to the theoretical chology theories; the effects of political insti- implausibility of the claim, that led many tutions; and measurement of social diffusion. physicists and chemists to doubt the exper- The full impact of the increased use of field imental success from the outset. Replication experiments is difficult to know. It is possible studies began within twenty-four hours of the that some small part of the recent heightened announcement. In a relatively short period of attention to causal identification in observa- time, the cold fusion claim was demolished.19 tional research in political science may have Compare this experience to how a similar been encouraged by the implicit contrast drama would unfold in political science field between the opaque and often implausible experimentation. The correctives of strong identification assumptions used in observa- theory and frequent replication are not avail- tional research and the more straightforward able in the case of field experimental findings. identification enjoyed by randomized inter- Unfortunately, field experiments tend to be ventions. expensive and time consuming. There would Although the past decade has seen many likely be no theory with precise predictions exciting findings and innovations, there are to cast doubt on the experimental result or important areas for growth and improve- provide easy-to-measure by-products of the ment. Perhaps most critically, field exper- experimental intervention to lend credence imentation has not provoked the healthy to the experimental claims. The lack of the- back and forth between theory and empiri- oretically induced priors is a problem for all cal findings that is typical in the natural sci- research, but it is especially significant when ences. Ideally, experiments would generate replication is not easy. How long would it robust empirical findings, and then theorists take political science to refute “cold fusion” would attempt to apply or produce (initially) results? If the answer is “until a series of new simple models that predict the experimental field experiments refute the initial finding,” results and, critically, make new experimen- then it might take many years. tally testable predictions. For the most part, though not in all cases, the field experiment 19 Unfortunately, I am not well trained in physics or literature in political science has advanced chemistry. This account draws on Gary Taubes’ Bad Science: The Short Life and Weird Times of Cold Fusion by producing measurements in new domains (1993). of inquiry rather than addressing theoretical Field Experiments in Political Science 135 puzzles raised by initial results. This shows Quality.” Typescript, Massachusetts Institute something about the tastes of the current of Technology. crop of field experimenters, however, the rel- Ansolabehere, Stephen D., and Charles Stew- 2005 atively slight role of theory in the evolution art III. . “Residual Votes Attributable to 67 365 89 of the political science field experiment liter- Technology.” Journal of Politics : – . ature so far may also reflect a lack of engage- Arceneaux, Kevin, Alan S. Gerber, and Donald P. Green. 2006. “Comparing Experimental and ment by our more theoretically inclined Matching Methods Using a Large-Scale Voter colleagues. Mobilization Experiment.” Political Analysis 14: 1–36. Arceneaux, Kevin, and David Nickerson. 2009. References “Who Is Mobilized to Vote? A Re-Analysis of Eleven Randomized Field Experiments.” Abramowitz, Alan I. 1988. “Explaining Senate American Journal of Political Science 53: 1–16. Election Outcomes.” American Political Science Bergan, Daniel E. 2009. “Does Grassroots Lobby- Review 82: 385–403. ing Work?: A Field Experiment Measuring the Achen, Christopher H., and Larry M. Bar- Effects of an E-Mail Lobbying Campaign on tels. 2004. “Blind Retrospection: Electoral Re- Legislative Behavior.” American Politics Research sponses to Droughts, Flu, and Shark Attacks.” 37: 327–52. Estudio/Working Paper 2004/199. Berger, Jonah, Marc Meredith, and S. Christian Adams, William C., and Dennis J. Smith. 1980. Wheeler. 2008. “Contextual Priming: Where “Effects of Telephone Canvassing on Turnout People Vote Affects How They Vote.” Pro- and Preferences: A Field Experiment.” Public ceedings of the National Academy of Sciences 105: Opinion Quarterly 44: 389–95. 8846–49. Addonizio, Elizabeth, Donald Green, and Brady, Henry E., and John E. McNulty. 2004. James M. Glaser. 2007. “Putting the Party “The Costs of Voting: Evidence from a Natu- Back into Politics: An Experiment Testing ral Experiment.” Paper presented at the annual Whether Election Day Festivals Increase Voter meeting of the Society for Political Methodol- Turnout.” PS: Political Science & Politics 40: 721– ogy, Palo Alto, CA. 27. Butler, Daniel M., and David W. Nickerson. 2009. Angrist, Joshua D. 1990. “Lifetime Earnings and “Are Legislators Responsive to Public Opin- the Vietnam Era Draft Lottery: Evidence from ion? Results from a Field Experiment.” Type- Social Security Administrative Records.” Amer- script, Yale University. ican Economic Review 80: 313–36. Chalmers, Iain. 2003. “Trying To Do More Good Angrist, Joshua D., Guido Imbens, and Donald B. Than Harm in Policy and Practice: The Role Rubin. 1996. “Identification of Causal Effects of Rigorous, Transparent, Up-to-Date Evalua- Using Instrumental Variables.” Journal of the tions.” Annals of the American Academy of Political American Statistical Association 91: 444–72. and Social Science 589: 22–40. Angrist, Joshua D., and Alan B. Krueger. 1991. Chin, Michelle L., Jon R. Bond, and Nehemia “Does Compulsory School Attendance Affect Geva. 2000. “A Foot in the Door: An Exper- Schooling and Earnings?” Quarterly Journal of imental Study of PAC and Constituency Economics 106: 979–1014. Effects on Access.” Journal of Politics 62: 534– Angrist, Joshua D., and Jorn-Steffen Pischke. 49. 2009. Mostly Harmless Econometrics: An Empiri- Dale, Allison, and Aaron Strauss. 2007. “Mobi- cist’s Companion. Princeton, NJ: Princeton Uni- lizing the Mobiles: How Text Messaging Can versity Press. Boost Youth Voter Turnout.” Working paper, Ansolabehere, Stephen D., and Alan S. Gerber. University of Michigan. 1994. “The Mismeasure of Campaign Spend- Davenport, Tiffany C., Alan S. Gerber, and ing: Evidence from the 1990 U.S. House Elec- Donald P. Green. 2010. “Field Experiments tions.” Journal of Politics 56: 1106–18. and the Study of Political Behavior.” In The Ansolabehere, Stephen D., and Shanto Iyengar. Oxford Handbook of American Elections and Polit- 1996. Going Negative: How Political Advertising ical Behavior, ed. Jan E. Leighley. New York: Divides and Shrinks the American Electorate.New Oxford University Press, 69–88. York: Free Press. Deaton, Angus S. 2009. “Instruments of Devel- Ansolabehere, Stephen D., and James M. Sny- opment: Randomization in the Tropics, and der. 1996. “Money, Elections, and Candidate the Search for the Elusive Keys to Economic 136 Alan S. Gerber

Development.” NBER Working Paper No. Gerber, Alan S., Donald P. Green, and Edward 14690. H. Kaplan. 2004. “The Illusion of Learning Eldersveld, Samuel J. 1956. “Experimental Pro- from Observational Research.” In Problems and paganda Techniques and Voting Behavior.” Methods in the Study of Politics, eds. Ian Shapiro, American Political Science Review 50: 154–65. Rogers Smith, and Tarek Massoud. New York: Eldersveld, Samuel J., and Richard W. Dodge. Cambridge University Press, 251–73. 1954. “Personal Contact or Mail Propaganda? Gerber, Alan S., Donald P. Green, and Christo- An Experiment in Voting Turnout and Attitude pher W. Larimer. 2008. “Social Pressure and Change.” In Public Opinion and Propaganda,ed. Voter Turnout: Evidence from a Large-Scale Daniel Katz. New York: Holt, Rinehart and Field Experiment.” American Political Science Winston, 532–42. Review 102: 33–48. Erikson, Robert S., and Thomas R. Palfrey. 2000. Gerber, Alan S., Donald P. Green, and David W. “Equilibria in Campaign Spending Games: Nickerson. 2001. “Testing for Publication Bias Theory and Data.” American Political Science in Political Science.” Political Analysis 9: 385– Review 94: 595–609. 92. Gerber, Alan S. 1998. “Estimating the Effect of Gerber, Alan S., Gregory A. Huber, and Ebonya Campaign Spending on Senate Election Out- Washington. in press. “Party Afﬁliation, Parti- comes Using Instrumental Variables.” Ameri- sanship, and Political Beliefs: A Field Experi- can Political Science Review 92: 401–11. ment.” American Political Science Review. Gerber, Alan S. 2004. “Does Campaign Spending Gerber, Alan S., Dean Karlan, and Daniel Bergan. Work?: Field Experiments Provide Evidence 2009. “Does the Media Matter? A Field Exper- and Suggest New Theory.” American Behavioral iment Measuring the Effect of Newspapers Scientist 47: 541–74. on Voting Behavior and Political Opinions.” Gerber, Alan S. in press. “New Directions in the American Economic Journal: Applied Economics 1: Study of Voter Mobilization: Combining Psy- 35–52. chology and Field Experimentation.” Gerber, Alan S., and Kyohei Yamada. 2008. Gerber, Alan S., and David Doherty. 2009.“Can “Field Experiment, Politics, and Culture: Test- Campaign Effects Be Accurately Measured ing Social Psychological Theories Regarding Using Surveys?: Evidence from a Field Exper- Social Norms Using a Field Experiment in iment.” Typescript, Yale University. Japan.” Working paper, ISPS Yale University. Gerber, Alan S., David Doherty, and Conor M. Gosnell, Harold F. 1927. Getting-Out-the-Vote: An Dowling. 2009. “Developing a Checklist for Experiment in the Stimulation of Voting. Chicago: Reporting the Design and Results of Social Sci- The University of Chicago Press. ence Experiments.” Typescript, Yale Univer- Green, Donald P., and Alan S. Gerber. 2004. Get sity. Out the Vote: How to Increase Voter Turnout. Gerber, Alan S., James G. Gimpel, Donald P. Washington, DC: Brookings Institution Press. Green, and Daron R. Shaw. in press. “The Size Green, Donald P., and Alan S. Gerber. 2008. Get and Duration of Campaign Television Adver- Out the Vote: How to Increase Voter Turnout. tising Effects: Results from a Large-Scale Ran- 2nd ed. Washington, DC: Brookings Institu- domized Experiment.” American Political Science tion Press. Review. Green, Donald P., and Jonathan S. Krasno. Gerber, Alan S., and Donald P. Green. 2000. 1988. “Salvation for the Spendthrift Incum- “The Effects of Canvassing, Direct Mail, and bent: Reestimating the Effects of Campaign Telephone Contact on Voter Turnout: A Field Spending in House Elections.” American Jour- Experiment.” American Political Science Review nal of Political Science 32: 884–907. 94: 653–63. Guan, Mei, and Donald P. Green. 2006.“Non- Gerber, Alan S., and Donald P. Green. 2008. Coercive Mobilization in State-Controlled “Field Experiments and Natural Experiments.” Elections: An Experimental Study in Bei- In Oxford Handbook of Political Methodology, eds. jing.” Comparative Political Studies 39: 1175– Janet M. Box-Steffensmeier, Henry E. Brady, 93. and David Collier. New York: Oxford Univer- Habyarimana, James, Macartan Humphreys, Dan sity Press, 357–81. Posner, and Jeremy Weinstein. 2007. “Why Gerber, Alan S., and Donald P. Green. 2011. Field Does Ethnic Diversity Undermine Public Experiments: Design, Analysis, and Interpretation. Goods Provision? An Experimental Approach.” Unpublished Manuscript, Yale University. American Political Science Review 101: 709–25. Field Experiments in Political Science 137

Harrison, Glenn W., and John A. List. 2004. Spending on Election Outcomes in the U.S. “Field Experiments.” Journal of Economic Lit- House.” Journal of Political Economy 102: 777– erature 42: 1009–55. 98. Healy, Andrew J., Neil Malhotra, and Cecilia Loudon, Irvine. 2000. The Tragedy of Childbed Hyunjung Mo. 2009. “Do Irrelevant Events Fever. Oxford: Oxford University Press. Affect Voters’ Decisions? Implications for Ret- Medical Research Council, Streptomycin in rospective Voting.” Stanford Graduate School Tuberculosis Trials Committee. 1948.“Strep- of Business Working Paper No. 2034. tomycin Treatment for Pulmonary Tuberculo- Humphreys, Macartan, and Jeremy M. Weinstein. sis.” British Medical Journal 2: 769–82. 2007. “Policing Politicians: Citizen Empower- Michelson, Melissa R. 2003. “Getting Out the ment and Political Accountability in Africa.” Latino Vote: How Door-to-Door Canvass- Paper presented at the annual meeting of ing Inﬂuences Voter Turnout in Rural Cen- the American Political Science Association, tral California.” Political Behavior 25: 247– Chicago. 63. Humphreys, Macartan, and Jeremy Weinstein. Michelson, Melissa R., Lisa Garcıa´ Bedolla, and 2009. “Field Experiments and the Political Margaret A. McConnell. 2009.“Heedingthe Economy of Development.” Annual Review of Call: The Effect of Targeted Two-Round Political Science 12: 367–78. Phonebanks on Voter Turnout.” Journal of Pol- Hyde, Susan D. 2010. “Experimenting in Democ- itics 71: 1549–63. racy Promotion: International Observers and Miller, Joanne M., and Jon A. Krosnick. 1998. the 2004 Presidential Elections in Indonesia.” “The Impact of Candidate Name Order on Perspectives on Politics 8: 511–27. Election Outcomes.” Public Opinion Quarterly Imbens, Guido W. 2009. “Better Late Than Noth- 62: 291–330. ing: Some Comments on Deaton (2009)and Miller, Roy E., David A. Bositis, and Denise L. Heckman and Urzua (2009).” National Bureau Baer. 1981. “Stimulating Voter Turnout in of Economic Research Working Paper No. a Primary: Field Experiment with a Precinct 14896. Committeeman.” International Political Science Jacobson, Gary C. 1978. “The Effects of Cam- Review 2: 445–60. paign Spending in Congressional Elections.” Nickerson, David W. 2008. “Is Voting Conta- American Political Science Review 72: 469– gious? Evidence from Two Field Experiments.” 91. American Political Science Review 102: 49– Jacobson, Gary C. 1985. “Money and Votes 57. Reconsidered: Congressional Elections, 1972– Olken, Benjamin. 2010. “Direct Democracy and 1982.” Public Choice 47: 7–62. Local Public Goods: Evidence from a Field Jacobson, Gary C. 1990. “The Effects of Cam- Experiment in Indonesia.” American Political paign Spending in House Elections: New Evi- Science Review 104: 243–67. dence for Old Arguments.” American Journal of Paluck, Elizabeth Levy, and Donald P. Green. Political Science 34: 334–62. 2009. “Deference, Dissent, and Dispute Res- Jacobson, Gary C. 1998. The Politics of Congressional olution: An Experimental Intervention Using Elections. New York: Longman. Mass Media to Change Norms and Behavior in John, Peter, and Tessa Brannan. 2008.“How Rwanda.” American Political Science Review 103: Different Are Telephoning and Canvassing? 622–44. Results from a ‘Get Out the Vote’ Field Panagopoulos, Costas, and Donald P. Green. Experiment in the British 2005 General Elec- 2008. “Field Experiments Testing the Impact tion.” British Journal of Political Science 38: 565– of Radio Advertisements on Electoral Compe- 74. tition.” American Journal of Political Science 52: Knack, Steve. 1994. “Does Rain Help the Repub- 156–68. licans? Theory and Evidence on Turnout and Pew Research Center for the People and the the Vote.” Public Choice 79: 187–209. Press. 1998. “Possible Consequences of Non- LaLonde, Robert J. 1986. “Evaluating the Econo- Response for Pre-Election Surveys: Race and metric Evaluations of Training Programs with Reluctant Respondents.” Survey Report, May Experimental Data.” American Economic Review 16. Retrieved from http://people-press.org/ 76: 604–20. report/89/possible-consequences-of-non- Levitt, Steven D. 1994. “Using Repeat Chal- response-for-pre-election-surveys (November lengers to Estimate the Effect of Campaign 1, 2010). 138 Alan S. Gerber

Posner, Richard A. 2004. Catastrophe: Risk and Reports.” Quarterly Journal of Political Science 2: Response. Oxford: Oxford University Press. 287–305. Rosenstone, Steven J., and John Mark Hansen. Verba, Sidney, Kay Lehman Schlozman, and 1993. Mobilization, Participation, and Democracy Henry E. Brady. 1995. Voice and Equality: Civic in America. New York: MacMillan. Voluntarism in American Politics. Cambridge, Rubin, Donald B. 1978. “Bayesian Inference for MA: Harvard University Press. Causal Effects: The Role of Randomization.” Wantchekon, Leonard. 2003. “Clientelism and The Annals of Statistics 6: 34–58. Voting Behavior: Evidence from a Field Exper- Rubin, Donald B. 1990. “Formal Modes of Sta- iment in Benin.” World Politics 55: 399– tistical Inference for Causal Effects.” Journal of 422. Statistical Planning and Inference 25: 279–92. Whitney, Simon N., and Carl E. Schneider. 2010. Taubes, Gary. 1993. Bad Science: The Short Life and “A Method to Estimate the Cost in Lives of Weird Times of Cold Fusion. New York: Random Ethics Board Review of Biomedical Research.” House. Paper presented at the “Is Medical Ethics Vavreck, Lynn. 2007. “The Exaggerated Effects of Really in the Best Interest of the Patient?” Con- Advertising on Turnout: The Dangers of Self- ference, Uppsala, Sweden. Part III

DECISION MAKING

CHAPTER 10

Attitude Change Experiments in Political Science

Allyson L. Holbrook

The importance of attitudes and the processes tudes play a central role in many of the demo- by which they are formed and changed is cratic processes studied by political scientists ubiquitous throughout political science. Per- reviewed in this volume (e.g., Gadarian and haps the most obvious example is research Lau; Hutchings; Lodge and Tabor; McGraw; exploring citizens’ attitudes toward candi- Nelson; Wilson and Eckel). dates, how these attitudes are influenced by political advertising and other persua- Defining Attitudes sive messages, and how these attitudes influence decisions and behavior (see McGraw’s Different definitions of attitudes have been chapter in this volume). Attitudes toward proposed by psychologists (e.g., Thurstone candidates are fundamental to the demo- 1931; Allport 1935;Bem1970). Perhaps the cratic process because they help voters make most widely accepted modern definition was vote choices, perhaps the most basic way proposed by Eagly and Chaiken (1993), who in which citizens can express their opin- defined an attitude as “a psychological ten- ions and influence government (e.g., Rosen- dency that is expressed by evaluating a par- stone and Hansen 1993). Other key attitudes ticular entity with some degree of favor or in the political domain include attitudes disfavor” (1). One key feature of an attitude toward specific policies that also help vot- is that it is directed toward a specific attitude ers make important decisions about voting, object. This attitude object can be a person, vote choice, and activism (e.g., Rosenstone place, idea, thing, experience, behavior, etc. and Hansen 1993). Attitudes toward insti- A second feature of an attitude is that it is tutions such as political parties and govern- evaluative and reflects the extent of positivity ment entities also influence people’s view of or negativity a person has toward the atti- government. Finally, attitudes toward other tude object.1 Most attitude change research groups in society (e.g., African Americans, women) may help determine support for spe- 1 Although attitudes have most often been conceptu- cific policies (e.g., Transue 2007). Thus, atti- alized as a single bipolar continuum with positivity

141 142 Allyson L. Holbrook in political science assesses change in explicit attitudes toward Barack Obama, have them attitudes, but recent interest in psychology watch several of his recent speeches, and then has also focused on implicit attitudes (see measure attitudes again. Alternatively, one Lodge and Taber’s chapter in this volume). could compare the attitudes of people who Early definitions of attitudes also defined have been exposed to different information. If them as being stable over time (Cantril 1934) attitudes differed as a function of the informa- and influencing behavior and thought (All- tion to which people were exposed, then one port 1935). However, more recent defini- could infer that differential attitude change tions conceive of attitudes as not only having or persuasion occurred. a valence, but also strength. Strong attitudes are stable, resist change, and/or influence behavior and thought, whereas weak atti- 1. Measuring Attitudes tudes do not (for a review, see Krosnick and Petty 1995). Very weak attitudes that are Attitudes are an inherently subjective con- based on little information and that people struct, and there is no current measure of atti- may construct on the spot when asked to tudes that is without some error. Researchers report their attitudes are similar to what some in political science have typically used three have labeled as “nonattitudes” (e.g., Converse types of measures to assess attitudes in 1964, 1970). Recent evidence suggests, how- both observational and experimental studies. ever, that even the weakest attitudes may One of the most commonly used measures not truly be nonattitudes, but rather logi- involves asking people self-report questions cal constructions based on whatever infor- inquiring whether they like or dislike (favor or mation people have obtained (Krosnick et al. oppose) a political candidate, policy, or other 2002). attitude object. In most cases, these questions not only capture the direction of the attitude (e.g., like, dislike), but also some measure of Attitude Change extremity (e.g., like a lot, like somewhat, like The key dependent variable of interest in this a little). These measures of attitudes are per- chapter is attitude change. For the purposes haps the most direct, but they rely on the of this chapter, attitude changes include pro- assumption that respondents are willing and cesses of attitude formation (i.e., a change able to report their attitudes, which may not from having no attitude toward an atti- always be true. Many such attitude reports tude object to having an attitude toward the may be subject to social desirability response object), as well as changes in an existing bias, whereby respondents are motivated to attitude (i.e., an existing attitude becoming report attitudes that are more socially desir- more or less positive or negative). Gener- able and avoid reporting those that might ally, attitudes researchers in psychology have make others look at them less favorably (e.g., conceived of attitude formation as a special attitudes toward African Americans or toward case of attitude change, arguing that the pro- legalizing same-sex marriage; Warner 1965; cesses involved in the former are often simi- Himmelfarb and Lickteig 1982). In addition, lar to those involved in the latter (Eagly and direct attitude questions may be affected by Chaiken 1993). Attitude change can be both response biases such as extreme response style directly measured and indirectly inferred. and acquiescence response bias (e.g., Baum- For example, one could measure people’s gartner and Steenkamp 2001). A second set of measures asks respondents toward the object at one end, negativity in the other, about their preferences regarding, for exam- and a neutral midpoint, theories of attitudes have also begun to consider the role of attitudinal ambiva- ple, political candidates or policies. In this lence. Attitudinal ambivalence occurs when a person’s case, to assess policy preferences regarding beliefs or affect toward an attitude object are in con- immigration, respondents might be asked: flict with one another (e.g., a person has both positive and negative beliefs about a political candidate; Eagly “Under current law, immigrants who come and Chaiken 1993). from other countries to the United States Attitude Change Experiments in Political Science 143 legally are entitled, from the very beginning, that it is the government’s responsibility to to government assistance such as Medicaid, pass and enforce such regulations. food stamps, or welfare on the same basis as A third type of attitudinal measure uses citizens. But some people say they should not attitude-expressive behaviors (e.g., financially be eligible until they have lived here for a year supporting a particular candidate or orga- or more. Which do you think? Do you think nization) as indicators of attitudes. These that immigrants who are here legally should behaviors have been assessed through self- be eligible for such services as soon as they report behavioral intention questions, retro- come, or should they not be eligible?” (Davis, spective self-reports of past behaviors, and Smith, and Marsden 2007, 660). Similarly, a direct observation of behaviors. Self-report respondent might be asked to report which measures of behavioral intentions assume candidate he or she prefers in an upcoming that respondents can accurately predict their election. own behavior and are willing to do so, but These questions assess attitudes indirectly such reports may be inaccurate, both because through preferences or choices (in which a respondents may not be able to accurately study participant is asked to choose among predict their own behavior under some con- a list of options or rank order them). Pref- ditions (e.g., Wolosin, Sherman, and Cann erences have been defined as “a compara- 1975) and because reports of behaviors may tive evaluation of (i.e., ranking over) a set be influenced by social desirability response of objects” (Druckman and Lupia 2000, 2). bias (e.g., Warner 1965). Retrospective Although attitudes are distinct from both reports of past behaviors assume that respon- preferences and choices, they are one influ- dents’ memory for past behaviors is accurate ence on preferences and choices. Preferences and that respondents are honestly report- may allow researchers to assess attitudes in ing these behaviors, although this may not ways that may be less affected by social always be the case (e.g., Belli et al. 1999). desirability bias (e.g., Henry and Sears 2002; Finally, direct measures of behaviors can be although see Berinsky 2002) and by some cumbersome and may be very limited. For response effects than more direct attitude example, willingness to act to express one’s questions (e.g., acquiescence response bias). opinion could be measured by giving study However, particular policy preference ques- participants the opportunity to sign either a tions usually frame a policy decision in terms prochoice or prolife petition. However, this of two (or more) choices, forcing respon- measure is very specific and may be influenced dents to choose the response that most closely by factors other than participants’ attitudes. matches their preference. Thus, preference Measures of behaviors only indirectly measures may not be as precise or sensitive as assess attitudes and may be influenced by more direct measures of attitudes. A prefer- other factors as well. There is a long history ence for one candidate over another does not of research in psychology examining whether indicate whether a person is positive or nega- attitudes predict behavior and the conditions tive toward the preferred candidate. In addi- under which they do and do not (Ajzen and tion, policy and candidate preferences may Fishbein 2005; also, for a review, see Eagly not only be influenced by attitudes toward the and Chaiken 1993). There are many rea- various policies, but also potentially by strate- sons why behaviors and attitudes may not gic concerns and beliefs about the effective- be entirely consistent. For example, behaviors ness of policies and the role of government. and behavioral intentions may be influenced For example, although a respondent may feel by factors other than attitudes, such as social very positive about reducing global warm- norms and a person’s perceived control of the ing, he or she may not support a government behavior (Ajzen and Fishbein 2005) and the policy requiring businesses to reduce carbon resources required to engage in the behavior emissions, because he or she either does not (e.g., Miller and Krosnick 2004). believe the policy would be effective at reduc- Behaviors and behavioral intentions may ing future global warming or does not believe also be influenced by strategic concerns. For 144 Allyson L. Holbrook example, consider the case of the primary for behavior and asked to report how many they the 2008 presidential election. In this elec- have done. The proportion of respondents tion, John McCain acquired enough delegates who have done the sensitive behavior can to become the presumptive Republican nom- be calculated as the difference in the means inee in early March 2008, well before the between the two groups without any respon- Republican primaries were complete. How- dents ever having to directly report that they ever, Barack Obama was not selected as the performed the sensitive behavior. Democratic nominee until June 2008. A per- Other researchers have used experiments son living in a state that allows its citizens to assess the effects of question wording or to choose in which party’s primary they want order on the quality of attitude measurement. to vote and that held its primary after John For example, Krosnick et al. (2002) examined McCain became the presumptive Republi- whether explicitly including or omitting a “no can nominee, but before the Democratic can- opinion” or “don’t know” response option in didate was determined, might strategically telephone-administered attitude survey ques- choose to vote in the Democratic primary tions improved or decreased data quality. and vote for the candidate he or she believed Their findings show that including an explicit would be least likely to beat the Republican in no opinion response option reduces data the general election. In fact, during this pri- quality and gives respondents a cue to avoid mary, there was speculation in the media that going through the cognitive steps necessary Republicans may have voted for Hillary Clin- to optimally answer survey questions. These ton in Democratic primaries (in states where findings inform researchers about how best this was allowed) for just this reason (and to to ask attitude questions in surveys. disrupt and draw out the process of deciding the Democratic nominee).2 Thus, measures of behaviors are indirect measures of atti- 2. Observational Research Designs tudes that may contain error when used as measures of attitudes, although they can Many researchers have implemented obser- sometimes be directly observed in ways that vational or nonexperimental research designs attitudes cannot. to examine attitude change using a variety of approaches. One of the most basic approaches has been to evaluate the associations between Experiments and Attitude Measurement one or more hypothesized causes of attitudes Experiments have been widely used in mea- and attitudes themselves in cross-sectional suring attitudes and in improving attitude data. For example, one might look at the measurement. In particular, researchers have association between the frequency with which used survey experiments for these purposes respondents report having read a daily news- (see Sniderman’s chapter in this volume). For paper and attitudes toward the war in Iraq. example, researchers using the list technique If a negative association is found, one might implement a survey experiment to assess sen- conclude that greater exposure to newspaper sitive attitudes, beliefs, or behaviors (e.g., coverage led to less positive attitudes toward Kuklinski and Cobb 1998). This technique the war in Iraq. involves, for example, randomly assigning The assumption underlying this kind of half of the respondents a list of nonsensi- research design is that an association between tive behaviors and asking how many they attitudes and a hypothesized ingredient of have done. The other half of respondents are such attitudes suggests that the latter influ- randomly assigned to be given the same list enced the former (for a review, see Kinder of nonsensitive behaviors plus one sensitive 1998) and that the size of the association (often assessed via multiple linear regression) reflects the impact of each hypothesized 2 Helman, Scott. 2008. “Many Voting for Clinton to Boost GOP Seek to Prolong Bitter Battle.” The Boston ingredient on attitudes (see Rahn, Krosnick, Globe, March 17,p.A1. and Breuning 1994). This approach has been Attitude Change Experiments in Political Science 145 used to provide support for a long list of in climate, not to human actions. To study factors that influence attitudes toward candi- the effect of the initial efforts of the Clinton dates, including party affiliations and policy administration and the debate that followed, positions (Campbell et al. 1960), prospective a nationally representative sample of respon- judgments of candidates’ likely performance dents was surveyed about their attitudes and in office (Fiorina 1981), perceptions of can- beliefs toward global warming before the didates’ personalities, emotional responses to media coverage and debate. A second nation- candidates (Abelson et al. 1982), and retro- ally representative sample of adults was inter- spective assessments of the national econ- viewed after the media coverage and debate omy (Kinder and Kiewiet 1979). The primary had taken place. The effects of the media cov- problem with this assumption is the old caveat erage and debate were assessed by compar- that “correlation is not causation.” Finding an ing attitudes and beliefs measured before the association between two variables does not debate to those measured after. rule out the possibility that the hypothesized This kind of design more effectively iso- dependent variable (in this case, attitudes) is lates the causal influence of an event than actually the cause of the hypothesized inde- a simple association between hypothesized pendent variable (newspaper reading) or that cause and effect, but it is not without difficul- a third variable (e.g., perhaps political knowl- ties. The first is that any changes in attitudes edge or interest) is the cause of both the inde- in the samples over time are attributed to a pendent and dependent variable of interest. specific event occurring between data collec- Thus, the internal validity, or the confidence tions. As a result, conclusions about causality with which one can conclude from these stud- are threatened by history or by the possibility ies that the independent variable caused the that an event other than the one of interest dependent variable, is very low. For exam- to the researcher might have caused the atti- ple, although many researchers have tested tude change. Furthermore, there are practi- predictors of candidate evaluations via this cal issues with this design. Conducting this approach, analyses of longitudinal data sug- kind of study is more expensive than a cross- gest that people may form candidate eval- sectional design because it involves multiple uations or preferences first and that many data collections, and one has to be aware “predictors” of these evaluations or prefer- that a naturally occurring event will occur in ences (including the reasons respondents list advance (so that attitudes can be measured for voting for or against the candidate when before the event). asked directly) are in fact rationalized from Another observational design is used when the evaluations rather than the other way some people are exposed to a naturally occur- around (e.g., Rahn et al. 1994). ring event and others are not, but the exper- A second, less common, observational imenter does not control who is in each design used as an alternative to experimen- group (e.g., Huber and Arceneaux 2007). tation is a repeated cross-sectional design. In Inferences about causality from this design this design, attitudes are measured before and are threatened by self-selection because in after a naturally occurring event. For exam- these types of designs, respondents are not ple, Krosnick, Holbrook, and Visser (2000) randomly assigned to the two conditions. reported the results of just such a study to Therefore, respondents in the two groups examine the effects of the Clinton adminis- may have differed before the event of inter- tration’s efforts in 1997 to bring attention to est. For example, Huber and Arceneaux stud- the issue of global warming. These efforts led ied the effect of campaign advertisements on to a debate in the media in which Clinton and attitudes toward political candidates by com- other Democrats argued that global warm- paring the attitudes of residents in noncon- ing was occurring, whereas Republican lead- tested states who resided in an area near an ers argued that there was little or no evidence adjoining highly contested state to those who of global warming and that fluctuations in resided in an area in an uncontested state that temperature were due to natural fluctuations was not near an adjoining highly contested 146 Allyson L. Holbrook state. Thus, respondents were not randomly cal concerns, including attrition and added assigned to condition, but self-selected into costs. Although this design is one of the best the two conditions by virtue of where they observational designs for establishing causal- lived. In many cases, researchers argue that ity, it is also one of the least commonly used respondents in different conditions in these because of the practical difficulties with data 3 types of studies are comparable along many collection. dimensions, but because random assignment The most common threat to these nonexis not used to assign respondents, one cannot perimental designs is that they tend to have be confident that the two groups are com- low internal validity (for a review of threats parable on all important variables that might to internal validity, see Campbell and Stan- affect results. ley 1963; McDermott’s chapter in this vol- A final common observational design ume). Thus, it is difficult to conclude that the involves longitudinal data collection in which hypothesized independent variable caused the the same people are interviewed or assessed hypothesized dependent variable. Some nonat multiple points in time. This approach can experimental designs can also be difficult and be used to assess attitudes before and after a expensive to implement, and both the inter- naturally occurring event, as with the multi- nal and external validity of their conclusions ple cross-sectional data collection approach may be undermined by other potential threats described previously. The repeated measures such as attrition or history. approach has some advantages, specifically Some of these observational designs have that one can examine individual-level corre- the added problem of not directly assess- lates of attitude change between time 1 and ing attitude change. Instead, these designs time 2. However, it introduces an additional rely on the inference that an association possible threat of testing to internal validity between a hypothesized ingredient or cause (see Campbell and Stanley 1963). That is to of attitudes and attitudes themselves implies say that being interviewed at time 1 could influence or change. Instead of measuring change attitudes measured at time 2. attitudes in these designs, one could instead A second difficulty with this design is the ask respondents to report the direction and problem of attrition, whereby it may be dif- extent of any attitude change or to report ficult to reinterview all people interviewed at their past attitudes about an issue and use this time 1 and time 2. This can be particularly as a measure of attitude change. Research in problematic if the people interviewed only psychology, however, suggests that respon- at time 1 differ from those interviewed at dents cannot do this accurately because they both times, what is sometimes called “selec- may not be able to accurately report their past tive nonresponse” (e.g., Miller and Wright attitudes (e.g., Markus 1986). 1995; see also Taris 2000). Selective nonre- In studying attitude change, these observa- sponse can threaten both the external validity tional designs have at least one other potential of a study’s findings (e.g., if the final sample difficulty, which is that it is often difficult to used in the analyses is not representative) and know or measure the information to which the internal validity (e.g., if attrition across people have been exposed that might influ- panel waves is nonequivalent across groups). ence their attitudes. People often cannot An alternative approach to analyzing lon- remember all the information about an atti- gitudinal data is to examine the effects of tude object to which they have been exposed, predictors measured at time 1 on attitudes yet, this information may influence their measured at time 2 controlling for attitudes attitudes, even if it is not recalled (Lodge, at time 1. Because causes occur temporally McGraw, and Stroh 1989). A researcher before their consequences, this procedure helps establish causality and results in greater 3 Analyses of existing data such as those collected by confidence in internal validity than any of the American National Election Study (ANES) surveys are exceptions, although this type of data anal- the other observational designs reviewed thus ysis using the ANES data is relatively rare in the far. This design faces quite a few practi- literature. Attitude Change Experiments in Political Science 147 cannot, therefore, rely on self-reports to measure either exposure or the ingredients Experimental: R O1 XO2 of respondents’ attitudes. In longitudinal Control: R O3 O4 designs, the researcher often has to infer that some event or stimulus occurring between Figure 10.1. Pre-Test–Post-Test Control Group measures of attitude change is responsible Design for observed changes. Often, this is done Source: (Campbell and Stanley 1963, 13) by looking at the nature of attitude change Stanley (1963). In the purest version of this and inferring the information that must have type of design, respondents are randomly led to that change. For example, Krosnick assigned to two conditions (represented by et al. (2000) found that strong Democrats in Figure 10.1). All respondents’ attitudes became more convinced that global warming R are first measured ( 1 and 3 in Figure 10.1), was occurring over time and inferred that this O O and then half of respondents who have been was because strong Democrats were attend- randomly assigned to the experimental con- ing to and/or being persuaded by Democratic dition receive a treatment ( in Figure 10.1). leaders. Similarly, they found that Republi- X This treatment could be, for example, expo- cans became less convinced that global warm- sure to a persuasive message or a priming ing was occurring and inferred that this was manipulation. Multiple experimental groups because strong Republicans were attending exposed to different treatments may also be to and/or being persuaded by messages from included. The key comparison in this design Republican leaders. This is a theoretically is whether the difference between 1 and 2 reasonable explanation for their findings, but O O in the experimental group is different from message exposure and persuasion have to that between 3 and 4 in the control con- be inferred from the observed pattern of O O dition. This experimental design controls all attitudes. threats to internal validity and allows the researcher to conclude with confidence that 3. Experimental Designs the experimental treatment caused any differences across conditions. A second version of this design (shown in To overcome many of the difficulties with Figure 10.2 and labeled as the “pre-test–post- observational designs, political scientists have test multiple experimental condition design”) turned to experiments to test hypotheses uses multiple experimental conditions rather about attitude change. There are two hall- than a control group. In this design, respon- marks of experimental designs. First, the dents are randomly assigned to two (or more) experimenter manipulates the independent experimental conditions with different treat- variable (e.g., what information is provided ments ( and in Figure 10.2). Again, all to respondents about a political candidate). Xa Xb respondents’ attitudes are measured before Second, respondents or participants are ran- and after the treatment ( 1 and 3 in Figure domly assigned to conditions (e.g., groups O O 10.2). As with the pre-test–post-test control or levels of the independent variable). This group design, the key comparison is whether allows the researcher to be confident that the difference between 1 and 2 in experi- the only difference across groups or levels O O mental group A is different from that between is the independent variable. If any changes 3 and 4 in experimental group B. If dif- or differences in the dependent variables are O O ferences are observed across experimental observed, then the researcher can be confident that these are caused by differences in the independent variable. Experimental Group A: R O1 X a O2 Most of the work on attitude change has Experimental Group B: R O3 X b O4 relied on two basic experimental designs. First are versions of the pre-test–post-test Figure 10.2. Pre-Test–Post-Test Multiple Expe- control group described by Campbell and rimental Condition Design 148 Allyson L. Holbrook

undergraduates are different from a generally Experimental: R X O 1 representative sample (e.g., they tend to be Control: R O2 more homogenous in terms of socioeconomic status, education, age, and often race and eth- Figure 10.3. Post-Test–Only Control Group nicity). When studying attitude change, the Design homogeneity of age among college under- Note: The key comparison in this design is whether graduates may be of particular concern, O1 differs from O2. as susceptibility to persuasion is greatest dur- groups, then the researcher can confidently ing early and late adulthood (Visser and attribute these differences to the experimen- Krosnick 1998). Although the general pro- tal manipulation. cesses involved in attitude change may be The second type of experimental design similar in college students and in general used to study attitude change is what Camp- population samples, and therefore it may be bell and Stanley (1963) call the post-test–only appropriate to use samples of students in control group design (Figure 10.3). In this research focusing on basic effects and mech- design, respondents are randomly assigned anisms, researchers studying potential mod- to either a control condition or one (or erators of attitude change processes on which more) experimental conditions. One group of college students are relatively homogenous respondents is randomly assigned to receive (e.g., education, age) should use more het- a treatment (X in Figure 10.3). Then the erogeneous samples. Implications of the use attitudes of both experimental and control of college students as participants in attitude groups are measured. change research continues to be an impor- A second version of this type of design tant research topic (see Druckman and Kam’s (shown in Figure 10.4) uses multiple exper- chapter in this volume). imental groups, but no control condition. A second weakness of experiments study- Respondents are randomly assigned to two ing attitude change is that they often use (or more) experimental conditions in which artificial stimuli and processes that do not they are exposed to different experimental accurately mirror people’s everyday world. treatments. Then respondents’ attitudes are For example, to avoid the influence of pre- measured. existing attitudes, knowledge, and beliefs about political candidates, researchers will Weaknesses of Experimental Designs sometimes how study participants form attitudes toward hypothetical candidates or poli- Although experimental designs provide much cies (e.g., Lodge et al. 1989; although see, higher internal validity than observational e.g., Kaid and Boydston 1987). This allows designs, they are not without weaknesses. researchers to accurately isolate the effects One potential weakness is that in many cases, of the information being presented; however, experiments studying attitude change used this process may be different from one by samples of undergraduate students. Although which people actually acquire information many laboratory experiments replicate when about candidates. conducted with representative samples (e.g., Experiments also typically vary from real- 2000 Krosnick, Visser, and Holbrook ), there world attitude formation and change con- are many important ways in which college texts in that information is acquired over a much shorter period of time. An experiment might last an hour in its entirety or, in some Experimental Group A: R X a O1 cases, be extended over several days or weeks, Experimental Group B: R X b O2 but the acquisition of information about Figure 10.4. Post-Test–Only Multiple policies and candidates often occurs over a Experimental Group Design period of months or even years. As such, Note: The key comparison here is whether O1 is attitude change studied in the laboratory different than O2. may not reflect the processes that actually Attitude Change Experiments in Political Science 149 occur during campaigns or in response to be obtained in other settings, with different media coverage or advertisements. Perhaps research participants and different research more important, attitude change studied dur- procedures” (Brewer 2000, 10). Ecological ing the brief time period of a laboratory validity is a type or subcategory of external experiment may not reflect the kind of change validity that deals with whether “the effect that persists over time or would be likely is representative of everyday life” (12). Prob- to influence behavior, although these conse- lems with the artificiality of the experimental quences of attitudes are ones that are often of context and processes, as well as lack of sam- great interest to researchers. Few laboratory ple representativeness, may reduce both the studies assess whether attitude change per- external and ecological validity of laboratory sists or influences later behavior (although see experiments assessing attitude change. Boninger et al. 1990; Druckman and Nelson Laboratory experiments may therefore 2003; Mutz and Reeves 2005). explore attitude change and formation pro- Experiments examining attitude change in cesses that do not reflect real-world processes, political science often do not fully incorporate and these experiments may show researchers two processes that have a great deal of influ- what can versus what does occur. For psychol- ence on persuasion: selective exposure and ogists interested in the psychological mech- selective elaboration. In facing the “buzzing, anisms underlying processes such as attitude blooming confusion” (James 1890, 462)of change, this may not pose a great concern. information that they are faced with each Political scientists studying attitude change, day, people make decisions about what infor- however, typically want to apply their find- mation to be exposed to (e.g., they choose ings to processes of attitude formation or whether to watch a particular television news change that do occur in the real world, such as show or whether to read about a topic on a the processes by which campaigns influence news web site). Despite this, many attitude voter evaluations of candidates or by which change experiments do not take into account media coverage of an issue affects the pub- or allow for selective exposure, although lic’s evaluation of a particular proposed pol- researchers have examined this process sep- icy. As a result, the potential issues with both arately (e.g., Huang and Price 2001). ecological and external validity in laboratory Furthermore, people choose which infor- experiments may be of concern to political mation they want to elaborate on and which scientists. they do not want to think about. Persua- sion processes can occur through both more and less thoughtful processes, and whether 4. Findings from Experimental persuasion occurs via high or low elabora- Attitude Change Research tion may have important consequences for its longevity and effects (e.g., Petty and The literature using experiments to study Cacioppo 1986). Although elaboration has attitude change in political science is exten- been shown to be a key process in persuasion sive and much too large to be reviewed in processes, relatively few experiments in polit- detail in this chapter. This literature can ical science have measured or assessed elabo- be organized in two ways. Experimental ration (although see Nelson and Garst 2005). attitude change research has had a major Thus, the primary weaknesses of exper- influence in a number of substantive areas imental designs, particularly as they have of research in political science. Most of been used to study attitude change in politi- the research examining attitude change has cal science, relate to external validity, specif- focused on attitudes toward either political ically ecological validity (see McDermott’s candidates or issues. The use of experimen- chapter in this volume). “External validity tal procedures has been widespread in studies refers to the question of whether an effect examining the processes by which peo- (and its underlying processes) that has been ple’s attitudes toward political candidates are demonstrated in one research setting would formed and changed (e.g., Nelson and Garst 150 Allyson L. Holbrook

2005; McGraw’s chapter in this volume); this research has focused on the role of respon- includes research on the effects of campaign dent characteristics such as race or gender advertising and media effects such as media (e.g., Kuklinski and Hurley 1994; Peffley and priming and agenda setting (e.g., Miller and Hurwitz 2007); preexisting attitudes; orienta- Krosnick 2000; Nelson’s chapter in this vol- tions or abilities such as political knowledge, ume). Experiments have also been widely sophistication, or expertise (e.g., Druckman used to study attitude change about political 2004;Gartner2008); and personality or indi- issues, in particular, processes such as issue vidual differences (e.g., need for cognition framing (e.g., Chong and Druckman 2007; or need to evaluate; Druckman and Nelson Transue 2007; Brader, Valentino, and Suhay 2003;Kam2005). 2008;Gartner2008; Nelson et al.’s chapter For example, Gartner (2008) conducted an in this volume). Experiments have been used experiment in which half of respondents were less frequently to assess change in attitudes randomly assigned to read a message report- toward groups in society (e.g., Gaffie´ 1992; ing that experts thought casualties would Glaser 2003), as well as attitudes toward gov- decrease in the future, and half were ran- ernment and other public institutions such as domly assigned to read a message reporting the Supreme Court (e.g., Iyengar, Peters, and that experts thought casualties would increase Kinder 1982; Mondak 1990). in the future. He found that the effect of A second approach to organizing and information about casualty predictions in the describing the experimental literature on atti- war in Iraq had a greater effect on attitudes tude change is to think about how experimen- toward the war among respondents who knew tation has contributed to an understanding of little about the war than among respondents attitude change processes. First, experiments who knew more about the war. have been used to assess and understand the Finally, experimentation has provided a ingredients of attitudes and what types of great deal of insight into the processes and persuasive attempts do and do not lead to mechanisms by which attitudes are formed attitude change, including the content of per- and changed. This includes insights into at suasive messages, the source of these mes- least three aspects of these processes. sages, and factors that influence resistance to Researchers have studied processes that influ- persuasion (e.g., Andreoli and Worchel 1978; ence how people weigh individual beliefs or Bizer and Petty 2005). For example, Bizer pieces of information in forming attitudes and Petty found that simply reporting that (e.g., Nelson, Clawson, and Oxley 1997; one is “opposed” to a policy or candidate led Miller and Krosnick 2000; Transue 2007). to greater resistance to persuasion than say- For example, Valentino, Hutchings, and ing that one is “supportive” of the policy or White (2002) found that respondents who candidate. This occurred regardless of which read a persuasive message criticizing George position was framed as “opposition.” In one W. Bush paired with subtle racial cues (e.g., study, study participants who were asked to pictures of blacks to illustrate some of the report how much they “opposed” their least arguments) used racial attitudes in assessing liked candidate showed greater resistance to Bush more than respondents who received persuasion than those who were asked how the same message paired with neutral visual much they “favored” their most liked candi- images (e.g., the Statue of Liberty). Other date, regardless of which candidate they pre- areas of research have also focused on the ferred. weights assigned to different pieces of infor- A second area in which experimentation mation, for example, research examining has provided key insights is in understand- framing effects (see Nelson et al.’s chapter ing potential moderators of attitude forma- in this volume). tion and change processes. Experiments have Researchers have also examined the pro- helped researchers understand when and for cesses by which people integrate information whom particular types of persuasive mes- into overall attitudes (e.g., Lodge et al. 1989). sages or information influence attitudes. This For example, Lodge et al. demonstrated Attitude Change Experiments in Political Science 151 that participants primarily formed evalua- complex, and, in many cases, more realistic, tions of candidates online (as information picture of attitude change processes. was received) rather than via a memory-based process (whereby evaluations are formed at the time they are reported based on the 5. Recommendations for Future information available about the candidate in Research memory). Finally, researchers have examined medi- Although experimentation has contributed a ators of the effects of persuasive messages, great deal to our understanding of attitude such as emotions and cognitive processes. change and persuasion in political science, For example, Brader, Valentino, and Suhay future research could make even greater con- (2008) found that people were less supportive tributions by minimizing some of the weak- of immigration when an article about immi- nesses of the experimental research done gration was accompanied by a Latino cue to date. First, researchers need to increase and that this effect was mediated by anx- external validity, particularly ecological valid- iety about immigration. Thus, reading the ity, by designing future studies to assess article about the costs of immigration paired what does happen rather than what does not with a Latino salience cue led respondents to happen. This can be done by using stimu- be the most anxious about immigration and, lus materials and other experimental proce- as a result, the most supportive of reducing dures that mirror the attitude formation and immigration. change processes that occur in the real world Of course, many experiments include ele- (as some researchers have already begun to ments of more than one of these approaches. do; e.g., Chong and Druckman 2007; Gart- For example, Miller and Krosnick (2000) ner 2008). Furthermore, researchers need found that people who see many news sto- to assess these processes with representa- ries about a political issue weighed that issue tive samples to increase the generalizabil- more heavily when evaluating a presidential ity of their findings. Finally, researchers can candidate than an issue about which they increase the external validity of research, not saw few news stories (commonly known as necessarily by making experiments more real- news media priming). The priming effect was istic, but by using experiments along with strongest among knowledgeable study partic- other types of designs (e.g., observational ipants who trusted the media (two moderators designs with higher external validity, but of the effect), and the primary mechanism was lower internal validity) to illustrate that find- the perceived importance of the issue. Thus, ings from multiple types of designs show sim- media coverage influenced evaluations of can- ilar findings (e.g., Forgette and Morris 2006). didates via issue importance, and this effect In addition to increasing external valid- was moderated by respondents’ knowledge of ity, researchers designing and conducting and trust in the media. future attitude change experiments could also Similarly, McGraw, Hasecke, and Con- increase their impact by not only assessing ger (2003) not only examined the processes attitude change, but also assessing whether by which information is integrated into can- the attitude change that is observed lasts over didate evaluations (contrasting online and any meaningful period of time or impacts memory-based processes), but they also fur- behavior. These consequences are one of the ther examined when and for whom each primary reasons political scientists are inter- process is more or less likely to occur. For ested in attitudes; thus, demonstrating that example, they found that both attitudinal observed attitude change also leads to these ambivalence and uncertainty are associated consequences (e.g., behavioral changes) is with more memory-based (vs. online) pro- key for demonstrating the importance of the cessing. These are just two examples of many research. Finally, future experiments study- that simultaneously examine more than one ing attitude change should take into ac- aspect of attitude change, providing a more count (either by measuring or manipulating) 152 Allyson L. Holbrook cognitive processes, such as selective expo- tion to Immigration? Anxiety, Group Cues, and sure and elaboration, that likely play a role in Immigration Threat.” American Journal of Polit- persuasion processes in actual political cam- ical Science 52: 959–78. 2000 paigns or media effects. Brewer, Marilyn. . “Research Design and Issues of Validity.” In Handbook of Research Methods in Social and Personality Psychology, eds. Harry T. Reis and Charles M. Judd. Cam- References bridge: Cambridge University Press, 3–16. Campbell, Angus, Phillip E. Converse, Warren E. Abelson, Robert P., Donald R. Kinder, Mark D. Miller, and Donald E. Stokes. 1960. The Amer- Peters, and Susan T. Fiske. 1982. “Affective and ican Voter. New York: John Wiley & Sons. Semantic Components in Political Person Per- Campbell, Donald T., and Julian C. Stanley. 1963. ception.” Journal of Personality and Social Psy- Experimental and Quasi-Experimental Designs for chology 42: 619–30. Research. Boston: Houghton Mifflin. Ajzen, Icek, and Martin Fishbein. 2005.“The Cantril, Hadley. 1934. “Attitudes in the Making.” Influence of Attitudes on Behavior.” In The Understanding the Child 4: 13–15. Handbook of Attitudes, eds. Doris Albarracın,´ Chong, Dennis, and James N. Druckman. 2007. Blair T. Johnson, and Mark P. Zanna. Mah- “Framing Public Opinion in Competitive wah, NJ: Erlbaum, 173–221. Democracies.” American Political Science Review Allport, Gordon W. 1935. “Attitudes.” In A Hand- 101: 637–55. book of Social Psychology, ed. Carl Murchison. Converse, Philip E. 1964. “The Nature of Belief Worcester, MA: Clark University Press, 798– Systems in Mass Publics.” In Ideology and Dis- 844. content, ed. David E. Apter. New York: Free Andreoli, Virginia, and Stephen Worchel. 1978. Press, 206–61. “Effects of Media, Communicator, and Mes- Converse, Philip E. 1970. “Attitudes and Non- sage Position on Attitude Change.” Public Opin- Attitudes: Continuation of a Dialogue.” In ion Quarterly 42: 59–70. The Quantitative Analysis of Social Problems, Baumgartner, Hans, and Jan-Benedict E. M. ed. Edward R. Tufte. Reading: MA: Addison- Steenkamp. 2001. “Response Styles in Market- Wesley, 168–89. ing Research: A Cross-National Investigation.” Davis, James A., Tom W. Smith, and Peter V. Journal of Marketing Research 38: 143–56. Marsden. 2007. General Social Surveys, 1972– Belli, Robert F., Michael W. Traugott, Mar- 2006 Cumulative Codebook. Chicago: National garet Young, and Katherine A. McGonagle. Opinion Research Center, The University of 1999. “Reducing Vote Over-Reporting in Sur- Chicago. veys: Social Desirability, Memory Failure, and Druckman, James N. 2004. “Political Preference Source Monitoring.” Public Opinion Quarterly Formation: Competition, Deliberation, and the 63: 90–108. (Ir)relevance of Framing Effects.” American Bem, Daryl J. 1970. Beliefs, Attitudes, and Human Political Science Review 98: 671–86. Affairs. Belmont, CA: Brooks/Cole. Druckman, James N., and Arthur Lupia. 2000. Berinsky, Adam J. 2002. “Political Context and “Preference Formation.” American Review of the Survey Response: The Dynamics of Racial Political Science 3: 1–24. Policy Opinion.” Journal of Politics 64: 567–84. Druckman, James N., and Kjersten R. Nelson. Bizer, George Y., and Richard E. Petty. 2005. 2003. “Framing and Deliberation: How Cit- “How We Conceptualize Our Attitudes Mat- izens’ Conversations Limit Elite Influence.” ters: The Effects of Valence Framing on the American Journal of Political Science 47: 729–45. Resistance of Political Attitudes.” Political Psy- Eagly, Alice H., and Shelly Chaiken. 1993. The Psy- chology 26: 553–68. chology of Attitudes. Fort Worth, TX: Harcourt Boninger, David S., Timothy C. Brock, Thomas Brace Jovanovich. D. Cook, Charles L. Gruder, and Daniel Fiorina, Morris P. 1981. Retrospective Voting in Romer. 1990. “Discovery of Reliable Attitude American National Elections. New Haven, CT: Change Persistence Resulting from a Trans- Yale University Press. mitter Tuning Set.” Psychological Science 1: 268– Forgette, Richard, and Jonathan S. Morris. 2006. 71. “High-Conflict Television News and Public Brader, Ted, Nicholas A. Valentino, and Elizabeth Opinion.” Political Research Quarterly 59: 447– Suhay. 2008. “What Triggers Public Opposi- 56. Attitude Change Experiments in Political Science 153

Gaffie,´ B. 1992. “The Processes of Minority Influ- Rudd, and V. Kerry Smith. 2002. “The Impact ence in an Ideological Confrontation.” Political of ‘No Opinion’ Response Options on Data Psychology 13: 407–27. Quality: Prevention of Non-Attitude Report- Gartner, Scott Sigmund. 2008. “The Multiple ing or an Invitation to Satisfice?” Public Opinion Effects of Casualties on Public Support for Quarterly 66: 371–403. War: An Experimental Approach.” American Krosnick, Jon A., Allyson L. Holbrook, and Penny Political Science Review 102: 95–106. S. Visser. 2000. “The Impact of the Fall 1997 Glaser, James M. 2003. “Social Context and Debate about Global Warming on American Inter-Group Political Attitudes: Experiments Public Opinion.” Public Understanding of Science in Group Conflict Theory.” British Journal of 9: 239–60. Political Science 33: 607–20. Krosnick, Jon A., and Richard E. Petty. 1995. Henry, P. J., and David O. Sears. 2002.“TheSym- “Attitude Strength: An Overview.” In Atti- bolic Racism 2000 Scale.” Political Psychology 23: tude Strength: Antecedents and Consequences, eds. 253–83. Richard E. Petty and Jon A. Krosnick. Hills- Himmelfarb, Samuel, and Carl Lickteig. 1982. dale, NJ: Erlbaum, 1–24. “Social Desirability and the Randomized Kuklinski, James H., and Michael D. Cobb. 1998. Response Technique.” Journal of Personality and “When White Southerners Converse about Social Psychology 43: 710–17. Race.” In Perception and Prejudice,eds.JonHur- Huang, Li-Ning, and Vincent Price. 2001. witz and Mark Peffley. New Haven, CT: Yale “Motivations, Goals, Information Search, and University Press, 35–57. Memory about Political Candidates.” Political Kuklinski, James H., and Norman L. Hurley. Psychology 22: 665–92. 1994. “On Hearing and Interpreting Political Huber, Gregory A., and Kevin Arceneaux. 2007. Messages: A Cautionary Tale of Citizen Cue- “Identifying the Persuasive Effects of Presiden- Taking.” Journal of Politics 56: 729–51. tial Advertising.” American Journal of Political Lodge, Milton, Kathleen McGraw, and Patrick Science 51: 957–77. Stroh. 1989. “An Impression-Driven Model of Iyengar, Shanto, Mark D. Peters, and Donald Candidate Evaluation.” American Political Sci- R. Kinder. 1982. “Experimental Demonstra- ence Review 83: 399–419. tions of the ‘Not-So-Minimal’ Consequences Markus, Gregory B. 1986. “Stability and Change of Television News Programs.” American Polit- in Political Attitudes: Observed, Recalled, and ical Science Review 76: 848–58. Explained.” Political Behavior 8: 21–44. James, William. 1890. The Principles of Psychology. McGraw, Kathleen M., Edward Hasecke, and New York: Holt. Kimberly Conger. 2003. “Ambivalence, Kaid, Lynda Lee, and John Boydston. 1987.“An Uncertainty, and Processes of Candidate Experimental Study of the Effectiveness of Evaluation.” Political Psychology 24: 421–48. Negative Political Advertisements.” Communi- Miller, Joanne M., and Jon A. Krosnick. 2000. cation Quarterly 35: 193–201. “News Media Impact on the Ingredients of Kam, Cindy D. 2005. “Who Toes the Party Presidential Evaluations: Politically Knowl- Line? Cues, Values, and Individual Differ- edgeable Citizens Are Guided by a Trusted ences.” Political Behavior 27: 163–82. Source.” American Journal of Political Science 44: Kinder, Donald R. 1998. “Opinion and Action in 301–15. the Realm of Politics.” In Handbook of Social Psy- Miller, Joanne M., and Jon A. Krosnick. 2004. chology, eds. Daniel T. Gilbert, Susan T. Fiske, “Threat as a Motivator of Political Activism: A and Gardner Lindzey. New York: McGraw- Field Experiment.” Political Psychology 25: 507– Hill, 778–867. 23. Kinder, Donald R., and D. Roderick Kiewiet. Miller, Richard B., and David W. Wright. 1995. 1979. “Economic Discontent and Political “Detecting and Correcting Attrition Bias in Behavior: The Role of Personal Grievances and Longitudinal Family Research.” Journal of Collective Economic Judgments in Congres- Marriage and Family 57: 921–29. sional Voting.” American Journal of Political Sci- Mondak, Jeffery J. 1990. “Perceived Legitimacy of ence 23: 495–527. Supreme Court Decisions: Three Functions of Krosnick, Jon A., Allyson L. Holbrook, Matthew Source Credibility.” Political Behavior 12: 363– K. Berent, Richard T. Carson, W. Michael 84. Hanemann, Raymond J. Kopp, Robert Mutz, Diana C., and Byron Reeves. 2005. Cameron Mitchell, Stanley Presser, Paul A. “The New Videomalaise: Effects of Televised 154 Allyson L. Holbrook

Incivility on Political Trust.” American Political Taris, Toon. 2000. A Primer in Longitudinal Data Science Review 99: 1–15. Analysis. Thousand Oaks, CA: Sage. Nelson, Thomas E., Rosalee A. Clawson, and Zoe Thurstone, Louis L., ed. 1931. The Measurement M. Oxley. 1997. “Media Framing of a Civil Lib- of Social Attitudes. Chicago: The University of erties Conﬂict and Its Effect on Tolerance.” Chicago Press. American Political Science Review 91: 567–83. Transue, John E. 2007. “Identity Salience, Id- Nelson, Thomas E., and Jennifer Garst. 2005. entity Acceptance, and Racial Policy Atti- “Values-Based Political Messages and Persua- tudes: American National Identity as a Uniting sion: Relationships among Speaker, Recipient, Force.” American Journal of Political Science 51: and Evoked Values.” Political Psychology 26: 78–91. 489–515. Valentino, Nicholas A., Vincent L. Hutchings, Pefﬂey, Mark, and Jon Hurwitz. 2007. “Persuasion and Ismail K. White. 2002. “Cues That Mat- and Resistance: Race and the Death Penalty in ter: How Political Ads Prime Racial Attitudes America.” American Journal of Political Science during Campaigns.” American Political Science 51: 996–1012. Review 96: 75–90. Petty, Richard E., and John T. Cacioppo. 1986. Visser, Penny S., and Jon A. Krosnick. 1998. Communication and Persuasion: Central and “Development of Attitude Strength over the Peripheral Routes to Attitude Change. New York: Life Cycle: Survey and Decline.” Journal of Per- Springer-Verlag. sonality and Social Psychology 75: 1389–410. Rahn, Wendy M., Jon A. Krosnick, and Marijke Warner, Stanley L. 1965. “Randomized Response: Breuning. 1994. “Rationalization and Deriva- A Survey Technique for Eliminating Evasive tion Processes in Survey Studies of Politi- Answer Bias.” Journal of the American Statistical cal Candidate Evaluation.” American Journal of Association 60: 63–69. Political Science 38: 582–600. Wolosin, Robert J., Steven J. Sherman, and Arnie Rosenstone, Steven J., and John Mark Hansen. Cann. 1975. “Predictions of Own and Other’s 1993. Mobilization, Participation, and Democracy Conformity.” Journal of Personality 43: 357– in America. New York: Macmillan. 78. CHAPTER 11

Conscious and Unconscious Information Processing with Implications for Experimental Political Science

Milton Lodge, Charles Taber, and Brad Verhulst

Affect-driven dual process models domi- “true” beliefs or attitudes and are willing and nate contemporary psychological theorizing able to accurately report them (Wittenbrink about how people think, reason, and decide 2007). (Chaiken and Trope 1999; Wilson, Lindsey, Most of our daily life is experienced uncon- and Schooler 2000; Gawronski and Boden- sciously, outside awareness. Consequently, it hausen 2006). Although most dual process is quixotic to focus exclusively on conscious models focus on accuracy-efﬁciency trade- attitudes while ignoring considerations that offs, hundreds of more recent experiments escape conscious awareness. Recent estimates document the pervasive effects of uncon- put the total human capacity for visual sen- scious thoughts, feelings, and behaviors on sory processing in the neighborhood of 10 attitude formation, attitude change, prefer- million bits per second, even though we can ences, and decision making. These stud- become conscious of only about 40 bits per ies reveal important differences between the second (Norretranders 1998). Although the inﬂuence of conscious and unconscious pro- absolute input from other sensory modali- cessing on how people think and reason. The ties such as touch, smell, and hearing is con- explicit incorporation of unconscious cogni- siderably less than visual sensory input, the tion into models of political beliefs challenges differential processing capacity between con- the extant understanding of mass beliefs. scious and unconscious perception in these Much of what we political scientists claim to domains is similarly lopsided in favor of know about citizens’ political beliefs and atti- unconscious processing. Importantly, peo- tudes is based on verbal self-reports. The vast ple can only consciously process approxi- majority of the empirical evidence in political mately seven chunks of information at any behavior research is based directly on verbal given time irrespective of the type of infor- responses to explicit questions. This reliance mation (Miller 1957). With these serious on explicit measures of political attitudes and limitations on conscious attention, various behaviors is problematic because these mea- heuristic devices have evolved to reduce the sures assume people have direct access to their amount of information that they must process

155 156 Milton Lodge, Charles Taber, and Brad Verhulst consciously. Where and when conscious a sequential priming paradigm (Fazio et al. information processing strategies prove to be 1986), or emotional transference procedures more or less effective than unconscious infor- such as the Affect Misattribution Procedure mation processing strategies is a critical ques- (AMP; Payne et al. 2005). In contrast, explicit tion. At the very least, the difference between attitudes are mediated by controlled, con- the staggering amount of sensory input and scious thought and can be measured by survey the constraints of conscious processing leaves techniques or other explicit verbal responses. open the possibility for the introduction of To simplify our discussion, we favor the unconscious processing into models of polit- terms conscious and unconscious when referring ical behavior and decision making. to information processing styles, and reserve Both conscious and unconscious processes the terms implicit and explicit for attitudes or are continuously at work, not only when peo- attitude measures. We do not assume a dis- ple make snap judgments, but also – contrary juncture between conscious and unconscious to 2,000 years of Western thought – when processing because people can process stim- people are called on to think hard and weigh uli both consciously and unconsciously, and pros and cons before forming an attitude or either style of information processing can making decisions. Research in the cognitive influence both implicit and explicit attitudes and neurocognitive sciences has used multi- (Monroe and Read 2008). ple labels to distinguish between these two Conscious processing is simply informa- styles of information processing in the formation processing of which we are aware. The tion and expression of beliefs, attitudes, goals, complement to this style of processing – and behavior, chief among them the con- unconscious processing – is that informa- cepts explicit and implicit, deliberative and tion processing of which we are unaware. automatic, or system 2 and system 1 process- Most habits and heuristics would fall into the ing. This research demonstrates not only that unconscious processing category, which does unconscious processing can influence con- not imply that we are unaware of our habits, scious attitudes and behaviors (e.g. Bargh, that we cannot use heuristics consciously Chen, and Burrows 1996), but also that con- (Lupia 1994; Lau and Redlawsk 2001), or that sciously activated goals can affect uncon- we can think carefully and effectively only if scious processing strategies (e.g., Chaiken we do so consciously (Wilson and Schooler and Maheswaran 1994; Aarts and Dijkster- 1991; Dijksterhuis 2004). huis 2000). Thus, although conscious and We can only distinguish empirically bet- unconscious processing strategies can operate ween conscious and unconscious information independently, it is also common for process- processing, including their joint or indepen- ing at one level to influence the processing at dent effects, by using experimental methods the other. because research on unconscious processing Related, although conceptually distinct requires control over how information is pre- from these processing strategies, are implicit sented to determine when and how uncon- and explicit attitudes. According to Green- scious information is being processed. Thus, wald, McGhee, and Schwartz (1998), the control required to identify the cognitive “Implicit attitudes are manifest as actions and affective mechanisms involved in these or judgments that are under the control of two styles of information processing negates automatically activated evaluation, without the possibility of observational research. the performer’s awareness of that causation” (4). Thus, implicit attitudes are automatic, evaluative tendencies that influence people’s 1. Conscious and Unconscious thoughts and behaviors outside their con- Processing in the Political Domain scious awareness. These implicit attitudes can be measured by multiple approaches, Political scientists routinely acknowledge including the Implicit Association Test (IAT; several unconscious processing mechanisms Greenwald et al. 1998), reaction times in in accounting for how people think, feel, or Conscious and Unconscious Information Processing 157 behave in the political world. We focus on (Lodge et al. 1995). Note that within this three areas of research in political science that context the measurement of the online eval- incorporate unconscious processing into their uation is explicit, but there is now empirical explanations of political attitudes and pref- evidence and theoretical rationale suggesting erences: online processing, implicit attitudes that online processing is automatic: people and measurement, and situations where the evaluate and integrate their evaluations into a stimulus may be noticed but its influence on summary judgment effortlessly, outside con- judgments and behaviors is unappreciated. scious awareness. A well-replicated finding is that people can integrate a great deal of complex infor- Online Information Processing mation into a summary evaluation in real The online model holds that beliefs and atti- time but prove unable to recall much of this tudes are constructed in real time as peo- information after a short time lapse (Hastie ple encounter information and integrate it and Park 1986;Redlawsk2001; Steenbergen into existing networks of associations in long- and Lodge 2003). Specifically, the number of term memory (Anderson and Barrios 1961; items people accurately recall decays expo- Hastie and Park 1986; Lodge, Steenbergen, nentially, so that – within days if not min- and Brau 1995). Affect plays the critical role utes – the information a person remembers in this online updating process. When people no longer captures the information that was form or revise their impressions of persons, presented in the message or predicts the eval- places, events, or issues, they spontaneously uation. Interestingly, the information people extract the affective value of the message and remember when engaged in online process- within milliseconds update their summary ing differs markedly from the information evaluation of the object. This “running tally” people recall when relying on memory-based integrates new information with one’s prior processing. Specifically, online tallies show evaluation of an object and is then restored to evidence of a primacy effect whereby a per- memory, where it is readily available for sub- son’s tally is anchored on the initial informa- sequent retrieval (Cassino and Lodge 2007). tion encountered about the object, whereas A central tenet of the online model is that the memory-based evaluations are heavily depen- process by which people form attitudes is not dent on the most recent information encoun- routinely mediated by conscious information tered. More problematic still in political processing: people do not intentionally form science practice is that the more people are or update tallies, but rather evaluate people, encouraged to ruminate about a message, events, and ideas spontaneously. the greater the impact of rationalizations In an experimental setting, the measure- on memory (Wilson, Hodges, and LaFleur ment of online processing proceeds in five 1995; Erisen, Taber, and Lodge 2006). When stages. First, participants evaluate all infor- they are called on to stop and think before mation that will be subsequently presented, responding, people tend to overemphasize plus other pieces of information that will not accessible information and construct an atti- appear in the message, so as to later check for tude based on what is temporarily accessible rationalization effects in recall. Next, there (Zaller and Feldman 1992). is a distracter task, perhaps questions asking Early descriptions of these mechanisms drew for demographics. Then, participants read too sharp a distinction between memory- information about one or more candidates or based and online processing. An either/or issues, typically embedded in narrative form view is theoretically flawed and empirically as a newspaper article or newscast. In the unfounded. The confusion stems from the fourth stage, participants evaluate the candi- failure to discriminate encoding from retrieval date or issue. Finally, participants are asked effects. The encoding process is inherently to recall the information in the message, unconscious and occurs automatically, re- followed by questions probing for gist and gardless of whether subsequent retrieval details about the candidate’s issue positions focuses on the online tally or a broader sample 158 Milton Lodge, Charles Taber, and Brad Verhulst of memory-based considerations. During prices of five hypothetical companies pre- encoding, affect and cognition become sented on a crawler at the bottom of the strongly linked in memory and difficult to television screen. Although participants were disentangle. Affective tags are attached to led to believe that the task assessed their concepts when an object is first evaluated ability to remember and evaluate the com- and subsequently strengthened every time a mercials while being distracted, the study person thinks about the object (Lodge and actually tested whether they could track the Taber 2005). Thus, when people rely on stock ticker information. As predicted by the online processing, they simply report their online model, participants were unable to general gut reactions that are embedded recall the pertinent stock information, yet in their affective tallies. When asked to their evaluations correlated strongly with the explain their attitudes, their justifications actual stock prices of the five companies. This rationalize their online tallies, biased by other result points to the automaticity of online accessible thoughts. evaluations: experimental subjects accurately On retrieval, affect is primary in three evaluated the companies’ stock performances senses. First, the affective component of a even when they actively focused on other concept enters the decision stream earlier information and were unable to recall the than the concept’s semantic associations (you stock prices. know whether you like or dislike Barack From this perspective, an online tally Obama before you remember that he is a anchors a person’s evaluation of an object and Democrat and an African American). Conse- spontaneously infuses the encoding, retrieval, quently, affective reactions anchor evaluative and comprehension of subsequent informa- judgments (Zajonc 1980). Second, semantic tion and its expression as a preference; it also information follows the oft-noted exponen- readies us to act aversively or appetitively tial forgetting curve for factual information, in accordance with our evaluations (Ito and whereas affective information remains Cacioppo 2005). Importantly, this readiness relatively stable over longer periods of to act need not be, and typically is not, medi- time (Lodge et al. 1995). Third, online ated by conscious thought. evaluations, spontaneously activated on mere exposure to a stimulus, bias the recall of information in affectively consistent ways, 2. Implicit Attitudes resulting in affectively driven rationalizations (Rahn, Krosnick, and Breuning 1994; Erisen, An appreciation of unconscious informa- Lodge, and Taber 2007). Positive candidate tion processing of implicit attitudes has evaluations, for example, heighten the acces- recently infiltrated political behavior re- sibility of other positive considerations in search. Because attitudes are latent con- memory. Thus, if people are asked to justify structs, they cannot be directly observed but their preferences, then they simply search for must be inferred from self-report or nonver- accessible reasons for their automatic affec- bal responses such as reaction time. For our tive reactions rather than the actual infor- purposes, let us define an attitude rather mation that formed the online evaluation generally as an evaluative orientation toward initially. an object, whether a politician, issue, social An exemplary empirical demonstration of group, or abstract concept. Although implicit online processing was carried out by Betsch measures can be used for both explicit et al. (2001), who had participants watch and implicit attitudes, the measurement of a series of television commercials, telling implicit attitudes demands indirect proce- them they would have to later recall and dures. evaluate the advertised products. Simultane- The basic theory underlying tests of asso- ously, subjects were engaged in a second, ciations among concepts and their subse- cognitively demanding distracter task: they quent effects on behavior is the associative were asked to read aloud the changing stock network model of cognition (Anderson 1983, Conscious and Unconscious Information Processing 159

1993).1 This theory is predicated on the priming paradigm (Fazio et al. 1986), the IAT well-documented observation that activa- (Greenwald et al. 1998), and the AMP (Payne tion spreads along semantically and affec- et al. 2005). Each procedure relies on slightly tively associated pathways (Collins and Quil- different cognitive mechanisms to assess atti- lian 1969; Collins and Loftus 1975; Fazio tudes. et al. 1986; Bargh 1999). The more often The sequential priming paradigm provides two concepts have been linked, the more the most direct test of the strength of asso- strongly they become associated. This is ciations (Figure 11.1). Here subjects are pre- reflected in faster retrieval of one concept sented with a prime followed by a target word when the other is primed, influence of one or phrase. The subject’s task is to respond concept on interpretations of the other, and to the target by pressing one response but- greater likelihood of paired retrieval in free ton if the target is a member of category memory tasks. Within milliseconds of per- X or another if the target is a member of ceiving a concept (whether word or image), category Y. Because the dependent variable activation spreads automatically to associated is the latency between the initial presenta- concepts, including semantic and affective tion of the target and the response, partici- links. For example, mere milliseconds after pants are instructed to respond “as quickly as seeing a picture of Barack Obama, activa- possible without making too many errors.” tion spreads to affective associations (online For strongly associated concepts, whether tallies) and then to other related concepts semantic (BUSH – REPUBLICAN) or affec- in memory such as “Democrat” or “war in tive (OBAMA – RAINBOW), the activa- Afghanistan.” Because social concepts are tion created by the prime allows people to affectively charged, this process will activate respond to the target faster than when the both primary (I like/dislike Obama) and sec- concepts are unrelated (TREE – REPUB- ondary (I love/hate Democrats) evaluative LICAN) or when nonword foils are used as associations (Fazio et al. 1986). primes (BLUM – REPUBLICAN). Here this Implicit measures capitalize on this obser- facilitation effect measures a relatively strong vation by measuring the empirical influence association between the prime and the target of carefully chosen stimuli on speed of in long-term memory compared to a baseline, response or content of recall. Although spe- whereas an inhibition effect would be signaled cific procedures differ for the various implicit by a slower-than-baseline response time. This attitude measures, they all seek to assess the simple priming paradigm produces robust uncontrolled, unintended, stimulus-driven, effects, demonstrating the associative nature autonomous, and unconscious affective of both semantic and affective memory. responses to stimuli. For example, reac- A variant of the sequential priming para- tion time approaches rely on the speed of digm allows experimenters to assess whether response to a particular primed target as these associations require conscious median indicator of the strength of association ation. Specifically, the experimenter can between the prime and target: the stronger manipulate the time from the onset of the the association (terrorist – bad), the faster the prime word to the onset of the target word, response. Absent conscious awareness, the an interval known as the stimulus onset asyn- response bypasses intentionality and, more chrony (SOA). Conscious expectancies require important, taps beliefs and preferences that more than 300 milliseconds to develop (Neely the individual may not be willing or able to 1977; Posner, Snyder, and Davidson 1980), so consciously express. any inhibition or facilitation effects observed There are several popular implicit attitude when the SOA is shorter than 300 millisec- measures, chief among them: the sequential onds are necessarily due to automatic activation of nonconscious associations (Bargh et al. 1992). As shown in Figure 11.1,when 1 A fully axiomatized computational model, which we call John Q. Public (JQP), appears in Kim, Taber, and the SOA is short, the target is presented near Lodge (2010). the peak of the activation of the stimuli, and 160 Milton Lodge, Charles Taber, and Brad Verhulst

Short SOA

Trial Begins Prime Target

Expectation Threshold of the Target Concept

Baseline Activation of the Target Concept

Less than 250ms

Long SOA

Trial Begins Prime Target

Threshold Activation of the Target Concept

Baseline Activation of the Target Concept

More than 750ms

Activation of the Target with a Congruent Prime/Target Pair

Activation of the Target with an Irrelevant Prime/Target Pair

Activation of the Target with an Incongruent Prime/Target Pair Figure 11.1. Spreading Activation in a Sequential Priming Paradigm for Short and Long Stimulus Onset Asynchrony Conscious and Unconscious Information Processing 161 as such, it takes less time for a participant to darin ideograms), analogous to classical con- respond to subsequent related stimuli. Alter- ditioning. During the repeated pairings, the natively, when the SOA is long, the activation evaluative associations from the attitudinal of the prime has returned to near baseline lev- stimuli transfer to the neutral stimuli. When els of activation when the target stimulus is participants are asked to rate how much they presented. Thus, no facilitation is expected at like the previously neutral stimulus, their a long SOA. Contemporary priming studies affect indicates the strength and direction of routinely present primes at subliminal speeds their attitude toward the previously affective as short as 14 milliseconds, many times faster stimulus. Because the affective transfer occurs than the blink of an eye, to remove all doubt outside conscious awareness, respondents are about conscious mediation or control. not motivated to alter or misrepresent their The IAT, in contrast to the sequential attitudes toward the previously neutral stim- priming paradigm, capitalizes on response uli. In fact, participants perform no better competition rather than spreading activation. than chance at identifying which stimuli were The IAT compares the difference between paired. Tests of the AMP demonstrate con- the average response time to two blocks of vergent validity with other attitude measures categorization trials when all trials within the (Payne et al. 2005). block are either affectively congruent (col- Each procedure has unique strengths and umn 3 in Table 11.1) or affectively incon- weaknesses. One of the strengths of the IAT gruent (column 5 in Table 11.1). The other in comparison to sequential priming is its blocks of trials are used to familiarize the strong test–retest reliability (Cunningham, participants with the task. Using the racial Preacher, and Banaji 2001). Priming exper- IAT as an example, in the congruent con- iments – especially those using subliminal dition, European American stereotypes are primes – are noisy, and participants may miss paired with pleasant words and African Amer- the prime in the blink of an eye. Sequential ican stereotypes are paired with unpleasant priming, however, allows the researcher to words. Because the evaluative tendencies for assess target-level facilitation and inhibition African American stereotypes and unpleas- effects between individual pairs of primes and ant words match well-practiced behavioral targets. This is important because it allows predispositions for many Americans, partic- one to examine the strength of the associ- ipants need not inhibit competing responses ation between the speciﬁc primes and par- before reacting to the stimuli, allowing most ticular targets. The researcher can modify people to respond relatively quickly to these the list of prime/target pairs to better cap- trials. Alternatively, in the incongruent exer- ture nuanced differences between category cise, where European American stereotypes subgroups (e.g., liked vs. disliked Republican are paired with unpleasant words and African politicians). American stereotypes are paired with pleasant The primary drawback of the IAT is that words, the categories are evaluatively incon- it leads participants to overemphasize the gruent and activate competing automatic stimulus categories (i.e., race) rather than behavioral tendencies that respondents must the characteristics of individual stimuli. In override before they categorize the stimuli, one particularly revealing study, participants and thus they respond relatively slowly. were presented with pleasant and unpleas- In the ﬁnal procedure we discuss – the ant words and photos of African American AMP (Payne et al. 2005) – the mechanism athletes and European American politicians at work is spontaneous affective transfer. (Mitchell, Nosek, and Banaji 2003). When But here, rather than assessing the exist- participants categorized the photos on the ing associations between concepts in mem- basis of race, they demonstrated the typi- ory, the AMP creates new associations by cal implicit racial bias – blacks were evalu- repeatedly pairing stimuli toward which peo- ated more negatively than whites. But, when ple already have an attitude (New York City) asked to categorize by occupation (athlete vs. with previously neutral stimuli (e.g., Man- politician), the reverse results were found: 162

Table 11.1: Schematic of Racial Implicit Association Test Using Pleasant and Unpleasant Words and European American and African American Stereotype Words

Experimental Stages 12 345

European American European American Pleasant- and African American Task description and African Congruent Incongruent Unpleasant Stereotypes American Stereotypes Categorization Task Categorization Task Categorization Categorization Categorization (Reversed) • Pleasant • Pleasant • Pleasant • European American Unpleasant • European American • Unpleasant • Task instructions Unpleasant • African American •• European American • African American European American • African American ••African American DEATH •• SCIENTIST DEATH •• ATHLETIC • LAUGH SAD •• COLLEGE SAD • ASSERTIVE • COLLEGE • • LAUGH LAZY •• COLLEGE • STUPID DEATH • • KITTEN • ASSERTIVE • THOUGHTFUL COLLEGE •• AGGRESSIVE Sample stimuli GRIEF • STUPID • AGGRESSIVE •• LAZY • ATHLETIC VOMIT •• THOUGHTFUL • KITTEN THOUGHTFUL • SAD • • LOVE AGGRESSIVE • ATHLETIC •• AGGRESSIVE SCIENTIST • • JOY ATHLETIC •• SCIENTIST SCIENTIST •• KITTEN

Note: Dots beside the words indicate whether the left or right button should be pressed when the word is presented. Conscious and Unconscious Information Processing 163 participants preferred the African American approach to understanding the structure of a faces to the European American faces. person’s political attitudes. For example, the Forcing participants to make categorical central claim of the hot cognition hypothesis is judgments may activate concepts in memory that all social concepts are affectively charged that are not actually part of an individual’s (Abelson 1963; Lodge and Taber 2005). This personal attitude, but rather tap awareness of evaluative component is directly associated cultural stereotypes. To rectify this, Olson with the concept in memory, where it is and Fazio (2004) asked participants to cat- automatically activated on mere exposure to egorize stimuli in terms of “I like” or “I the concept (Fazio et al. 1986; Bargh 1999). don’t like,” rather than good or bad. Impor- This is not to say, however, that automatic tantly, this personalized version of the IAT semantic associations are less important. Both strengthens the correlation between the IAT types of associations are integral aspects of a and explicit measures of racism, although person’s attitude. In fact, using the sequen- still revealing a depressingly large proportion tial priming paradigm, the influence of each of Americans who endorse but are unwill- type of association can be directly tested, and ing to express blatantly prejudicial attitudes. the unique contributions of the semantic and Furthermore, the personalized IAT is more affective components can be assessed. strongly linked to the participant’s actual Implicit attitude measures also make it behaviors than the traditional IAT (Dovidio, possible to compare the associations between Kawakami, and Gaertner 2002; Olson and concepts that may vary across different Fazio 2004). groups of people and that may differ from Because the AMP is a relatively novel the associations identified by explicit attitude procedure, its strengths and weaknesses are measures. For example, survey evidence sug- less well identified. The procedural similarity gests that the ideological beliefs of liberals between the AMP and the sequential priming and conservatives are categorically different: paradigm suggests that the same limitations liberals and conservatives see the political may exist for both measures (Deutsch and world in different ways (Conover and Feld- Gawronski 2009). Early evidence, however, man 1984). This suggests that the structure suggests that the AMP may be more resis- of a liberal’s attitudes, and thus the associa- tant to conscious control than either the IAT tions among concepts in memory, depend on or sequential priming because even if partic- his or her political orientation. Implicit atti- ipants are motivated to obscure their explicit tude measures provide a method to test these attitudes, they cannot identify which neutral differential associations across groups. Three stimuli were paired with which attitude object studies have attempted to do just this, and (Payne et al. 2005). their results merit discussion. One of the key benefits of using implicit First, we find marked differences between rather than explicit attitude measures is that automatic facilitation and inhibition effects implicit attitudes are typically more predic- for political sophisticates and nonsophisti- tive of actual behavior (Swanson, Rudman, cates (Lodge and Taber 2005). Citizens who and Greenwald 2001; Maison, Greenwald, have repeatedly thought about and evalu- and Bruin 2004; Amodio and Devine 2006). ated political leaders, groups, and issues have This finding is underscored by the fact that stronger associations among the ideological the relationship between implicit and explicit concepts, resulting in more extreme experi- attitude measures is notoriously weak (Nosek mental facilitation and inhibition effects. Cit- 2005). As such, simply using implicit attitude izens with below average political interest measures can improve our understanding of and knowledge have weaker affective and behavior. semantic links in memory for many polit- Modified versions of implicit attitude mea- ical concepts, and therefore do not display sures can also be used to answer interesting the same pattern of facilitation and inhibi- questions that go well beyond the simple mea- tion that indicates hot cognition (McGraw surement of preferences and provide a unique and Steenbergen 1995). Thinking about an 164 Milton Lodge, Charles Taber, and Brad Verhulst attitude object, whether intentionally or These studies demonstrate that implicit unintentionally, brings the attitude object attitude measures make it possible to assess into memory and activates other associated the content of ideological preferences in a constructs. The more often two constructs way that explicit self-report measures simply are associated in memory, the stronger the cannot do. As is evident from the preceding association between them. This overall pat- discussion, implicit attitude measures can be tern suggests that because of their interest employed to test a wide variety of research in politics, sophisticates have formed affec- questions. These measures can be used as tive and semantic associations toward a broad both independent and dependent variables, range of political objects, and these feelings or they can be used to examine the cognitive and thoughts come spontaneously to mind structure of interconnected (or not) attitudes on mere (even subliminal) exposure to the and associations between concepts. concept. In a related study, Lavine et al. (2002) Unconsciously Presented or demonstrated that primes selectively facil- Unnoticed Stimuli itate different target concepts in different groups of respondents. In this study, high The final example of how unconscious pro- authoritarians responded more quickly to cessing has permeated research in political ambiguously threatening prime/target pairs science is in demonstrations of the impact of (arms-weapons), whereas low authoritari- unconscious stimuli on how people think and ans responded more quickly to semantically reason about political issues. First, let us oper- related neutral prime/target pairs (arms-legs). ationalize the distinction between conscious This pattern of differential activation sug- and unconscious stimuli. Visual stimuli take gests that authoritarians perceive concepts approximately 10 milliseconds to reach the in a categorically different way when com- objective perceptual threshold − as measured pared with nonauthoritarians. Thus, words by brain activity – and between 60 and 100 have categorically different automatic mean- milliseconds to reach the subjective thresh- ings for different groups of people. old – after which conscious processing is pos- Finally, Taber (2009) assessed the uncon- sible (Bargh and Pietromonaco 1982). If the scious associations between racially charged objective threshold is not passed, then there political issues, specifically testing whether is no registration of the stimulus and percep- attitudinal structures corresponded with tion does not occur: a nonevent. If the objec- principled conservatism or symbolic racism. tive threshold is passed but the subjective is “Affirmative action” was significantly associ- not, then we have unconscious perception. ated with African American stereotypes for In this case, a sensory experience is regis- both supporters and opponents of affirmative tered, but people are unaware of their per- action. Contrary to the expectations of prin- ception. Finally, if the subjective threshold is cipled conservatism, however, neither “indi- passed, then the stimulus event enters con- vidualism” nor “big government” was asso- scious awareness and we have the possibility ciated with the issue. In a follow-up study, of conscious perception, but only if the indi- “individualism” and “big government” were vidual attends to the stimuli. But just because associated with affirmative action only when a stimulus passes the subjective threshold white conservatives were explicitly asked to does not guarantee that it will be processed think about the issue when preparing for consciously. a debate with an affirmative action sup- Accordingly, a stimulus may be processed porter, and not when they expected a like- unconsciously under two conditions. First, an minded conservative. Such findings suggest objectively perceived stimulus may not reach that principled conservatism allows affirma- conscious awareness because it occurred too tive action opponents to rationalize their rapidly or peripherally to be noticed, thus opposition to affirmative action in ideological necessitating unconscious processing. Alter- terms. natively, and this is the most common reason Conscious and Unconscious Information Processing 165 for a stimulus being processed unconsciously ity toward the experimenter. The participants in the real world, the individual “sees” the were completely unaware that their behaviors stimulus but fails to recognize its influence on were driven by unconscious motives. thoughts, feelings, preferences, and choices. Extending the notion of unnoticed prim- The event influences behavior, but its impact ing into the political arena, Mendelberg remains unappreciated. In both cases, the (2001) demonstrated that exposure to racially stimulus influences information processing charged advertising propelled racially resent- outside of awareness. ful people to hold more prejudicial beliefs. The distinction between conscious and Specifically, in the 1988 presidential election, unconscious perception should not be seen George H. W. Bush ran an attack ad against as either/or but as interdependent. Our tri- his opponent Michael Dukakis that featured partite distinction implies an inherent order- an African American prisoner named Willie ing because all stimuli that pass the subjective Horton. Horton was released from prison on threshold necessarily pass the objective a weekend furlough, where he escaped from threshold. Thus, it is important to keep police custody, raped a woman, and assaulted in mind that unconscious processes are her husband. The ad attacked Dukakis’ poli- omnipresent and inevitably influence subse- cies on crime; however, it was the racial quent thoughts, feelings, and behaviors. As undertones of the ad that drove racial resent- such, all conscious thoughts and feelings have ment and led to the increase in negative an unconscious origin. Consequently, some- evaluations of Dukakis (also see Kinder and times we just do things without thinking, and Sanders 1996). Importantly, when the racial sometimes thoughts just seem to pop into component of the ad was made explicit to mind. viewers, the negative effect on evaluations Unconscious stimuli are ubiquitous in the evaporated. Thus, insofar as people were not real world: the playthings of advertisers who consciously aware of the racial component of use beautiful women to peddle cigarettes and the ad, the ad was effective. sports cars, where the American flag and Thinking carefully, however, does not upbeat music provide the backdrop for presi- inevitably negate the impact of unconsciously dential candidates, or where discordant music perceived or unnoticed influences. In a series sets the tone for political attack ads. Whether of experiments, Wilson et al. (1995) and Wil- consciously unnoticed or simply unappreci- son and Schooler (1991) found that when par- ated, such “incidental” stimuli commonly and ticipants are encouraged to stop and think powerfully influence political beliefs and atti- before making a choice, they overanalyze tudes, in part because they are so easy to available information, bring to mind less manipulate. important considerations, and, consequently, A good example of how unconscious stim- make poorer choices than when making a uli influence conscious thought and behavior snap judgment. Current research also sug- is a now classic set of experiments conducted gests that as people increasingly engage in by Bargh et al. (1996). They primed vari- effortful deliberation, the quality of the deci- ous concepts outside the participants’ aware- sion deteriorates in both objective terms (peo- ness and recorded their subsequent behaviors. ple make the wrong choice) and in subjective Remarkably, after being primed with the terms (higher levels of postdecision regret; concept “elderly,” participants walked more Dijksterhuis 2004). In addition to this coun- slowly to the elevator after completing the terintuitive finding, it appears that people experiment. When participants were primed make the best choices if they are distracted with words related to rudeness or polite- from engaging in conscious thought or ness, they were, correspondingly, more or less discouraged from thinking of the reasons for likely to interrupt an experimenter engaged their preferences. in an extended conversation. When primed Along these lines, Verhulst, Lodge, and with African American (vs. European Ameri- Taber (2007) explored the influence of sub- can) faces, participants expressed more hostil- liminal primes on evaluations of political 166 Milton Lodge, Charles Taber, and Brad Verhulst candidates when participants are given sub- function of where people voted – whether in stantive issue information about the can- schools, churches, or firehouses − with voters didate’s preferences. Commonly, people more likely to favor raising state taxes to sup- believe that the effects of subliminal primes port education if voting in schools. This effect are fleeting and that they would be over- held even after controlling for their political whelmed by actual, substantive information. views. Clearly, the voters knew what build- After all, candidate-voter issue proximity is ing they were in but were, in all likelihood, among the most robust predictors of candi- not consciously aware of its influence on their date evaluations and vote choice (Black 1948; behavior. Berelson, Lazarsfeld, and McPhee 1954; Another major area of research pointing Downs 1957). Interestingly, we found that to robust effects of unappreciated influences subliminal primes influenced candidate eval- on judgment is the effect of facial attractive- uations for sophisticated participants: posi- ness on evaluations, attitudes, and behavior. tive primes led sophisticates to like the can- “Beautiful is good” stereotyping is alive in didate more, and negative primes less. In a politics, where attractive candidates are seen follow-up study, we randomly assigned par- as possessing more integrity, competence, ticipants to think carefully about the candi- likeableness, and fitness for public office, even dates. Rather than reducing the impact of the though people deny being affected by the can- subliminal primes, careful, conscious deliber- didates’ appearance (Rosenberg and McCaf- ation increased the impact of the primes on ferty 1987; Verhulst, Lodge, and Lavine candidate evaluations, even though partici- 2010). Three large meta-analyses covering pants were unaware of being primed. Again, more than 1,000 peer-reviewed psychologi- the effect appeared among the most sophisti- cal studies of physical attractiveness confirm cated participants, confirming the theoretical significant experimental and correlational hypothesis that only knowledgeable partici- effects on a broad range of social attitudes pants have the density of semantic and affec- and behaviors (Eagly et al. 1991; Feingold tive associations necessary for implicit primes 1992; Langlois et al. 2000). Typically, physi- to influence the quality of thought. Although cal attractiveness is noticed, but people con- more research in this area is needed, it sistently fail to appreciate its impact on now appears that the unconscious informa- evaluations and behavior; however, the mag- tion can drive conscious deliberation, cul- nitude of these effects is roughly the same as minating in preferences that are strongly other variables in the social sciences (Eagly influenced by information participants never et al. 1991). consciously perceived. These studies and many more demon- Erisen et al. (2007) addressed this issue strate the influence of unappreciated infor- from a slightly different angle. Specifically, mation on perception, social judgments, they encouraged participants to stop and and behavior. The take-home point here think about political issues while being sub- is that our thoughts, attitudes, and behav- liminally primed with smiling, frowning, or iors extend deep into our unconsciousness. neutral cartoon faces. In this set of studies, Similar noticed but unappreciated effects are the subliminal primes biased subsequent cog- undoubtedly ubiquitous in everyday life out- nitive deliberation. Participants primed with side the laboratory (Bargh 1997), but exper- positive images listed more positive thoughts, imental methods are crucial to tease out the whereas those primed with negative images effects. list more negative thoughts. Again, participants were completely unaware of the primes, yet this biased set of prime-induced thoughts 3. Conclusion: The Unconscious led to more extreme attitudes. Mind in a Conscious World In one particularly telling example, Berger, Meredith, and Wheeler (2008) showed that People have known for many years budgetary support for education varied as a that unconsciously perceived thoughts and Conscious and Unconscious Information Processing 167 feelings influence behavior. Recent theoret- Personality, eds. Silvan Tomkins and Samuel ical advances and experimental procedures Messick. New York: Wiley, 227–98. allow social scientists to measure and behav- Amodio, David M., and Patricia G. Devine. 2006. iorally validate the effects of unconscious pro- “Stereotypes and Prejudice: Their Automatic and Controlled Components.” Journal of Per- cessing and implicit attitudes on complex 91 652 61 behaviors. Implicit attitude measures are now sonality and Social Psychology : – . Anderson, John R. 1983. The Architecture of Cog- at the forefront of contemporary psychology nition. Cambridge, MA: Harvard University and marketing research and are working their Press. way into political science. This research offers Anderson, John R. 1993. Rules of the Mind. Hills- the potential to explore more deeply the psy- dale, NJ: Erlbaum. chological mechanisms that lead citizens to Anderson, Norman H., and Alfred A. Barrios. form all sorts of political attitudes and exam- 1961. “Primacy Effects in Person Impression ine the way these political attitudes influence Formation.” Journal of Abnormal Social Psychol- various political behaviors. ogy 43: 346–50. 1997 The attitude-behavior connection in most Bargh, John A. . “The Automaticity of Every- political science research is routinely made day Life.” In Advances in Social Cognition,Volume by inference and assumption. Political scien- X: The Automaticity of Everyday Life, Advances in Social Cognition Series, ed. Robert S. Wyer, tists are interested in behavioral variables – Jr. London: Psychology Press, 1–62. voting, petitioning, attending rallies, con- Bargh, John A. 1999. “The Cognitive Monster: tributing to campaigns – but our variables The Case against Controllability of Automatic are most often verbalized intentions or rec- Stereotype Effects.” In Dual-Process Theories in ollections and not observed behaviors. This Social Psychology, eds. Shelley Chaiken and Yaa- accentuates the potential for social desirabil- cov Trope. New York: Guilford Press, 361–82. ity to influence responses. Moreover, citizens Bargh, John A., Shelley Chaiken, Rajen Govender, can only verbalize the thoughts and behav- and Felicia Pratto. 1992. “The Generality of the ioral intentions of which they are aware. The Automatic Attitude Activation Effect.” Jour- nal of Personality and Social Psychology 62: 893– reconceptualization of political information 912 processing that we propose challenges our . Bargh, John A., Mark Chen, and Lara Burrows. discipline’s reigning assumption that politi- 1996. “Automaticity of Social Behavior: Direct cal beliefs, attitudes, and behaviors are rooted Effects of Trait Construct and Stereotype Acti- in conscious considerations; it also questions vation on Action.” Journal of Personality and our disciplines’ reliance on survey research Social Psychology 71: 230–44. and promises better explanations of politi- Bargh, John A., and Paula Pietromonaco. 1982. cal behavior. We see strong theoretical and “Automatic Information Processing and Social growing empirical reasons to believe that Perception: The Influence of Trait Information incorporating unconscious processing into Presented Outside of Conscious Awareness on our models of political attitudes and behavior Impression Formation.” Journal of Personality 43 437 49 will result in a more complete understanding and Social Psychology : – . of the relationships among and between polit- Berelson, Bernard, Paul Lazarsfeld, and William McPhee. 1954. Voting: A Study of Opinion For- ical beliefs, preferences, choices, and conse- mation in a Presidential Campaign. Chicago: The quential political behaviors. University of Chicago Press. Berger, Jonah, Marc Meredith, and S. Christian Wheeler. 2008. “Contextual Priming: Where References People Vote Affects How They Vote.” Pro- ceedings of the National Academy of Sciences 105: Aarts, Henk, and Ap Dijksterhuis. 2000. “Habits 8846–9. as Knowledge Structures: Automaticity in Betsch, Tilman, Henning Plessner, Christine Goal-Directed Behavior.” Journal of Personal- Schwieren, and Robert Gutig. 2001. “I Like ity and Social Psychology 78: 53–63. It but I Don’t Know Why: A Value Account Abelson, Robert. 1963. “Computer Simulation of Approach to Implied Attitude Formation.” Per- ‘Hot’ Cognition.” In Computer Simulation of sonality and Social Psychology Bulletin 27: 242–53. 168 Milton Lodge, Charles Taber, and Brad Verhulst

Black, Duncan. 1948. “On the Rationale of Group tiveness Stereotype.” Psychological Bulletin 110: Decision Making.” Journal of Political Economy 109–29. 56: 23–34. Erisen, Cengiz, Milton Lodge, and Charles Taber. Cassino, Dan, and Milton Lodge. 2007.“ThePri- 2007. “The Role of Affect in Political Deliber- macy of Affect in Political Evaluations.” In The ation.” Paper presented at the annual meeting Affect Effect: Dynamics of Emotion in Political of the American Political Science Association, Thinking and Behavior, eds. W. Russell New- Chicago. man, George E. Marcus, Ann N. Crigler, and Fazio, Russell, David Sanbonmatsu, Martha Pow- Michael MacKuen. Chicago: The University of ell, and Frank Kardes. 1986. “On the Automatic Chicago Press, 101–23. Activation of Attitudes.” Journal of Personality Chaiken, Shelly, and Durairaj Maheswaran. 1994. and Social Psychology 50: 229–38. “Heuristic Processing Can Bias Systematic Feingold, Alan. 1992. “Good-looking People Are Processing: Effects of Source Credibility, Not What We Think.” Psychological Bulletin Argument Ambiguity, and Task Importance on 111: 304-41. Attitude Judgment.” Journal of Personality and Gawronski, Bertram, and Galen V. Bodenhausen. Social Psychology 66: 460–73. 2006. “Associative and Propositional Pro- Chaiken, Shelley, and Yaacov Trope. 1999. Dual- cesses in Evaluation: An Integrative Review of Process Theories in Social Psychology. New York: Implicit and Explicit Attitude Change.” Psycho- Guilford Press. logical Bulletin 132: 692–731. Collins, Allan M., and Elizabeth F. Loftus. 1975. Greenwald, Anthony G., Debbie E. McGhee, and “A Spreading-Activation Theory of Seman- Jordan L. K. Schwartz. 1998. “Measuring Indi- tic Processing.” Psychological Review 82: 407– vidual Differences in Implicit Cognition: The 28. Implicit Association Test.” Journal of Personality Collins, Allan M., and M. Ross Quillian. 1969. and Social Psychology 74: 1464–80. “Retrieval Time from Semantic Memory.” Hastie, Reid, and Bernadette Park. 1986.“The Journal of Verbal Learning and Verbal Behavior Relationship between Memory and Judgment 8: 240–47. Depends on Whether the Judgment Task Conover, Pamela J., and Stanley Feldman. 1984. Is Memory-Based or On-Line.” Psychological “How People Organize the Political World: A Review 93: 258–68. Schematic Model.” American Journal of Political Ito, Tiffany, and John Cacioppo. 2005. “Attitudes Science 28: 95–126. as Mental States of Readiness: Using Physio- Cunningham, William, Kristopher Preacher, and logical Measures to Study Implicit Attitudes.” Mahzarin Banaji. 2001. “Implicit Attitude Mea- In Implicit Measures of Attitudes, eds. Bernd Wit- sures: Consistency, Stability, and Convergent tenbrink and Norbert Schwartz. New York: Validity.” Psychological Science 1: 163–70. Guilford Press, 125–58. Deutsch, Roland, and Bertram Gawronski. 2009. Kim, Sung-youn, Charles Taber, and Milton “When the Method Makes a Difference: Lodge. 2009. “A Computational Model of the Antagonistic Effects on ‘Automatic Evaluations Citizen as Motivated Reasoner: Modeling the as a Function of Task Characteristics of the Dynamics of the 2000 Presidential Election.” Measure.’ ” Journal of Experimental Social Psy- Political Behavior 32: 1–28. chology 45: 101–14. Kinder,DonaldR.,andLynnM.Sanders.1996. Dijksterhuis, Ap. 2004. “Think Different: The Divided by Color: Racial Politics and Democratic Merits of Unconscious Thought in Preference Ideals. Chicago: The University of Chicago Development and Decision Making.” Journal Press. of Personality and Social Psychology 87: 586–98. Langlios, Judith, Lisa Kalakanis, Adam Ruben- Dovidio, John, Kerry Kawakami, and Samuel stein, Andrea Larson, Monica Hallam, and Gaertner. 2002. “Implicit and Explicit Preju- Monica Smoot. 2000. “Maxims or Myths dice and Interracial Interaction.” Journal of Per- of Beauty? A Meta-Analytic and Theoretical sonality and Social Psychology 82: 62–8. Review.” Psychological Bulletin 126: 390–423. Downs, Anthony. 1957. An Economic Theory of Lau, Richard, and David Redlawsk. 2001. “Advan- Democracy. New York: Addison-Wesley. tages and Disadvantages of Cognitive Heuris- Eagly, Alice H., Richard D. Ashmore, Mona G. tics in Political Decision Making.” American Makhijani, and Laura C. Longo. 1991. “What Journal of Political Science 45: 951–71. Is Beautiful Is Good, but . . . : A Meta-Analytic Lavine, Howard, Milton Lodge, Jamie Polichak, Review of Research on the Physical Attrac- and Charles Taber. 2002. “Explicating the Conscious and Unconscious Information Processing 169

Black Box through Experimentation: Studies of Nosek, Brian A. 2005. “Moderators of the Rela- Authoritarianism and Threat.” Political Analysis tionship between Implicit and Explicit Evalua- 10: 342–60. tion.” Journal of Experimental Psychology: General Lodge, Milton, and Charles Taber. 2005.“The 134: 565–84. Automaticity of Affect for Political Leaders, Olson, Michael, and Russell Fazio. 2004. “Reduc- Groups, and Issues: An Experimental Test of ing the Inﬂuence of Extrapersonal Associa- the Hot Cognition Hypothesis.” Political Psy- tions on the Implicit Association Test.” Jour- chology 26: 455–82. nal of Personality and Social Psychology 86: 654– Lodge, Milton, Marco Steenbergen, and Shawn 67. Brau. 1995. “The Responsive Voter: Campaign Payne, Keith, Clara Cheng, Olesya Govorun, Information and the Dynamics of Candidate and Brandon Stewart. 2005. “An Inkblot for Evaluation.” American Political Science Review Attitudes: Affect Misattribution as Implicit 89: 309–26. Measurement.” Journal of Personality and Social Lupia, Arthur. 1994. “Shortcuts versus Encyclope- Psychology 89: 277–93. dias: Information and Voting Behavior in Cal- Posner, Michael, Charles Snyder, and Brian ifornia Insurance Reform Elections.” American Davidson. 1980. “Facilitation and Inhibition in Political Science Review 88: 63–76. the Processing of Signals.” Journal of Experi- Maison, Dominika, Anthony G. Greenwald, and mental Psychology: General 109: 160–74. Ralph H. Bruin. 2004. “Predictive Validity Rahn, Wendy, Jon Krosnick, and Marijke Breun- of the Implicit Association Test in Studies ing. 1994. “Rationalization and Derivation of Brands, Consumer Attitudes, and Behav- Processes in Survey Studies of Candidate Eval- ior.” Journal of Consumer Psychology 14: 405– uation.” American Journal of Political Science 38: 15. 582–600. McGraw, Kathleen, and Marco Steenbergen. Redlawsk, David P. 2001. “You Must Remember 1995. “Pictures in the Head: Memory Repre- This: A Test of the On-Line Model of Voting.” sentations of Political Candidates.” In Politi- Journal of Politics 63: 29–58. cal Judgment: Structure and Process, eds. Mil- Rosenberg, Shawn, and Patrick McCafferty. 1987. ton Lodge and Kathleen McGraw. Ann “The Image and the Vote: Manipulating Arbor: University of Michigan Press, 15– Voter’s Preferences.” Public Opinion Quarterly 42. 51: 31–47. Mendelberg, Tali. 2001. The Race Card: Campaign Steenbergen, Marco, and Milton Lodge. 2003. Strategy, Implicit Messages and the Norm of Equal- “Process Matters: Cognitive Models of Can- ity. Princeton, NJ: Princeton University Press. didate Evaluation.” In Electoral Democracy, eds. Miller, George. 1957. “The Magical Number Michael MacKuen and George Rabinowitz. Seven, Plus or Minus Two: Some Limits on Ann Arbor: University of Michigan Press, Our Capacity for Processing Information.” Psy- 125–71. chological Review 63: 81–97. Swanson, Jane E., Laurie Rudman, and Anthony Mitchell, Jason, Brian Nosek, and Mahzarin G. Greenwald. 2001. “Using the Implicit Asso- Banaji. 2003. “Contextual Variations in ciation Test to Investigate Attitude-Behaviour Implicit Evaluation.” Journal of Experimental Consistency for Stigmatised Behavior.” Cogni- Psychology: General 132: 455–69. tion & Emotion 15: 207–30. Monroe, Brian M., and Stephen J. Read. 2008. Taber, Charles. 2009. “Principles of Color: “A General Connectionist Model of Attitude Implicit Race, Ideology, and Opposition to Structure and Change: The ACS (Attitudes as Race-Conscious Policies.” Unpublished paper, Constraint Satisfaction) Model.” Psychological Stony Brook University. Review 115: 733–59. Verhulst, Brad, Milton Lodge, and Howard Neely, James. 1977. “Semantic Priming and Lavine. 2010. “The Attractiveness Halo: Why Retrieval from Lexical Memory: Roles of Inhi- Some Candidates Are Perceived More Favor- bitionless Spreading Activation and Limited ably Than Others.” Journal of Nonverbal Behav- Capacity Attention.” Journal of Experimental ior 34: 111–17. Psychology: General 106: 226–54. Verhulst, Brad, Milton Lodge, and Charles Taber. Norretranders, Tor. 1998. The User Illusion: Cut- 2007. “Automatic Projection: How Incidental ting Consciousness Down to Size. New York: Pen- Affect Alters the Perceptions of Political Candi- guin Press. dates.” Paper presented at the annual meeting 170 Milton Lodge, Charles Taber, and Brad Verhulst

of the Midwest Political Science Association, sions.” Journal of Personality and Social Psychology Chicago. 60: 181–92. Wilson, Timothy D., Sarah D. Hodges, and S. J. Wittenbrink, Bernd. 2007. “Measuring Attitudes LaFleur. 1995. “Effects of Introspecting about through Priming.” In Implicit Measures of Atti- Reasons: Inferring Attitudes from Accessible tudes, eds. Bernd Wittenbrink and Norbert Thoughts.” Journal of Personality and Social Psy- Schwartz. New York: Guilford Press, 17–58. chology 69: 16–28. Zajonc, Robert. 1980. “Feeling and Thinking: Wilson, Timothy D., Samuel Lindsey, and Tonya Preferences Need No Inferences.” American Y. Schooler. 2000. “A Model of Dual Atti- Psychologist 35: 117–23. tudes.” Psychological Review 107: 101–26. Zaller John, and Stanley Feldman. 1992. “A Simple Wilson, Timothy D., and Jonathan W. Schooler. Theory of Survey Response: Answering Ques- 1991. “Thinking Too Much: Introspection Can tions versus Revealing Preferences.” American Reduce the Quality of Preferences and Deci- Journal of Political Science 35: 579–616. CHAPTER 12

Political Knowledge

Cheryl Boudreau and Arthur Lupia

In political surveys, many citizens fail to political knowledge questions include “What answer, or provide incorrect answers to, fact- is the political office held by [name of cur- based questions about political figures and rent vice president, British prime minister, institutions. A common inference drawn from or chief justice of the United States]?” and such failures is that citizens’ poor perfor- “Which political party has the most seats in mance on surveys reflects their incompetence the U.S. House of Representatives?” Many in democratically meaningful contexts such as people have used responses to survey-based voting booths. political knowledge questions to criticize the The scholarly home for such findings is public for its general incompetence. the academic literature on political knowl- In recent years, these criticisms have come edge. A common analytic definition of polit- under increasing scrutiny (e.g., Graber 1984; ical knowledge is that it is a measure of a Popkin 1994). Some scholars raised ques- citizen’s ability to provide correct answers to tions about the practice of basing broad a specific set of fact-based questions.1 Typical

1 The Oxford English Dictionary defines “knowledge” as questions can provide evidence of a citizen’s knowl- “(i) expertise, and skills acquired by a person through edge of the specific facts mentioned in specific questions, experience or education; the theoretical or practical the extent to which such responses provide reliable evidence of broader kinds of political knowledge is understanding of a subject; (ii) what is known in a 2006 particular field or in total; facts and information; or typically limited (Lupia ). These limitations are (iii) awareness or familiarity gained by experience of often due to the narrow focus and small number of a fact or situation.” In this definition, information such questions that appear in any given survey. That is required for knowledge, but it is not the same as said, given that the literature is widely known as refer- knowledge. We (the authors) regard what is measured ring to all such data as evidence of “political knowl- by responses to the survey questions referenced in edge” and given that we want to focus our space in this this chapter as “political information.” Whether such handbook on how experimental work has advanced responses constitute a “political knowledge” measure that very literature, we follow the flawed naming con- first requires that we ask the question, “Knowledge vention to minimize confusion. of what?” Scholars working in this area rarely ask We thank Adam Seth Levine, Yanna Krupnikov, James this question and simply confound information and Kuklinski, Nicholas Valentino, and John Bullock for knowledge. Although answers to fact-based survey helpful comments.

171 172 Cheryl Boudreau and Arthur Lupia generalizations of citizen competence or show that what citizens actually know about knowledge on a relatively small set of idiosyn- politics, and how such knowledge affects their cratic, fact-based survey questions (e.g., choices, is very different than the conven- Lupia 2006). Others uncovered logical and tional wisdom alleges. factual errors in claims about the kinds of In this chapter, we report on two kinds political knowledge that are needed to make of experiments that clarify what voters know important political choices competently (e.g., and why it matters. In Section 1, we add- Lupia and McCubbins 1998; Gibson and ress the question, “What is political know- Caldeira 2009). ledge?” by describing experiments that A common theme in this new research is manipulate the survey context from which that many critiques of the public are based on most political knowledge measures are vague or erroneous assumptions about a key derived. These experiments reveal that exist- relationship – the relationship between how ing knowledge measures are significantly survey respondents answer political knowl- affected by question wording, variations in edge questions and these same respondents’ respondents’ incentives to think before they abilities to accomplish politically relevant answer, whether respondents feel threatened tasks (by which we mean the ability to make by unusual aspects of survey interview con- the choice one would have made if knowl- texts, and personality variations that make edgeable about a set of relevant facts). Indeed, some respondents unwilling to give correct many scholars simply presumed that survey- answers to survey interviewers even when based political knowledge measures can be they know the answers. treated as valid representations of citizens’ In Section 2, we describe experiments that general knowledge of politics. They also pre- clarify the relationship between what citizens sumed that not offering correct responses to know and their competence. These experi- these survey questions signaled incompetence ments compare the choices that people make at politically relevant tasks such as voting. given different kinds of information. They Given how often this literature criticized cit- show when people can (and cannot) make izens for what they did not know, it is ironic competent decisions despite lacking answers that its authors gave so little thought to the to fact-based political knowledge questions. conditions under which these presumptions Collectively, these clarifications provide a were true. different view of citizen competence than Experimental political science has added is found in most nonexperimental work on clarity, precision, and new insight into such political knowledge. There are many cases for matters. Unlike nonexperimental studies, which not knowing the answers to survey- experiments allow scholars to 1) randomly based political knowledge questions reveals assign subjects to treatment and control very little about citizens’ competence. groups; 2) systematically manipulate relevant aspects of survey interview and decision- making environments; and 3) directly observe 1. Experiments on Properties of the answers subjects give (and the choices Survey-Based Knowledge Measures they make) under different conditions. These features of experiments have enabled scholars Many citizens do not correctly answer politi- to clarify how particular aspects of the sur- cal knowledge questions on traditional survey production process contribute to citizens’ veys. This claim is not controversial. Less poor performance on survey-based political clear is what these questions tell us about cit- knowledge questions. Experiments have also izens’ knowledge more generally. clarified the relationship between the ability Nonexperimental studies have attempted to answer certain questions and the ability to to show that responses to such questions con- make important decisions competently. To stitute valid measures of political knowledge be sure, there are many things about politics in two ways. One way is to correlate answers that citizens do not know. But experiments to fact-based questions with factors such as Political Knowledge 173 interest in politics and turnout. The under- tion, from the American National Election lying assumption is that because people who Studies (ANES), is as follows: turn out to vote, or who are interested in poli- Now we have a set of questions concerning tics, are also more likely to answer the survey various public figures. We want to see how questions correctly, the questions are valid much information about them gets out to the measures of knowledge. This approach is public from television, newspapers and problematic because these underlying factors the like. . . . What about . . . William Rehn- are not themselves credible measures of polit- quist – What job or political office does he ical knowledge. For example, a person can be NOW hold? interested in politics without being knowl- From the 1980s through 2004,theANES edgeable and can be knowledgeable without hired coders to code transcribed versions being particularly interested. of respondents’ verbatim answers as simply Another attempted means of validating “correct” or “incorrect.” The codes, and not survey responses as knowledge measures is the original responses, were included in the to use factor analysis. Here, the claim is public ANES dataset. For decades, scholars that if the same kinds of people answer the treated the codes as valid measures of respon- same kinds of questions correctly, then the dents’ knowledge. Although many analysts questions must be effectively tapping a more used these data to proclaim voter ignorance general knowledge domain. The error in and incompetence, an irony is that almost no this claim can be seen by recalling previ- scholar questioned whether the coding pro- ous critiques of the use of factor analysis cedure itself was valid. in intelligence estimation. As Gould (1996) Gibson and Caldeira (2009) changed that. describes, “The key error of factor analy- They raised important questions about which sis lies in reification, or the conversion of verbatim responses should be counted as cor- abstractions into putative real entities” (48). rect. In 2004, for example, William Rehnquist In other words, factor analysis yields mean- was Chief Justice of the United States. Upon ingful results only if the selection of questions inspecting transcribed versions of ANES themselves is derived from a credible theory responses, Gibson and Caldeira found many of information and choice. Lupia (2006, 224) problems with the coding. For example, the finds that the selection of specific questions is ANES counted as correct only responses almost entirely subjective, reflecting the often that included “Chief Justice” and “Supreme idiosyncratic tastes of the authors involved. Court.” Respondents who said that Rehn- Hence, modern factor analytic claims are of quist is on the Supreme Court without includ- little relevance to the question of whether ing chief justice or respondents who correctly survey-based political knowledge questions said that he was a federal judge were coded as capture citizens’ true knowledge about incorrect. politics. Gibson and Caldeira’s (2009) examina- Gibson and Caldeira (2009) offer an exper- tion of the transcripts often revealed substan- iment that questions the validity of existing tial respondent knowledge in answers that data and also provides a more effective means had been coded as incorrect. They argued of measuring what citizens know. They argue that past practices likely produced “a seri- that “much of what we know – or think we ous and substantial underestimation of the know – about public knowledge of law and extent to which ordinary people know about courts is based upon flawed measures and pro- the nation’s highest court” (429). cedures” (429). To assess the extent of this underesti- In particular, many of the most famous mation, they embedded an experiment in a survey-based political knowledge measures nationally drawn telephone survey of 259 come from open-ended recall questions (i.e., respondents. Respondents were asked to questions where respondents answer in their identify the current or most recent political own words rather than choosing from a small office held by William Rehnquist, John G. set of answers). An example of such a ques- Roberts, and Bill Frist. A control group was 174 Cheryl Boudreau and Arthur Lupia asked these questions in the traditional (open- complete interviews quickly. Respondents often ended) format. A treatment group was asked do not want to prolong the interview. Ques- to identify the same individuals in a multiple- tions are asked, and answers are expected, in choice format. quick succession. Moreover, political knowl- With respect to Chief Justice Rehnquist, edge questions typically appear in the survey and using the traditional ANES method of with no advance notice. And the typical sur- scoring open-ended responses as correct or vey provides no incentive for respondents to incorrect, twelve percent of respondents cor- answer the questions correctly. rectly identified Rehnquist as Chief Justice. This “pop quiz” atmosphere is very dif- Another thirty percent identified him as a ferent from circumstances in which having Supreme Court justice; however, because particular kinds of political knowledge mat- these responses did not explicitly refer to him ters most, such as elections. Election dates as Chief Justice, the ANES measure would are typically known in advance. Hence, peo- have counted these responses as incorrect. ple who want to become informed have an The treatment group, in contrast, was asked opportunity to do so before they cast a vote. to state whether Rehnquist, Lewis F. Powell, To determine the extent to which odd sur- or Byron R. White was Chief Justice (with vey interview attributes contribute to poor the order of the response options random- performance on political knowledge quizzes, ized across respondents). When asked the Prior and Lupia (2008) assigned more than question in this format, seventy-one percent 1,200 randomly selected members of Knowl- correctly selected Rehnquist. Gibson and edge Networks’ national Internet panel to Caldeira (2009) observed comparable results one of four experimental groups. The confor Bill Frist and John G. Roberts. trol group was asked fourteen political knowl- The substantive impact of Gibson and edge questions in a typical survey interview Caldeira’s (2009) findings is that “the Amer- environment, with little time to answer the ican people know orders of magnitude more questions (sixty seconds from the moment about their Supreme Court than most other that the question first appeared on screen) studies have documented” (430). The broader and no motivation to answer correctly. Treat- methodological implication is that the combi- ment groups received greater opportunity nation of open-ended questions, the ANES’s and/or incentive to engage the questions. One coding scheme, and a lack of fact-checking treatment group was offered $1 for every by critics of citizen knowledge who used the question answered correctly. Another group ANES data contributed to an overly negative was offered twenty-four hours to respond to image of the public. Gibson and Caldeira’s the fourteen questions. The third treatment work subsequently caused the ANES to group was offered time and money. restructure how it solicits and codes politi- Even though Prior and Lupia’s (2008) cal knowledge (Krosnick et al. 2008). Hence, questions were selected to be quite dif- in this case, an experimental design not ficult, each treatment produced a signifi- only influenced our understanding of polit- cant increase in questions answered correctly. ical knowledge, but also improved how the Compared to the control group, simply offer- concept is now measured. ing $1 for correct answers increased the aver- Other experiments on survey interview age number of correct answers by eleven attributes suggest further trouble for con- percent. Offering extra time produced an ventional interpretations of traditional polit- eighteen-percent increase over the control ical knowledge measures. Prior and Lupia group. Time and money together increased (2008) ask whether “seemingly arbitrary fea- the average number of questions answered tures of survey interviews” affect the valid- correctly by twenty-four percent relative to ity of knowledge measures (169). They con- the control group. tend that the typical survey-based political The effect of money alone is noteworthy knowledge assessment occurs in an unusual because the only difference between the con- circumstance. Interviewers have incentives to trol and treatment groups is that the latter is Political Knowledge 175 paid a small amount for each correct answer. Questions were drawn from sources such as Treatment group respondents did not have the ANES. Stereotype threat was manipu- time to look up correct answers. Hence, the lated in two ways. The first manipulation was treatment group’s performance gain indicates interviewer gender. Respondents were ran- that low motivation makes people appear less domly assigned to be interviewed by men or knowledgeable than they actually are. women. The second manipulation pertained Looking at experimental effects across to the “threat cue.” Treatment groups were population groups reinforces the conclusion. told that “the survey you are participating in The largest effects are on respondents who this evening has been shown to produce gen- report that they follow politics “some of the der differences in previous research.” Control time” (rather than “most of the time” or groups were not told of any gender differ- “not at all”). For them, simply paying $1 ences. yields a thirty-two-percent increase in correct They found that men scored higher than answers relative to members of the control women overall, which is consistent with non- group who report following politics “some of experimental findings. However, the exper- the time.” Hence, for people whose attention imental variations affected the inequality. to politics is infrequent, the typical survey When there was no threat cue, or when interview context provides insufficient moti- women were interviewed by other women, vation for searching the true content of their there was no significant difference between memories. This finding implies that “conven- men’s and women’s scores. When the threat tional knowledge measures confound respon- cue or male interviewers were introduced, dents’ recall of political facts with variation in the asymmetry emerged – but neither factor their motivation to exert effort during sur- affected men’s scores. The effects were confined vey interviews [and, hence,] provide unreli- to women and decreased their scores. This able assessments of what many citizens know experiment suggests important limits on the when they make political decisions” (Prior extent to which survey-based political knowl- and Lupia 2008, 169). edge tests can be considered valid measures of Other experiments examine how social what the population as a whole knows about roles and survey contexts interact to affect politics. respondent performance. McClone, Aron- Other experiments show that whether sur- son, and Kobrynowicz (2006) noted that men vey questions allow or encourage “don’t tend to score better than women on survey- know” responses affects political knowledge based political knowledge tests. Conventional measures. These experiments suggest that explanations for this asymmetry included “don’t know” options may cause scholars to the notion that men are more interested in underestimate what the public knows about politics. politics. The reason is that some people are McClone et al. (2006) saw “stereotype less likely than others to offer answers when threat” (Steele and Aronson 1995)asan they are uncertain. alternative explanation. In a typical stereo- For example, Jeffrey Mondak and his col- type threat experiment, members of stig- leagues (Mondak 2001; Mondak and Davis matized and nonstigmatized groups are 2001; Mondak and Anderson 2004) designed randomly assigned to experimental groups. several split ballot experiments (i.e., random A treatment group is given a test along with assignment of survey respondents to experi- a cue suggesting that members of their stig- mental conditions) in two surveys, the 1998 matized group have not performed well in ANES Pilot Study and a Tallahassee-based the past. The control group receives the test survey. In each survey, the control group without the cue. received knowledge questions that began with McClone et al.’s (2006) phone-based a “don’t know”–inducing prompt, “Do you experiment was conducted on 141 under- happen to know . . . ,” and interviewers were graduates, 70 men and 71 women. A ten- instructed not to probe further after an initial question index measured political knowledge. “don’t know” response. Treatment groups 176 Cheryl Boudreau and Arthur Lupia received questions with identical substan- Miller and Orr (2008) found that discour- tive content, but different implementation. In aging “don’t know” (rather than encourag- both surveys, treatment groups heard a guessing it) led to a substantial drop in the use inducing phrase such as “even if you’re not of the “don’t know” option. They also found sure I’d like you to tell me your best guess.” In that discouraging “don’t know” (rather than the ANES version, moreover, the interviewer encouraging it) corresponded to an increase first recorded any “don’t know” responses and in the average percentage of correct answers then probed further for substantive answers given per respondent. to determine whether respondents who ini- The most interesting thing about the tially responded “don’t know” actually knew comparison between the “don’t know”– about the concept in question. encouraged and “don’t know”–omitted groups In each experiment, respondents were sig- is that the increase in percentage correct nificantly less likely to choose “don’t know” (from sixty-one percent to seventy percent) when they were encouraged to guess. Inter- was higher than the increase in percent- viewer probing decreased “don’t knows” even age incorrect (from twenty-one percent to further (Mondak 2001). Moreover, women twenty-nine percent). This is interesting be- were significantly more likely than men to cause each question had three response respond “don’t know” even when encouraged options. Hence, if the “don’t know”– to guess (Mondak and Anderson 2004). In this encouraged and “don’t know”–omitted analysis, discouraging “don’t knows” reduced groups were equivalent, and if all that omit- the extent to which men outperform women ting “don’t know” options does is cause (in terms of questions answered correctly) by respondents to guess haphazardly, then about half. respondents who would have otherwise cho- Such experiments also show that many sen “don’t know” should have only a one-in- respondents chose “don’t know” for reasons three chance of answering correctly. Hence, other than ignorance. Mondak and Davis if respondents were simply guessing, then (2001) analyzed the responses offered by the increase in average percentage incorrect respondents who initially claimed not to should be roughly double the increase in aver- know the answer. These responses were sig- age percentage correct. Instead, the increase nificantly more likely to be correct than in corrects was larger than the increase in responses that would have emerged from incorrects. Miller and Orr’s (2008) experi- blind guessing. Taken together, Mondak ment shows that the “don’t know” option and his colleagues show that many previous attracts not only respondents who lack knowl- political knowledge measures confound what edge about the questions’ substance, but also respondents know with how willing they are people who possess relevant information but to answer questions when they are uncertain are reticent to respond for other reasons (e.g., of themselves. lack of confidence or risk aversion). Building on these findings, Miller and Orr Although Miller and Orr’s (2008)work (2008) designed an experiment where the suggests that “don’t know” responses hide “don’t know” option was not just discouraged, partial knowledge, research by Sturgis, it was eliminated altogether. It was run on 965 Allum, and Smith (2008) suggests a differ- undergraduates via Internet surveys. Each ent conclusion. They integrated a split ballot respondent received eight multiple-choice experiment into a British telephone survey. political knowledge questions, and each ques- Each respondent was asked three knowledge- tion contained three substantive response related questions. Each question contained options. What differed across their experi- a statement, and the respondent was asked mental groups was question format. The first to indicate whether it was true or false. One group’s questions encouraged “don’t know.” thousand and six respondents were randomly The second group’s questions discouraged assigned to one of three conditions. In one “don’t know.” For the third group, the “don’t condition, the question’s preamble included know” option was simply unavailable. the “don’t know”–encouraging phrase “If you Political Knowledge 177 don’t know, just say so and we will skip In sum, many scholars and analysts use to the next one.” In a second condition, fact-based survey questions to draw broad that phrase was substituted with the “don’t conclusions about public ignorance. Exper- know”–discouraging phrase “If you don’t iments on survey-based political knowledge know, please give me your best guess.” In the measures have shown that many underappre- third condition, the original “don’t know”– ciated attributes of survey interview contexts encouraging statement was included, but the (including question wording, respondent response options were changed. Instead of incentives, and personality variations) are simply saying true or false, respondents in this significant determinants of past outcomes. group could say whether the statement was Hence, as a general matter, survey-based “probably true,” “definitely true,” “probably political knowledge measures are much less false,” or “definitely false.” Moreover, in the valid indicators of what citizens know about first and third conditions, respondents who politics than many critics previously claimed. initially said “don’t know” were later asked to provide their best guess. Sturgisetal.(2008) find that discour- 2. When Do Citizens Need aging “don’t know” responses significantly to Know the Facts? decreased their frequency (from thirty-three percent to nine percent). Providing the “def- In addition to shedding light on the valid- initely” and “probably” response options ity of survey-based political knowledge mea- also reduced “don’t know” responses (to sures, experiments clarify the conditions twenty-three percent). Turning their atten- under which knowing particular facts about tion to partial knowledge, they analyze the politics is necessary, sufficient, or relevant answers given by respondents who initially to a citizen’s ability to make competent responded “don’t know” and then chose true choices. Nonexperimental research on this or false after interviewer probing. For two topic has often tried to characterize the rela- of the three questions, probing elicited cor- tionship between political knowledge and rect answers at a rate no better than chance. politically relevant choices with brief anec- For the other question, two thirds of the dotes or with regression coefficients that new responses elicited were correct. From presume a simple linear and unconditional these results, they conclude that “when peo- relationship between the factors. These char- ple who initially select a ‘don’t know’ alterna- acterizations are never derived from direct tive are subsequently asked to provide a ‘best evidence or rigorous theory. For example, guess,’ they fare statistically no better than many nonexperimental critics of voter com- chance” (90). But their results suggest a dif- petence have merely presumed that any fact ferent interpretation – that there are types they deem worth knowing must also be a nec- of people and question content for which essary condition for others to make compe- encouraging “don’t knows” represses partial tent political decisions. knowledge and that there is still much to With experiments, scholars can evalu- learn about the types of questions and peo- ate hypotheses about relationships between ple that make such repressions more or less knowledge of specific facts and competence likely. Furthermore, with only three true– at various politically relevant tasks. To illus- false questions, their experiment does not trate how and why experiments are well suited provide a sufficient basis for privileging their for this purpose, we discuss several exam- conclusions over those of Miller and Orr ples. These examples reveal conditions under (2008). As they note, “The availability of which people who lack certain kinds of infor- three options from which to choose [the mation do, and do not, make competent Miller-Orr method] may motivate respon- choices nevertheless. dents to draw on their partial knowledge, Lupia and McCubbins (1998) acknowl- whereas the true/false format might not” edge that many citizens lack factual knowl- (779). edge about politics, but they emphasize that 178 Cheryl Boudreau and Arthur Lupia uninformed citizens may be able to learn would have made if they knew the coin toss what they need to know from political parties, outcome in advance. Similarly, when a suffi- interest groups, and the like (i.e., speakers). ciently large penalty for lying or probability They use a formal model to identify condi- of verification is imposed on the speaker, sub- tions under which citizens can make compe- jects trust the speaker’s statements and make tent choices as a result of such interactions. correct predictions at very high rates. They then use experiments to evaluate These experiments highlight conditions key theoretical conclusions. In some of these under which uninformed citizens can increase experiments, subjects guess the outcomes of their competence by learning from others. unseen coin tosses. Coin tosses are used But the experiments evaluate only a few of because they, like many elections, confront Lupia and McCubbins’ (1998) theoretical people with binary choices. Coin tosses are implications. One question that their experi- also easy to describe to experimental subjects, ments leave open is whether a speaker’s state- which makes many interesting variations of a ments are equally helpful to more and less coin toss guessing game easier to explain. knowledgeable citizens. On the one hand, it Before subjects make predictions, another is possible that a speaker’s statements will be subject (acting as “the speaker”) makes a state- more helpful to citizens who already know a ment about whether the coin landed on heads lot about the choices they face. On the other or tails. Subjects are told that the speaker is hand, a speaker’s statements may be more under no obligation to reveal what he or she helpful to people who know less. knows about the coin truthfully. After receiv- Boudreau (2009) replicates Lupia and ing the speaker’s statement, subjects predict McCubbins’ (1998) experiments but substi- the coin toss outcome. tutes math problems for coin tosses. An To evaluate key theoretical hypotheses, advantage of using math problems is that sub- Lupia and McCubbins (1998) systematically jects vary in their levels of preexisting knowl- manipulate multiple aspects of the informa- edge. Some subjects know a lot about how to tional context, including whether speakers get solve math problems. Others do not. A second paid when subjects make correct (or incor- advantage is that there exists a valid, reliable, rect) predictions. They also vary other fac- and agreed-on measure of how knowledge- tors that cause lying to be costly or increase able subjects are about this type of decision – the probability that false statements will be SAT math scores. Thus, Boudreau collects revealed (i.e., verification). In other words, subjects’ SAT math scores prior to the exper- to evaluate conditions under which subjects iments. She uses the experiments to clarify can learn enough to make correct predictions, conditions under which a speaker’s state- Lupia and McCubbins vary attributes of the ments about the answers to the math prob- speaker and the context in which subjects pre- lems help low-SAT subjects perform as well dict coin toss outcomes. as high-SAT subjects. Lupia and McCubbins’ (1998) experi- Boudreau (2009) finds that when the ments show that under the detailed sets of speaker is paid for subjects’ success, is sub- conditions identified by their theory, subject to a sufficiently large lying penalty, or jects almost always make correct predictions. faces a sufficiently high probability of verifi- Specifically, when subjects perceive a speaker cation, both low-SAT and high-SAT subjects as being knowledgeable and having common achieve large improvements in their decisions interests (i.e., as benefiting when subjects pre- (relative to counterparts in the control group, dict correctly), subjects trust the speaker’s who do the problems without a speaker). statements. When they are in conditions Low-SAT subjects improve so much that it under which such perceptions are likely to reduces the achievement gap between them be correct, these subjects make correct pre- and high-SAT subjects. This gap closes even dictions at a very high rate – one that is when the size of the penalty for lying or prob- substantially greater than chance and often ability of verification is reduced (and is, thus, indistinguishable from the predictions they made more realistic because the speaker may Political Knowledge 179 have an incentive to lie). This result occurs measures, approximately seventy percent of because high-SAT subjects do not improve subjects voted correctly. However, Lau and their decisions (and apparently ignore the Redlawsk (2001) identified conditions that speaker’s statements), but low-SAT subjects hinder subjects’ ability to vote correctly. For typically improve their decisions enough to example, they find that although heuristics make them comparable to those of high-SAT significantly increase the ability to vote cor- subjects. By using an experimental task for rectly among subjects who score high on which subjects vary in their levels of knowl- their political knowledge and political inter- edge, Boudreau further clarifies the condi- est index, they decrease less knowledgeable tions under which less informed citizens can and less interested subjects’ ability to vote 2 make competent choices. correctly. Lau and Redlawsk also find that One of the strengths of Boudreau’s (2009) characteristics of the information environ- and Lupia and McCubbins’ (1998) exper- ment limit subjects’ ability to vote correctly. iments is that subjects make decisions for Specifically, subjects are less likely to vote which there is an objectively correct or incor- correctly when the number of primary can- rect choice under different conditions. This didates increases from two to four and when approach is advantageous because it allows the choice between the candidates is more dif- them to measure precisely whether and when ficult (i.e., the candidates are more similar). a speaker’s statements help subjects make a Thus, although Lau and Redlawsk observe greater number of correct decisions than they high levels of correct voting in the aggre- would have made on their own. However, gate, they also show that there are conditions what does it mean for citizens to make cor- under which aspects of the information envi- rect decisions in electoral contexts? Lau and ronment have detrimental effects on subjects’ Redlawsk (1997, 2001) address this question ability to make correct decisions. by conducting experiments in which subjects Continuing Lau and Redlawsk’s (2001) learn about and vote for candidates in mock emphasis on the political environment, Kuk- primary and general elections. linski et al. (2001) use experiments to assess Subjects in Lau and Redlawsk’s (1997, the effects that other aspects of the environ- 2001) experiments are provided with differ- ment have on citizens’ decisions. In contrast ent types of information about fictional can- to Lau and Redlawsk’s focus on correct vot- didates. Subjects access this information by ing, Kuklinski et al. contend that the ability clicking on labels (e.g., “Walker’s stand on to make trade-offs is fundamental to being defense spending”) that appear on computer a competent citizen. They conduct survey screens. Subjects can also learn about a candi- experiments in which they measure subjects’ date’s partisanship, ideology, and appearance, ability to make trade-offs among competing as well as endorsements and polls. After sub- goals for health care reform. jects gather information about the primary Subjects in Kuklinski et al.’s (2001) exper- election candidates, they vote for one of these iments view seven different health care goals candidates. Subjects repeat this process for (e.g., universal coverage, no increase in taxes, the general election candidates. At the com- uniform quality of care), and they rate on a pletion of the experiments, subjects receive all scale of one to ten how much of each goal information that was available for two candi- a health care plan must achieve for them dates from the primary election (not just the to consider the plan acceptable. The key to information they clicked on during the exper- this experiment is that the health care goals iment). Subjects are then asked whether they conflict with one another; that is, no health would have voted for the same candidate if care plan can realistically achieve all goals. they had this information when they made In various treatment groups, Kuklinski et al. their decisions. manipulate the conditions under which Lau and Redlawsk (1997) find that sub- 2 The index combines subjects’ levels of politi- jects, in the aggregate, are adept at vot- cal knowledge, political behavior, political interest, ing correctly. According to one of their political discussion, and media use. 180 Cheryl Boudreau and Arthur Lupia subjects rate the seven health care goals. In vey experiments, Gilens (2001) suggests that one group, subjects are given general infor- policy-specific knowledge may be more rele- mation about the need for trade-offs when vant than more general conceptions of polit- designing any program. In a second group, ical knowledge. Gilens randomly determines subjects are given motivational instructions whether subjects receive specific informa- that encourage them to take their decisions tion about two policy issues (crime and for- seriously. In a third group, subjects are given eign aid). Subjects in the treatment group both general information and motivational receive information about two news stories, instructions. In a fourth group, subjects are one showing that the crime rate in America given diagnostic information about the exact has decreased for the seventh year in a row trade-offs involved in health care reform and one showing that the amount of money (e.g., that we cannot provide health cover- spent on foreign aid has decreased and is now age for everyone and simultaneously keep less than one cent of every dollar that the taxes low). In a fifth group, this diagnos- American government spends. Control group tic information is provided along with moti- subjects simply learn that two news stories vational instructions. Kuklinski et al. then have been released, one pertaining to a gov- observe whether and under what conditions ernment report about the crime rate and one information and/or motivation improves sub- pertaining to a report about American foreign jects’ ability to make trade-offs (measured aid. Gilens then asks all subjects about their as the extent to which subjects reduce level of support for government spending on their demands for conflicting goals), rela- prison construction and foreign aid. tive to subjects in the control group who do Gilens’ (2001) results demonstrate that not receive any information or motivational the provision of policy-specific information instructions. significantly influences subjects’ opinions. Kuklinski et al.’s (2001) experiment reveals Specifically, treatment group subjects (who conditions under which new information learn that the crime rate and foreign aid improves subjects’ ability to make trade- spending have decreased) are much less likely offs and eliminates differences between more than the control group to support increas- and less knowledgeable subjects. Specifi- ing government spending on prison construc- cally, where control group subjects tended tion and decreasing American spending on not to make trade-offs, treatment group foreign aid. Gilens shows that policy-specific subjects were much more likely to do so information has a stronger influence on sub- when both general information and motiva- jects who possess high levels of general polit- tional instructions were provided. Kuklinski ical knowledge. Gilens suggests that citizens’ et al. also show that diagnostic information ignorance of policy-specific facts (and not a about health care trade-offs induces sub- lack of general political knowledge) is what jects to make these trade-offs, regardless of hinders them from expressing their opinions whether they are motivated to do so and effectively on certain policy issues. regardless of their knowledge level. Indeed, Kuklinski et al. (2000) also assess whether when diagnostic information is provided, less policy-specific facts are relevant to citizen knowledgeable subjects are just as capable of opinions. In contrast to Gilens (2001), Kuk- making trade-offs as more knowledgeable linski et al. distinguish between citizens who subjects. In this way, Kuklinski et al. demon- are uninformed (i.e., who lack information strate that when information in the envi- about particular policies) and citizens who are ronment is sufficiently diagnostic, it can misinformed (i.e., who hold incorrect beliefs substitute for preexisting knowledge about about particular policies). Indeed, Kuklin- politics and eliminate differences between ski et al. suggest that the problem facing more and less knowledgeable subjects. our democracy is not that citizens are unin- Other experiments clarify the effect of formed, but rather that citizens confidently policy-specific knowledge on citizens’ abil- hold incorrect beliefs and base their opinions ities to express their opinions. Using sur- upon them. Political Knowledge 181

Kuklinski et al. (2000) assess experimen- that treatment group subjects either did not tally whether and when the provision of absorb these facts or failed to change their correct policy-specific information induces opinions in light of them. However, when a citizens to abandon incorrect beliefs and fact about welfare is presented in a way that express different opinions. In one treatment explicitly exposes and corrects subjects’ incor- group, subjects receive six facts about welfare rect beliefs (as in the follow-up experiments), (e.g., the percentage of families on welfare, subjects adjust their opinions about welfare the percentage of the federal budget devoted spending accordingly. In this way, Kuklinski to welfare) before they express their opinions. et al. demonstrate that policy-specific infor- In another treatment group, subjects first take mation can induce misinformed citizens to a multiple-choice quiz on these six facts about abandon their incorrect beliefs and express welfare. After answering each quiz question, informed opinions, but only when it is pre- subjects in this treatment group are asked how sented in a way that is meaningful and rele- confident they are of their answer. The pur- vant to them. pose of the quiz and the follow-up confidence Taken together, experiments have shown questions is to gauge subjects’ beliefs about conditions under which knowledge of the welfare and how confidently they hold them. kinds of facts that have been the basis of In the control group, subjects simply express previous political knowledge tests are nei- their opinions about welfare policy. ther necessary nor sufficient for competence Kuklinski et al. (2000) also conduct follow- at subsequent tasks. Some of these condi- up experiments in which the information tions pertain to the kind of information avail- about welfare is made more salient and mean- able. Other conditions pertain to the con- ingful than the six facts provided in the text in which the information is delivered. initial experiments. To this end, Kuklinski Collectively, these clarifications provide a et al. ask subjects what percentage of the fed- different view of citizen competence than eral budget they believe is spent on welfare is found in most nonexperimental work on and what percentage they believe should be political knowledge. Although there are many spent on welfare. Immediately after answer- cases in which lacking information reduces ing these two questions, subjects in the treat- citizens’ competence, experiments clarify ment group are told the correct fact, which for important conditions under which things are most subjects is that actual welfare spending is different. In particular, if there are relatively lower than either their estimate or their stated few options from which to choose (as is true in preference. Control group subjects answer many elections); if people can be motivated to these two questions but do not receive the pay attention to new information; if the infor- correct fact about actual welfare spending. At mation is highly relevant to making a compe- the completion of these experiments, subjects tent decision; and if people’s prior knowledge in both groups express their level of support of the topic, the speaker, or even the context for welfare spending. leads them to make effective decisions about In both experiments, Kuklinski et al. whom and what to believe, then even peo- (2001) find that subjects are grossly mis- ple who appear to lack political knowledge informed about welfare policy. Indeed, the as conventionally defined can vote with the percentage of subjects who answer particu- same level of competence as they would if lar multiple-choice quiz questions incorrectly better informed. ranges from sixty-seven percent to ninety percent. Even more troubling is their finding that subjects who have the least accurate beliefs 3. Conclusions are the most confident in them. Kuklinski et al. also show that the opinions of sub- Our argument in this chapter is as follows. jects who receive the six facts about welfare First, there are many ways in which the survey are no different from the opinions of sub- questions that scholars use to measure politi- jects in the control group, which indicates cal knowledge are not valid indicators of what 182 Cheryl Boudreau and Arthur Lupia citizens know about politics. Second, for cir- ation of Public Ignorance of the High Court.” cumstances in which such measures are valid, Journal of Politics 71: 429–41. it is not clear that this knowledge is neces- Gilens, Martin. 2001. “Political Ignorance and sary, sufficient, or relevant to a citizen’s abil- Collective Policy Preferences.” American Polit- ical Science Review 95: 379–96. ity to perform important democratic tasks, 1996 such as voting competently. Third, exper- Gould, Stephen J. . The Mismeasure of Man. rev. ed. New York: W.W. Norton. iments have shed important new light on Graber, Doris. 1984. Processing the News: How both of these issues. Collectively, experiments People Tame the Information Tide. New York: are not only helping scholars more effec- Longman. tively interpret existing data, but they are also Krosnick, Jon A., Arthur Lupia, Matthew DeBell, helping scholars develop better knowledge and Darrell Donakowski. 2008. “Problems with measures. In a very short period of time, ANES Questions Measuring Political Knowl- experiments have transformed the study of edge.” Retrieved from www.electionstudies. political knowledge. org/announce/newsltr/20080324Political 5 2010 Yet, experiments have just begun to scratch KnowledgeMemo.pdf (November , ). Kuklinski, James H., Paul J. Quirk, Jennifer Jerit, the surface of the multifaceted ways in which 2001 thought and choice interact. They have exam- and Robert F. Rich. . “The Political Envi- ronment and Citizen Competence.” American ined only a few of the many attributes of sur- Journal of Political Science 45: 410–24. vey interviews that can affect responses. They Kuklinski, James H., Paul J. Quirk, Jennifer Jerit, have also examined only a few of the many David Schwieder, and Robert F. Rich. 2000. ways that particular kinds of political knowl- “Misinformation and the Currency of Demo- edge can affect politically relevant decisions cratic Citizenship.” Journal of Politics 62: 790– and opinions. Their different results have also 816. raised questions about whether and when par- Lau, Richard R., and David P. Redlawsk. 1997. ticular types of information eliminate differ- “Voting Correctly.” American Political Science 91 585 98 ences between more and less knowledgeable Review : – . 2001 citizens (compare Lau and Redlawsk [2001] to Lau, Richard R., and David P. Redlawsk. . Kuklinski et al. [2001] and Boudreau [2009]). “Advantages and Disadvantages of Cogni- tive Heuristics in Political Decision Making.” As research in political cognition, the psy- American Journal of Political Science 45: 951– chology of the survey response, and political 71. communication evolves, more questions will Lupia, Arthur. 2006. “How Elitism Undermines be raised about the validity of extant knowl- the Study of Voter Competence.” Critical edge measures and whether particular kinds Review 18: 217–32. of knowledge are relevant to democratic out- Lupia, Arthur, and Mathew D. McCubbins. 1998. comes. Although such questions can be stud- The Democratic Dilemma: Can Citizens Learn ied in many ways, experiments should take What They Need to Know? New York: Cam- center stage in future research. Hence, the bridge University Press. McClone, Matthew S., Joshua Aronson, and Diane experiments we describe represent the begin- 2006 ning, rather than the end, of a new attempt Kobrynowicz. . “Stereotype Threat and the Gender Gap in Political Knowledge.” Psy- to better understand what people know about chology of Women Quarterly 30: 392–98. politics and why it matters. Miller, Melissa, and Shannon K. Orr. 2008. “Experimenting with a ‘Third Way’ in Political Knowledge Estimation.” Public Opinion Quar- References terly 72: 768–80. Mondak, Jeffery J. 2001. “Developing Valid Boudreau, Cheryl. 2009. “Closing the Gap: Knowledge Scales.” American Journal of Polit- When Do Cues Eliminate Differences between ical Science 45: 224–38. Sophisticated and Unsophisticated Citizens?” Mondak, Jeffery J., and Mary R. Anderson. 2004. Journal of Politics 71: 964–76. “The Knowledge Gap: A Reexamination of Gibson, James L., and Gregory A. Caldeira. 2009. Gender-Based Differences in Political Knowl- “Knowing the Supreme Court? A Reconsider- edge.” Journal of Politics 66: 492–513. Political Knowledge 183

Mondak, Jeffery J., and Belinda Creel Davis. Quick Recall from Political Learning Skills.” 2001. “Asked and Answered: Knowledge Lev- American Journal of Political Science 52: 168–82. els When We Will Not Take ‘Don’t Know’ for Steele, Claude M., and Joshua Aronson. 1995. an Answer.” Political Behavior 23: 199–224. “Stereotype Threat and the Intellectual Test Popkin, Samuel L. 1994. The Reasoning Voter: Com- Performance of African-Americans.” Journal of munication and Persuasion in Presidential Cam- Personality and Social Psychology 69: 787–811. paigns. Chicago: The University of Chicago Sturgis, Patrick, Nick Allum, and Patten Smith. Press. 2008. “An Experiment on the Measurement of Prior, Markus, and Arthur Lupia. 2008.“Money, Political Knowledge in Surveys.” Public Opinion Time, and Political Knowledge: Distinguishing Quarterly 72: 90–102.

Part IV

VOTE CHOICE, CANDIDATE EVALUATIONS, AND TURNOUT

CHAPTER 13

Candidate Impressions and Evaluations

Kathleen M. McGraw

For citizens to responsibly exercise one of that has contributed to our understanding of their primary democratic duties, namely, vot- citizens’ impressions and evaluations of polit- ing, two preliminary psychological processes ical candidates, as well as to identify questions must occur. First, citizens must learn some- that future experiments might answer. thing about the candidates, that is, come to some understanding, even if amorphous, about the candidates’ characteristics and pri- 1. Clariﬁcation of Basic Concepts orities. Second, citizens must reach a summary judgment about the candidates. There Two sets of conceptual distinctions should is a long and distinguished history of scholarly be made clear at the outset. First, the chapter studies of the linked processes of voting, per- title distinguishes between candidate impres- ceptions of candidates, and evaluations of them. sions and evaluations. By “impressions,” I In his recent review of the voting behavior mean an individual’s mental representation – literature, Bartels (2010) notes, “The appar- the cognitive structure stored in memory – ent failure of causal modeling [of observa- consisting of knowledge and beliefs about tional data] to answer fundamental questions another person.2 “Evaluation,” in contrast, about voting behavior produced a variety of refers to a summary global judgment ranging disparate reactions” (240), including scholars from very negative to very positive. Depend- turning to experimentation to better under- ing on the underlying cognitive processes that stand these basic processes. My goal in this contribute to the formation of the evaluation chapter is to outline experimental work1

chapter in this volume) and candidate advertisements Thanks to Jamie Druckman, Don Green, and Jim Kuk- (Gadarian and Lau’s chapter in this volume). linski for comments, and to Sarah Bryner for research 2 Many scholars (e.g., Nimmo and Savage 1976; assistance. Hacker 2004) refer to candidate images rather than 1 I am setting aside relevant research resulting from impressions. Both are deﬁned as cognitive represen- studies of the media (Nelson, Bryner, and Carnahan’s tations and so are synonymous.

187 188 Kathleen M. McGraw

(i.e., online or memory based), it may or may still know little about how the processes of not be part of the cognitive representation. impression formation, evaluation, and vote Second, I began this chapter by linking choice are linked. Consequently, my focus in candidate impressions and evaluations to vot- the remainder of this chapter is on impres- ing choices, and it is undoubtedly the case that sions and evaluations, rather than voting. those processes are connected. There is an understandable tendency among researchers and readers to assume that processes of polit- 2. Shortcomings of Observational ical impression formation and evaluation are Studies tantamount to voting choices, but it is an error to do so. Impression formation and Although the many observational studies on evaluation are concerned with single tar- candidate impressions and evaluations have gets, whereas a vote choice, like any choice, provided valuable insights, they also suf- requires selecting from a set of two or more fer from some limitations. Two bear men- alternatives. Whereas most observational and tioning. The first is the ability to draw experimental studies in this research tradi- strong inferences about causal determinants. tion are situated in a campaign context, it It is widely understood that variables such is worth noting that impressions and evalu- as the American National Election Stud- ations of individuals who eventually become ies (ANES) candidate like–dislike questions candidates for office often begin prior to the are contaminated by post hoc rationaliza- electoral context. For example, many Amer- tions, and so researchers must be cautious in icans had developed impressions and evalua- treating them as causes of the vote (Rahn, tions of Arnold Schwarzenegger long before Krosnick, and Breuning 1994; Lodge and he decided to run for office (and most of Steenbergen 1995). The same problem exists those Americans never had, and never will for virtually all factors presumed to be causal have, the opportunity to decide whether to determinants in the observational literature vote for him). The same holds true for (e.g., trait inferences, emotional responses, Hillary Rodham Clinton. I do not believe perceived issue positions). Experiments pro- these are unusual examples. Many local, state, vide scholars with the opportunity to evaluate and national political leaders are known from the causal impact of theoretically meaningful other areas of life before entering the elec- predictors.3 toral arena. Moreover, citizens continue to Second, candidates and other politicians update their impressions and evaluations once in the real world inevitably suffer from what candidates become elected officials and are no experimentalists refer to as confounds. In a longer seeking office. It is unfortunate that research design, confounding occurs when research has not captured these pre- and post- a second (or more) extraneous variable per- candidate phenomena. fectly covaries with the theoretical variable of Moreover, it is a mistake to assume, even interest. So, for example, the two candidates when a vote choice is the clear end point in the 2008 presidential election (Obama of candidate evaluation, that the underlying and McCain) differed on several important processes are equivalent. Behavioral decision dimensions that may have been consequen- theorists have long recognized that “choosing tial for citizens’ evaluations of them – parti- one alternative from a set can invoke dif- san affiliation, race, age, experience, image, ferent psychological processes than judging vice presidential picks, campaign strategies, alternatives” (Johnson and Russo 1984, 549). favorability of media coverage – making it dif- Lau and Redlawsk (2006) provide the clear- ficult for analysts to specify with any degree est discussion of this disjunction in the polit- of precision which characteristics were critical science literature. Global evaluations of ical. Experimentation allows researchers to individual candidates need to be translated into a decision about how to vote, and maybe 3 Panel studies can also be a solution to the causality even a decision about whether to vote. We problem. Candidate Impressions and Evaluations 189 disentangle – that is, manipulate indepen- suggest a trend away from a heavy reliance dently – candidate and situational factors to on college student samples in experimental determine which have a meaningful causal work more generally in political science.4 impact. The potential problems associated with college student samples, problems that are often exaggerated, are well known and beyond the 3. Principal Characteristics of scope of this chapter. Suffice it to say that Experiments on Candidate I concur with Druckman and Kam (chap- Impressions and Evaluations ter in this volume), who claim that “student subjects do not intrinsically pose a problem The experimental designs used to study can- for a study’s external validity.” In particu- didate impressions and evaluations can be lar, there is little theoretical reason to believe characterized along three key dimensions. that the basic cognitive processes underlying The first is the static versus dynamic dimen- the formation of impressions and evaluations sion. The vast majority of experiments in this of candidates vary across different samples. tradition rely on a static, one-shot presen- And, to the extent that theoretically mean- tation of information about a political can- ingful differences might exist (e.g., in terms of didate, using (more or less realistic) paper cognitive ability and political sophistication), “campaign fact sheets,” videotapes, or elec- experimental researchers have been open to tronic formats. Not only is the presentation exploring the impact of those moderators. of information static, but participants are A third important distinction is how the typically provided with the information that partisan affiliation of the candidate is handled the researcher deems relevant, rather than in the experimental design.5 Three possibil- being responsible for seeking out informa- ities exist. The first is to vary the partisan tion. Exposure, in other words, is held con- affiliation of the stimulus candidate so that stant. partisanship becomes an independent vari- In the late 1980s, Rick Lau and David able that is fully crossed with other manip- Redlawsk (2006) began to develop a dynamic ulated variables (e.g., Riggle et al. 1992). The process tracing methodology, where the second approach is to make explicit the parti- information that is available about candidates sanship of the target candidate and hold it comes and goes, and research participants are constant (e.g., Lodge, McGraw, and Stroh responsible for choosing the information they 1989). Finally, in some studies, the parti- want to learn. This represents a significant san affiliation of the candidate is left out methodological development. Although very altogether (McGraw, Hasecke, and Conger different in many important respects, the [2003] is an example, but there are many oth- static and dynamic paradigms do share two ers). Partisan attachments exert an enormous features. First, in the research conducted to impact on citizens’ impressions and evalua- date, both involve a simulated campaign that tions of political candidates, which would sug- takes place in a single, short period of time gest that partisan affiliation should be central (Mitchell [2008] is an exception). Second, to these experiments. However, experimen- both paradigms rely, for the most part, on talists face trade-offs, as all researchers do, hypothetical candidates. I do not know of any experimental studies involving real can- 4 McGraw and Hoekstra (1994) review experiments published in the top disciplinary journals between didates over an extended period of real time. 1950 and 1992 and conclude that sixty-five percent of A second dimension involves the research published experiments involved college student par- participants. Although the majority of exper- ticipants. More recently, Kam, Wilking, and Zech- meister (2007) examine experiments published in the imental studies in this tradition rely on sam- same journals between 1990 and 2006, concluding ples of convenience rather than representa- that twenty-five percent relied on college student tive samples, some studies rely on college samples. 5 I have in mind the American political context here, students, whereas others draw from nonstu- and so identification of candidates is as Republican or dent volunteer populations. Recent analyses Democratic, although the point is more general. 190 Kathleen M. McGraw and the first option – manipulation of candi- edly wide ranging, the experimental literature date partisanship – comes with a sometimes tends to focus on traits and policy positions. hefty cost, namely, doubling the number of Traits are a central component of ordi- treatments and thus the number of partici- nary and political impressions, so they have pants. But the failure to manipulate partisan- received a tremendous amount of theoreti- ship (i.e., by holding it constant or ignoring cal and empirical attention. Because traits are it) carries risks, beyond abstract worries about unobservable, they must be inferred from the external validity. First, if partisanship is not observable qualities of the politician. Because manipulated, it is impossible to ascertain behavior is often ambiguous, there is rarely whether the impact of a key manipulation an inevitable correspondence between a par- holds for candidates of both parties. Second, ticular behavioral episode and the resulting in some instances, the presence of informa- trait inference. Although there is a seemingly tion about a candidate’s partisan affiliation infinite number of traits available in ordi- can serve to dampen, and even eliminate, the nary language, the most common traits used impact of other manipulated variables. For to characterize politicians tend to fall into example, the Riggle et al. (1992) study found a limited number of categories: competence that candidate attractiveness had no impact (“intelligent,” “hard working”), leadership when partisanship was available. Similarly, (“inspiring,” “[not] weak”), integrity (“hon- Stroud, Glaser, and Salovey (2006) found that est,” “moral”), and empathy (“compassion- emotional expressions on the part of candi- ate,” “cares about people”). It is clear that trait dates had no affect when partisan information inferences are consequential for evaluations was provided. In short, when researchers do of political candidates and vote choices (Funk not manipulate partisan affiliation, they risk 1996; Kinder 1998). Of the four dimensions, failing to detect contingency effects or over- competence appears to be most influential, stating the influence of other factors. at least in terms of evaluations of presidential candidates (Markus 1982; Kinder 1986). Much of the available data are cross-sectional, 4. Content of Candidate Impressions raising the very real possibility that trait inferences are rationalizations, rather than causes As noted in the previous section, impres- of evaluations. However, experimental work sions are cognitive structures, consisting of has verified that traits play a causal role in what we know and believe about another shaping candidate evaluations (Huddy and person, the information we have learned, Terkildsen 1993; Funk 1996). and the inferences we have drawn. Citizens’ Policy positions also play a prominent role impressions of political officials are rich and in impressions and evaluations of candidates. multifaceted, consisting of trait inferences, Although the authors of The American Voter knowledge about political attributes such (Campbell et al. 1960) suggested issues are as partisanship, beliefs about issue positions a relatively peripheral component of evalua- and competencies, personal characteristics tions and vote choices, more recent work sug- and history, family, group associations, and gests a more prominent role. For example, hobbies and personal proclivities6 (Miller, issues that people consider important have Wattenberg, and Malanchuk 1986;McGraw, a substantial impact on presidential candi- Fischle, and Stenner 2000). Although the date evaluations (Krosnick 1988). Relatedly, content of political impressions is undoubt- experimental and observational research on media priming effects indicates that issues that are highlighted in the media have a siz- 6 In this context, it is worth remembering Delli Carpini and Keeter’s (1996, 76) playful observation: “Over able impact on evaluations of political lead- the fifty years of survey items we examined, only two ers (Iyengar and Kinder 1987; Krosnick and issue stands of public officials could be identified by Kinder 1990). more than three-quarters of those surveyed. . . . [one] was George Bush’s 1989 disclosure that he hates Having established that traits and pol- broccoli!” icy positions are important components of Candidate Impressions and Evaluations 191 candidate impressions and that they play a Of the four stereotype categories, gen- causal role in evaluations, it is important der has received the most scholarly attention, to determine their sources. It is custom- and the results from observational and exper- 7 ary to categorize citizens as flexible infor- imental studies converge. All else equal, mation processors, capable of engaging in female candidates are ascribed stereotypical both data-driven (individuating) and theory- feminine traits, whereas male candidates are driven (stereotypical) processing, in line with described in stereotypical masculine terms. the theoretical predictions drawn from dual In addition, gender-based stereotypes extend processing models (Fiske and Neuberg 1990). to perceived competency in various policy By individuation, I mean judgments based domains, as well as to inferences about parti- on the specific information that is avail- sanship and ideology. Huddy and Terkild- able, without reference to stereotypical cat- sen’s (1993) experiment provides the most egories. It is clear from the literature that sophisticated analysis of the complex links citizens’ judgments about traits and pol- among candidate gender, traits, and issue icy positions are rooted to some extent in competency. the actual behaviors manifested by candi- Racial considerations are also consequen- dates (see, e.g., Kinder [1986] for sensible tial for a wide range of public opinion candidate-specific differences in trait percep- phenomena.8 However, on the specific questions, and Ansolabehere and Iyengar [1995] tion of whether racial stereotypes (beliefs for evidence that citizens learn issue positions about the characteristics of members of racial from advertising). Although the content of groups, as opposed to the affective phe- citizens’ impressions can be grounded in the nomenon of racial prejudice) have an impact information they learn about specific candi- on candidate perceptions and evaluations, dates, that information need not be accurate there is surprisingly little evidence, and that or objective. After all, candidates (and their which exists is decidedly mixed. Different opponents) structure communication strate- experiments have reached different conclu- gies to manipulate these perceptions (I return sions about the extent to which African to this theme in the next section). American candidates are inferred to pos- Trait and policy inferences also result sess positive and negative trait characteristics from stereotyping, a consequence of cate- (Colleau et al. 1990; Williams 1990; Mosko- gorizing individuals into different groups or witz and Stroh 1994; Sigelman et al. 1995). types. As a type of cognitive structure, stereo- White Americans infer that African American types contain elements that can be applied candidates support more liberal policy posi- by default to individual members of the tions (Williams 1990; Sigelman et al. 1995; group, particularly in low information set- McDermott 1998), consistent with the broad tings. Although many stereotypes might be racial gap in policy preferences observed in activated when thinking about candidates, the population at large (Kinder and Sanders political scientists tend to focus on physical 1996). The extent to which these policy appearance, gender, race, and partisanship. inferences are produced by racial or parti- The first three promote stereotyping because san stereotypes is unclear, however, because they are physical characteristics that are acti- the majority of African American citizens vated by visual cues. There is good reason to and candidates are affiliated with the Demo- believe that stereotypes based in these cate- cratic Party, and the aforementioned studies gories are especially powerful because infer- did not independently manipulate candidate ences drawn from physical cues tend to be partisanship. even more automatic than those drawn from 1989 verbal sources (Gilbert ). Gender, race, 7 Readers should consult Dolan and Sanbonmatsu’s and partisanship also promote stereotyping chapter in this volume for an extended discussion because they are politically meaningful cate- of this literature. 8 Because of space limitations, I focus here on attitudes gories that play a prominent role in American about African Americans and African American can- politics. didates. 192 Kathleen M. McGraw

More generally, as Hutchings and Valen- and Hartman 1989). The causal model sug- tino (2004) conclude, “we still are unsure why gested – but not yet tested – by these disparate whites do not support black political can- findings is that the partisan affiliation of the didates” (400). Several recent studies sug- candidate produces inferences about policy gest promising avenues for future research. positions (grounded in both actual commu- Schneider and Bos (2009) make the insight- nication strategies and stereotypes), which in ful argument that previous research on racial turn generate trait inferences. stereotypes and candidate inference has been The final stereotypical category is phys- misguided in the assumption that stereotypes ical appearance, which has an impact on 9 of African American politicians are equivalent trait inferences and evaluation. Absent to stereotypes of African American people in any other information about a candidate’s general, and their data on this point are com- qualities, more attractive facial appearances pelling, with little overlap in the content of produce more positive trait inferences and the two. Hajnal’s (2006) research suggests evaluations (Rosenberg et al. 1986;Sigel- that the candidate’s status is important, with man, Sigelman, and Fowler 1987). However, African American incumbents viewed more as noted previously, when information about favorably than African American challengers. partisanship is available, physical attractive- Berinsky and Mendelberg’s (2005) analysis of ness appears to have no effect, suggesting lim- Jewish stereotypes – in particular, the links its to its impact (Riggle et al. 1992). Facial between, and consequences of, acceptable and maturity has also been implicated. People unacceptable stereotypes – provides a psycho- attribute more warmth, honesty, and sub- logically astute framework for understand- missiveness to baby-faced adults (possessing political stereotypes and judgment more ing large eyes, round chins, and thick lips), generally. whereas more mature facial features (char- The third stereotypical category to be acterized by small eyes, square jaws, and considered is partisanship, and here obser- thinner lips) elicit attributions of dominance vational and experimental studies nicely and strength (Zebrowitz 1994). In a creative converge in the domain of policy infer- experimental demonstration of the politi- ences: citizens hold clear and surpris- cal consequences of facial maturity, Keating, ingly consensual beliefs about the policy Randall, and Kendrick (1999) manipulate the positions and issue competencies that “go facial images of recent presidents through with” partisan affiliation (e.g., Feldman and digital techniques, finding that subtle changes Conover 1983; Rahn 1993). Hayes (2005), in facial features affected the trait ratings of in an extension of Petrocik’s (1996) theory well-known leaders in a theoretically mean- of issue ownership, demonstrates that citi- ingful fashion. zens also associate specific traits with can- Two recent experimental research pro- didates from the two parties (i.e., the pub- grams demonstrating the potentially power- lic views Republicans as stronger leaders and ful effects of facial appearance are note- more moral, whereas Democrats are seen as worthy. Iyengar and his colleagues (e.g., more compassionate and empathetic). Hayes’ Bailenson et al. 2008), using a morphing tech- study is observational, and experimental con- nology to digitally alter a candidate’s appear- firmation of this finding would be useful. ance to make it more similar to research par- Hayes also assumes that partisan trait infer- ticipants, show that facial similarity produces ences are derived from the policy positions more positive trait inferences and higher lev- taken by candidates of different parties. This els of candidate support, above and beyond assumption is consistent with experimental work; that is, although people frequently 9 There is not, as far as I know, any evidence that make inferences between candidate traits and physical appearance has an impact on issue position issue information, they are more likely to inferences. Given the connections between trait and policy inferences (Rapoport et al. 1989), it is possible infer candidate traits from issue positions that physical appearance has some (probably minor) rather than the reverse (Rapoport, Metcalf, impact on policy inferences. Candidate Impressions and Evaluations 193 the impact of partisan and policy similar- forth to describe the processes underlying the ity. These facial similarity effects are particu- formation of evaluations of politicians.10 The larly evident among weak partisans and inde- first posits that evaluations are formed online, pendents, and for unfamiliar candidates (see with continuous updating of the summary Iyengar’s chapter in this volume). Todorov evaluation as new information is encountered (in Hall et al. 2009) shows that rapid judg- (Hastie and Park 1986; Lodge et al. 1989; ments of competence, based on facial appear- Lodge and Steenbergen 1995). Under the ances, predict the outcomes of real guber- alternative, memory-based processing, opin- natorial, Senate, and House campaigns (and, ions are constructed at the time an opin- consistent with the trait literature previ- ion is expressed by retrieving specific pieces ously described, inferences of competence of information from long-term memory and from facial appearance, rather than other trait integrating that information to create a sum- inferences, are key; Hall et al. [2009] provide mary judgment (Zaller 1992; Zaller and Feld- a good review of Todorov’s research pro- man 1992). gram). These facial appearance effects appear The two most important experimental to be automatic and outside conscious aware- studies of the cognitive process models of can- ness, consistent with contemporary psycho- didate evaluation are Lodge and Steenbergen logical understandings of automaticity and (1995) and Lau and Redlawsk (2006). Nei- social thought (Andersen et al. 2007). ther, interestingly, is notable for the manip- Taken as a whole, the experimental litera- ulation of specific independent variables that ture provides considerable empirical evidence shed light on the causal mechanisms that pro- that stereotyping processes have an impact duce each type of processing. Rather, their on candidate impressions. Yet, there are also importance is the result of ambitious research many important questions that remain unad- designs and careful measurement. dressed. For example, too few studies in this Lodge and Steenbergen (1995) represent tradition manipulate, or take into account, an advance on prior experimental studies of the partisanship of the target candidate, mak- candidate evaluation in at least three ways. ing it impossible to determine the magnitude For one, it is the first study to consider (if any) of other characteristics on trait and these processes in a comparative context (i.e., issue inferences when this politically impor- participants received information about two tant characteristic is made salient. Similarly, candidates running for office). Importantly, there has been little consideration of the char- despite the comparative context, the experi- acteristics of individuals and situations that mental instructions and dependent variables moderate the impact of factors such as race, focus on evaluations of the candidates, not gender, and physical appearance. Finally, it a choice between them. Second, Lodge and is clear that traits and issue positions are Steenbergen extend the time frame beyond important components of impressions and the typical brief, single laboratory sitting by causal determinants of candidate evaluations. collecting recall and evaluation data over a However, as implied in the preceding discus- month-long period. Their empirical analyses sion, we know very little about how trait and provide support for the dominance of online issue inferences are linked and the underlying processing, consistent with most of the pre- causal dynamics that connect a given candi- vious experimentally based literature.11 The date’s characteristics, the intervening inferences, and the resulting summary evaluation. 10 Experiments have been critical because most observational studies fail to include measures for one or both of the variables that are necessary for empirically testing the two models: a measure for information that is 5. Cognitive Process Models of retrieved from memory, and a measure capturing the Candidate Evaluation totality of information that was received. Because of this, it is difficult to compare the results from observational and experimental studies in this area. Experiments have been critical in the devel- 11 Lavine (2002)andMcGraw(2003) conclude that the opment and testing of the two models put literature, much of it based on experiments, generally 194 Kathleen M. McGraw third important advance of the Lodge and (also see Redlawsk 2001) suggest that eval- Steenbergen study lies in the compelling uations of political candidates have a signifi- implications they draw from their results. cant memory-based component when a vote They argue forcefully that experimental, as choice is required. Their data are compelling opposed to observational, methods are neces- on this point, and so provide a significant sary to understand candidate evaluation pro- challenge to the conclusions reached in the cesses because it is only with experiments Stony Brook studies, which implied that vot- that researchers can know with certainty the ing is the result of online processes (despite information that individuals have received. the fact that none of those studies required a Second, they argue that if political scien- vote choice from the participants). Although tists focus on information holding and reten- Lau and Redlawsk may very well be cor- tion, then they are likely to underestimate rect in their conclusion that voting – mak- the extent to which campaigns and media ing a choice – promotes memory-based pro- have an impact on citizens. Finally, Lodge cessing, an alternative explanation exists. The and Steenbergen reach the normative con- learning environment in the dynamic process clusion that information holding (i.e., recall) tracing methodology is complex. Research is not a proper standard of “good citizenship.” participants learn about four or six can- Rather, they contend that what really matters didates at once, with information stream- is the type of information that citizens receive ing by at a fast pace – a pace that may and whether they are responsive to that undermine their ability to encode infor- information. mation and update multiple online tallies. Lau and Redlawsk’s (2006) inventive re- Consequently, it is possible that task com- search has also substantially expanded our plexity, rather than the vote choice per understanding of a wide range of phenom- se, is responsible for the memory-based ena relevant to candidate evaluation and vote results because complex tasks disrupt nor- choice. As I previously noted, Lau and Red- mal (in this context, online) processing rou- lawsk developed an innovative dynamic pro- tines (Kruglanski and Sleeth-Keppler 2007). cess tracing methodology that stands in stark I recognize that the complexity of Lau contrast to the static campaign paradigm used and Redlawsk’s (2006) design was deliber- in previous experimental studies. Contrary ate because they believe the method matches to Lodge and Steenbergen (1995), the elec- the complexity of real-world learning about tion context and the necessity of an eventual candidates. As a result, the “methodological vote is salient and central to their research artifact” explanation put forth here is consis- design. In regard to the two processing tent with their preferred conclusion, namely, models, Lau and Redlawsk’s (2006) results that voting promotes memory-based processing. Additional research will be needed to supports the conclusion that candidate evaluations (as evaluate these competing explanations. opposed to policy opinion formation) largely result Surprisingly, aside from the vote choice from online processes. There are exceptions to this 2001 and other contextual factors considered by conclusion (e.g., Redlawsk ; Lau and Redlawsk 2006 2006; Mitchell 2008). Moreover, the experimental Lau and Redlawsk ( ) and by Rahn, literature demonstrates that theoretically meaning- Aldrich, and Borgida’s (1994) study of infor- ful individual differences moderate the propensity mation format, there has been no other to engage in online versus memory-based processing (see my discussion in following sections of this research examining how structural or con- chapter). The conclusions reached from the avail- textual factors influence the propensity to able observational studies are more mixed. Rahn et al. engage in online versus memory-based pro- (1994) and Graber (1984) reach conclusions consistent with online processing, whereas Just et al. (1996) cessing, and certainly this is an avenue for conclude that a mix of online and memory-based pro- future research. For example, one might cessing occurs. The Kelley and Mirer (1974) analyses imagine that different visual images (i.e., per- support a memory-based conclusion, but the absence of a measure capable of capturing online processing sonalizing or depersonalizing the candidate) renders that conclusion suspect. would have an impact (McGraw and Dolan Candidate Impressions and Evaluations 195

[2007] present evidence consistent with this 6. Considering the Impact of logic). In contrast to the scarcity of research Strategic Candidate Behaviors focusing on the contextual moderators of the two processing models, there has been Much of the experimental literature on can- a good deal of research identifying individ- didate impressions and evaluations fails to ual difference moderators. Politically sophis- take into account the self-presentational tac- ticated individuals are more likely to engage tics that politicians engage in to influence in online processing, whereas those who are their constituents.12 I noted elsewhere that less sophisticated tend to engage in memory- “these processes are two sides of the same based processing (McGraw, Lodge, and Stroh coin . . . and a complete understanding of 1990; McGraw and Pinney 1990;McGrawet what ordinary citizens think about politicians al. 2003; McGraw and Dolan 2007). In addi- will be out of reach until political psycholo- tion, people with a high need to evaluate and gists take into account the strategic interplay people who are “entity theorists” (i.e., who between elites and the mass public” (McGraw believe that other people’s personalities are 2003, 395). There, I took a positive approach stable and fixed) are more likely to manifest by reviewing research that is suggestive as to online processing (McGraw and Dolan 2007). this strategic interplay; here, I focus on some There is a surprising disconnect between of the many unanswered questions that exper- the two empirical traditions that I have iments are well suited to answer. reviewed so far. That is, research on the con- Fenno’s (1978) influential presentation of tent of candidate impressions has not con- “home style” – referring to three sets of activ- sidered process, or “how citizens assemble ities in which elected representatives engage – their views, how they put the various ingredi- provides a good starting point. The first activ- ents together” (Kinder 1998, 812). Similarly, ity set is self-presentation along three dimen- work on online and memory-based process- sions: conveying qualifications (competence ing, incorporating global net tallies, has not and honesty), a sense of identification with been sensitive to the possibility that different constituents, and empathy. These dimensions kinds of information and inferences may have dovetail nicely with the trait ascriptions exam- their impact through different psychological ined in the research I describe in this chapter. process mechanisms. However, the ultimate objective in Fenno’s One final thought on the two pro- description of political self-presentation is cessing models of candidate evaluation: it not high approval ratings (evaluations) or trait is increasingly common for scholars, both inferences, but rather the more nebulous and experimentalists and scholars working with fragile concept of trust, including the willing- observational data, to reject the either-or ness on the part of constituents to provide lee- approach, despite the fact that those kinds of way to the representative on legislative deci- conclusions characterize most of the empir- sions. There has been no experimental work ical work. Rather, scholars often conclude linking politicians’ self-presentational tactics, that a hybrid model, incorporating both trait inferences, and constituent trust. online and memory-based components, may Fenno’s (1978) second category of “home be more psychologically realistic (Hastie and style” activities involves allocation of reso- Pennington 1989; Zaller 1992;Justetal. urces to the district (travel back home, staff, 1996;Lavine2002;McGraw2003; Lau and casework, communication efforts, etc.). Vir- Redlawsk 2006; Lodge, Taber, and Verhulst’s tually all analyses of the impact of con- chapter in this volume). In theory, a hybrid stituency service are observational, and the model is almost certainly correct. But in prac- research results from those analyses are tice, it is not clear what this really means, how unpromising: “nearly every scholar who has we would go about empirically testing for it, or the conditions under which we would 12 I am setting aside work on campaign advertisements expect hybrid processing to occur. (Gadarian and Lau’s chapter in this volume). 196 Kathleen M. McGraw analyzed resource data, beginning with would be to consider how citizens respond to Fenno, has turned up negative results” (Rivers elites’ attempts to modify their issue positions and Fiorina 1991, 17). In part, as Rivers to be consistent with public opinion and, in and Fiorina demonstrate, the null results are particular, when and why citizens view such attributable to a problem of endogeneity (i.e., movements positively as “democratic respon- that allocation of resources is dependent on siveness” (Stimson, MacKuen, and Erikson the representative’s expectation of electoral 1995) or pejoratively as “pandering” (Jacobs success). There has been very little exper- and Shapiro 2000). McGraw, Lodge, and imental work aimed at understanding rep- Jones (2002) provide some preliminary exper- resentatives’ responsiveness to constituents imental evidence on these questions, finding or how various types of constituency service that the arousal of suspicion of pandering is influence citizens’ impressions and evalua- the result of a complex interplay between sit- tions of elected officials (Cover and Brum- uational cues and agreement with the mes- berg [1982] and Butler and Brockman [2009] sage. If aroused, suspicion of pandering does provide notable exceptions), but such designs have negative consequences for evaluations of have the potential to provide a great deal of public officials. insight into these important phenomena. Second, consider Petrocik’s (1996)the- Unlike the first two, Fenno’s (1978)third ory of “issue ownership,” which argues that “home style” activity, explanation of Wash- parties develop reputations for being more ington activity to constituents, has been the skilled at handling certain policy domains. focus of a fair amount of experimental work Specific candidates, in turn, are perceived (see McGraw [2003] for a review). It is clear as more credible over issues “owned” by from that work that different types of expla- their party, so they strive to make the issues nations for policy decisions and corrupt or associated with their party “the program- scandalous acts have systematic effects on a matic meaning of the election and the criteria host of judgments: attributions of credit and by which voters make their choice” (Petro- blame, inferences about specific trait charac- cik 1996, 828). In a campaign advertising teristics, and global evaluations of politicians. study, Ansolabehere and Iyengar (1994)pro- This is a research area, too, however, where a vide convincing arguments for why experi- number of unanswered questions remain. For mental investigations of the issue ownership example, we know very little about the conse- hypothesis are warranted (e.g., in campaigns quences of political explanations outside the where other candidate characteristics, such as laboratory (so this is a call for more observa- sex, race, or prior experience, are confounded tional studies). In addition, the experimental with party, it is impossible to determine which work has focused on particular types of expla- characteristic has “ownership” of the issue). nations – excuses and justifications13 – and, as Ansolabehere and Iyengar find effects on a result, we know much less about the impact voting choice but not on other dimensions of other accounts – in particular, apologies. of candidate evaluation (consistent with the This should be of interest, given the appar- argument that the two are distinct processes), ent proliferation of apologies in contempo- suggesting that further experimental work on rary politics and the high level of attention to when and why issue ownership is consequen- apology discourse in the media. tial for candidate impressions and evaluations It is also worth considering how experi- would be useful. mental work might contribute to our under- The third aspect of candidate position tak- standing of the impact of “position taking” ing that would benefit from experimental (Mayhew 1974) on evaluations and impres- investigation is ambiguity. Political scientists sions. Three areas seem promising. The first have long emphasized that politicians often have an incentive to adopt ambiguous issue positions (Downs 1957;Key1958) because 13 Excuses involve a denial of full or partial responsibility for some outcomes, whereas justifications attempt “by shunning clear stands, they avoid offend- to redefine evaluations of an act and its consequences. ing constituents who hold contrary positions; Candidate Impressions and Evaluations 197 ambiguity maximizes support” (Page 1976, Bailenson, Jeremy N., Shanto Iyengar, Nick Yee, 742). Scholars have also formalized the incen- and Nathan A. Collins. 2008. “Facial Similarity tives for taking ambiguous positions, as well between Voters and Candidates Causes Influ- 72 935 61 as the circumstances under which they ought ence.” Public Opinion Quarterly : – . 2010 to be avoided. Although there is some obser- Bartels, Larry M. . “The Study of Electoral vational work examining the causes and con- Behavior.” In The Oxford Handbook of Ameri- can Elections and Political Behavior, ed. Jan E. sequences of ambiguous position taking (Page 1972 1983 Leighley. New York: Oxford University Press, and Brody ; Campbell ), there is 239–61. little experimental work that examines the Berinsky, Adam J., and Tali Mendelberg. 2005. factors that lead citizens to view a posi- “The Indirect Effects of Discredited Stereo- tion as ambiguous, or the conditions under types.” American Journal of Political Science 49: which ambiguous positions produce positive 846–65. or negative consequences for a candidate who Butler, Daniel M., and David E. Broockman. espouses them (Tomz and van Houweling 2009. “Who Helps DeShawn Register to Vote? [2009] provide a provocative recent excep- A Field Experiment on State Legislators.” tion). Unpublished manuscript, Yale University. Campbell, James E. 1983. “The Electoral Conse- quences of Issue Ambiguity: An Examination of the Presidential Candidates’ Issue Positions 7. Conclusions from 1968 to 1980.” Political Behavior 5: 277– 91. Experiments have provided a platform for Campbell, Angus, Philip Converse, Warren important advances in our understanding of Miller, and Donald Stokes. 1960. The Ameri- candidate impressions and evaluations. Yet, can Voter. New York: Wiley. as I highlight, there are many significant Colleau, Sophie M., Kevin Glynn, Steven questions that remain to be answered. In Lybrand, Richard M. Merelman, Paula Mohan, 1990 particular, we need to understand the recipro- and James E. Wall. . “Symbolic Racism in Candidate Evaluation: An Experiment.” Politi- cal linkages among citizen inferences, evalua- 12 385 402 tions and choices, and candidate strategies by cal Behavior : – . Cover, Albert D., and Bruce S. Brumberg. 1982. developing more comprehensive theories that “Baby Books and Ballots: The Impact of integrate psychological and political princi- Congressional Mail on Constituent Opinion.” ples. It is my hope that the next generation American Political Science Review 76: 347–59. of political science experimentalists will meet Delli Carpini, Michael X., and Scott Keeter. 1996. these challenges. What Americans Know about Politics and Why It Matters. New Haven, CT: Yale University Press. References Downs, Anthony. 1957. An Economic Theory of Democracy. New York: Harper & Row. Andersen, Susan M., Gordon B. Moskowitz, Irene Feldman, Stanley, and Pamela Johnston Conover. V. Blair, and Brian A. Nosek. 2007.“Auto- 1983. “Candidates, Issues, and Voters: The matic Thought.” In Social Psychology: Handbook Role of Inference in Political Perception.” Jour- of Basic Principles, eds. Arie W. Kruglanski and nal of Politics 45: 810–39. E. Tory Higgins. New York: Guilford Press, Fenno, Richard E. 1978. Home Style: House Mem- 138–75. bers in Their Districts. Boston: Little, Brown. Ansolabehere, Stephen S., and Shanto Iyengar. Fiske, Susan T., and Steven L. Neuberg. 1990. 1994. “Riding the Wave and Issue Ownership: “A Continuum Model of Impression Forma- The Importance of Issues in Political Adver- tion: From Category-Based to Individuating tising and News.” Public Opinion Quarterly 58: Processes as a Function of Information, Moti- 335–57. vation, and Attention.” In Advances in Experi- Ansolabehere, Stephen S., and Shanto Iyengar. mental Psychology. vol. 23, ed. Mark P. Zanna. 1995. Going Negative: How Political Advertising San Diego: Academic Press, 1–74. Divides and Shrinks the American Electorate.New Funk, Carolyn L. 1996. “Understanding Trait York: Free Press. Inferences in Candidate Images.” In Research 198 Kathleen M. McGraw

in Micropolitics. vol. 5, eds. Michael X. Delli mation.” Journal of Consumer Research 11: 542– Carpini, Leonie Huddy, and Robert Y. Shapiro. 50. Greenwich, CT: JAI Press, 97–123. Just, Marion R., Ann N. Crigler, Dean E. Alger, Gilbert, Daniel T. 1989. “Thinking Lightly about Timothy E. Cook, and Montague Kern. 1996. Others: Automatic Components of the Social Crosstalk: Citizens, Candidates, and the Media in a Inference Process.” In Unintended Thought, eds. Presidential Campaign. Chicago: The University John S. Uleman and John A. Bargh. New York: of Chicago Press. Guilford Press, 189–211. Kam, Cindy D., Jennifer R. Wilking, and Eliza- Graber, Doris A. 1984. Processing the News: How beth J. Zechmeister. 2007. “Beyond the ‘Nar- People Tame the Information Tide. New York: row Data Base’: Another Convenience Sample Longman. for Experimental Research.” Political Behavior Hacker, Kenneth L. 2004. Presidential Candidate 29: 415–40. Images. Lanham, MD: Rowman & Littleﬁeld. Keating, Caroline F., David Randall, and Timothy Hajnal, Zoltan L. 2006. Changing White Attitudes Kendrick. 1999. “Presidential Physiognomies: toward Black Political Leadership. New York: Altered Images, Altered Perceptions.” Political Cambridge University Press. Psychology 20: 593–610. Hall, Crystal C., Amir Goren, Shelly Chaiken, and Kelley, Stanley, and Thad Mirer. 1974. “The Sim- Alexander Todorov. 2009. “Shallow Cues with ple Act of Voting.” American Political Science Deep Effects: Trait Judgments from Faces and Review 61: 572–79. Voting Decisions.” In The Political Psychology Key, Vladimir O. 1958. Politics, Parties, and Pressure of Democratic Leadership, eds. Eugene Borgida, Groups. 4th ed. New York: Crowell. Christopher M. Federico, and John L. Sullivan. Kinder, Donald R. 1986. “Presidential Charac- New York: Oxford University Press, 73–99. ter Revisited.” In Political Cognition: The 19th Hayes, Danny. 2005. “Candidate Qualities Annual Carnegie Symposium on Cognition, eds. through a Partisan Lens: A Theory of Trait Richard R. Lau and David O. Sears. Hillsdale, Ownership.” American Journal of Political Sci- NJ: Lawrence Erlbaum, 233–56. ence 49: 908–23. Kinder, Donald R. 1998. “Opinion and Action Hastie, Reid, and Bernadette B. Park. 1986.“The in the Realm of Politics.” In The Handbook of Relationship between Memory and Judgment Social Psychology. vol. 2, 4th ed., eds. Daniel T. Depends on Whether the Task Is Memory- Gilbert, Susan T. Fiske, and Gardner Lindzey. Based or On-Line.” Psychological Review 93: New York: McGraw-Hill, 778–867. 258–68. Kinder,DonaldR.,andLynnM.Sanders.1996. Hastie, Reid, and Nancy Pennington. 1989. Divided by Color: Racial Politics and Democratic “Notes on the Distinction between Memory- Ideals. Chicago: The University of Chicago Based Versus On-Line Judgments.” In On- Press. Line Cognition in Person Perception,ed.JohnM. Krosnick, Jon A. 1988. “The Role of Attitude Bassili. Hillsdale, NJ: Erlbaum, 1–17. Importance in Social Evaluation: A Study Huddy, Leonie, and Nayda Terkildsen. 1993. of Policy Preferences, Presidential Candidate “Gender Stereotypes and the Perception of Evaluations, and Voting Behavior.” Journal of Male and Female Candidates.” American Jour- Personality and Social Psychology 55: 196–210. nal of Political Science 37: 119–47. Krosnick, Jon A., and Donald R. Kinder. 1990. Hutchings, Vincent L., and Nicholas Valentino. “Altering the Foundations of Support for the 2004. “The Centrality of Race in American President through Priming.” American Political Politics.” Annual Review of Political Science 7: Science Review 84: 497–512. 383–408. Kruglanski, Arie W., and David Sleeth-Keppler. Iyengar, Shanto, and Donald R. Kinder. 1987. 2007. “The Principles of Social Judgment.” In News That Matters: Television and American Social Psychology: Handbook of Basic Principles. Opinion. Chicago: The University of Chicago 2nd ed., eds. Arie W. Kruglanski and E. Tory Press. Higgins. New York: Guilford Press, 116–37. Jacobs, Lawrence R., and Robert Y. Shapiro. Lau, Richard R., and David P. Redlawsk. 2006. 2000. Politicians Don’t Pander: Political Manip- How Voters Decide: Information Processing dur- ulation and the Loss of Democratic Responsiveness. ing Election Campaigns. New York: Cambridge Chicago: The University of Chicago Press. University Press. Johnson, Eric J., and J. Edward Russo. 1984. Lavine, Howard. 2002. “On-line Versus Me- “Product Familiarity and Learning New Infor- mory-Based Process Models of Candidate Candidate Impressions and Evaluations 199

Evaluation.” In Political Psychology, ed. Kristen Miller, Arthur H., Martin P. Wattenberg, and R. Monroe. Mahwah, NJ: Erlbaum, 225–47. Oksana Malanchuk. 1986. “Schematic Assess- Lodge, Milton, Kathleen M. McGraw, and Patrick ments of Presidential Candidates.” American Stroh. 1989. “An Impression-Driven Model of Political Science Review 79: 359–72. Candidate Evaluation.” American Political Sci- Mitchell, Dona-Gene. 2008. “It’s about Time: ence Review 83: 399–420. The Dynamics of Information Processing Lodge, Milton, and Marco Steenbergen. 1995. in Political Campaigns.” Unpublished dis- “The Responsive Voter: Campaign Informa- sertation, University of Illinois at Urbana- tion and the Dynamics of Candidate Evalua- Champaign. tion.” American Political Science Review 89: 309– Moskowitz, David, and Patrick Stroh. 1994. “Psy- 26. chological Sources of Electoral Racism.” Polit- Markus, Gregory B. 1982. “Political Attitudes dur- ical Psychology 15: 307–29. ing an Election Year: A Report on the 1980 Nimmo, Dan, and Robert L. Savage. 1976. Can- NES Panel Study.” American Political Science didates and Their Images: Concepts, Methods, and Review 76: 538–60. Findings. Santa Monica, CA: Goodyear. Mayhew, David. 1974. Congress: The Electoral Con- Page, Benjamin I. 1976. “The Theory of Politi- nection. New Haven, CT: Yale University Press. cal Ambiguity.” American Political Science Review McDermott, Monika L. 1998. “Race and Gender 70: 742–52. Cues in Low-Information Elections.” Political Page, Benjamin I., and Richard Brody. 1972. “Pol- Research Quarterly 51: 895–918. icy Voting and the Electoral Process: The McGraw, Kathleen M. 2003. “Political Impres- Vietnam War Issue.” American Political Science sions: Formation and Management.” In Hand- Review 66: 979–95. book of Political Psychology, eds. David Sears, Petrocik, John. 1996. “Issue Ownership in Pres- Leonie Huddy, and Robert Jervis. New York: idential Elections, with a 1980 Case Study.” Oxford University Press. American Journal of Political Science 40: 825–50. McGraw, Kathleen M., and Thomas Dolan. 2007. Rahn, Wendy M. 1993. “The Role of Partisan “Personifying the State: Consequences for Atti- Stereotypes in Information Processing about tude Formation.” Political Psychology 28: 299– Political Candidates.” American Journal of Polit- 328. ical Science 37: 472–96. McGraw, Kathleen M., Mark Fischle, and Karen Rahn, Wendy M., John H. Aldrich, and Eugene Stenner. 2000. “What People ‘Know’ Depends Borgida. 1994. “Individual and Contextual on How They Are Asked.” Unpublished Variations in Political Candidate Appraisal.” manuscript, Ohio State University. American Political Science Review 88: 193–99. McGraw, Kathleen M., Edward Hasecke, and Rahn, Wendy M., Jon A. Krosnick, and Marike Kimberly Conger. 2003. “Ambivalence, Un- Breuning. 1994. “Rationalization and Deriva- certainty, and Processes of Candidate Evalu- tion Processes in Survey Studies of Politi- ation.” Political Psychology 24: 421–48. cal Candidate Evaluation.” American Journal of McGraw, Kathleen M., and Valerie Hoekstra. Political Science 38: 582–600. 1994. “Experimentation in Political Science: Rapoport, Ronald B., Kelly L. Metcalf, and Jon A. Historical Trends and Future Directions.” In Hartman. 1989. “Candidate Traits and Voter Research in Micropolitics, eds. Michael X. Delli Inferences: An Experimental Study.” Journal of Carpini, Leonie Huddy, and Robert Y. Shapiro. Politics 51: 917–32. Greenwich, CT: JAI Press, 3–29. Redlawsk, David. 2001. “You Must Remember McGraw, Kathleen M., Milton Lodge, and Jeffrey This: A Test of the On-Line Model of Vot- Jones. 2002. “The Pandering Politicians of Sus- ing.” Journal of Politics 63: 29–58. picious Minds.” Journal of Politics 64: 362–83. Riggle, Ellen D., Victor C. Ottati, Robert S. Wyer, McGraw, Kathleen M., Milton Lodge, and Patrick James H. Kuklinski, and Norbert Schwarz. Stroh. 1990. “On-Line Processing in Candi- 1992. “Bases of Political Judgments: The Role date Evaluation: The Effects of Issue Order, of Stereotypic and Nonstereotypic Judgment.” Issue Salience and Sophistication.” Political Political Behavior 14: 67–87. Behavior 12: 41–58. Rivers, Douglas, and Morris P. Fiorina. 1991. McGraw, Kathleen M., and Neil Pinney. 1990. “Constituency Service, Reputation, and the “The Effects of General and Domain-Speciﬁc Incumbency Advantage.” In Home Style and Expertise on Political Memory and Judgment Washington Work: Studies of Congressional Pol- Processes.” Social Cognition 8: 9–30. itics,eds.MorrisP.FiorinaandDavidW. 200 Kathleen M. McGraw

Rohde. Ann Arbor: University of Michigan Stroud, Laura, Jack Glaser, and Peter Salovey. Press, 17–45. 2006. “The Effects of Partisanship and Can- Rosenberg, Shawn, Lisa Bohan, Patrick McCaf- didate Emotionality on Voter Preference.” ferty, and Kevin Harris. 1986. “The Image and Imagination, Cognition, and Personality 25: 25– the Vote: The Effect of Candidate Presenta- 44. tion on Voter Preference.” American Journal of Tomz, Michael, and Robert P. van Houweling. Political Science 30: 108–27. 2009. “The Electoral Implications of Can- Schneider, Monica, and Angela Bos. 2009.“It didate Ambiguity.” American Political Science Don’t Matter if You’re Black or White? Explor- Review 103: 83–98. ing the Content of Stereotypes of Black Politi- Williams, Linda F. 1990. “White/Black Percep- cians.” Paper presented at the annual meeting tions of the Electability of Black Political of the Midwest Political Science Association, Candidates.” National Political Science Review 2: Chicago. 145–64. Sigelman, Lee, Carol K. Sigelman, and C. Fowler. Zaller, John R. 1992. The Nature and Origins of 1987. “A Bird of a Different Feather? An Exper- Mass Opinion. New York: Cambridge Univer- imental Investigation of Physical Attractiveness sity Press. and the Electability of Female Candidates.” Zaller, John R., and Stanley Feldman. 1992. Social Psychological Quarterly 50: 32–43. “A Simple Theory of the Survey Response: Sigelman, Carol K., Lee Sigelman, Barbara Answering Questions versus Revealing Prefer- Walkosz, and Michael Nitz. 1995. “Black Can- ences.” American Journal of Political Science 36: didates, White Voters: Understanding Racial 579–616. Bias in Political Perceptions.” American Jour- Zebrowitz, Leslie A. 1994. “Facial Maturity and nal of Political Science 39: 243–65. Political Prospects: Persuasive, Culpable, and Stimson, James A., Michael B. MacKuen, and Powerful Faces.” In Beliefs, Reasoning, and Deci- Robert S. Erikson. 1995. “Dynamic Repre- sion Making: Psych-Logic in Honor of Bob Abelson, sentation.” American Political Science Review 89: eds. Roger C. Schank and Ellen Langer. Hills- 543–65. dale, NJ: Erlbaum, 315–45. CHAPTER 14

Media and Politics

Thomas E. Nelson, Sarah M. Bryner, and Dustin M. Carnahan

Nobody needs another summary of mass research, consider the media’s constant bug- media research right now; there are plenty bear: public perceptions of ideological bias in of fine, current surveys of the field (Kinder the news. The usual form of this complaint 2003). Our chapter discusses specifically how is that media organizations subtly stump for experimental research provides insight into liberal causes (Goldberg 2001). Although this the relationship between the media and the complaint often amounts to little more than political world. We are especially interested strategic bluster, it is conceivable that the in important questions that experimentation increasing differentiation of the media mar- is well suited to address. Experimentation has ketplace will encourage news organizations to been vital to the development of scholar- become more forthright in displaying overt ship in this area, but we should also recog- liberal or conservative commentary (Dalton, nize when it is best to step away and choose Beck, and Huckfeldt 1998). Experimentation another method. can certainly advance our understanding of Causation and experimentation go to- the consequences of media bias, ideological or gether hand in glove, and questions of otherwise. Scholars can rigorously examine, causation are paramount in both lay and using experimental methods, whether rival scholarly thought about the media (Iyengar coverage of current affairs contributes to dif- 1990). Questions about the social, economic, ferent perceptions and opinions among their and organizational factors that determine regular consumers (Entman 2004). Experi- mass media content are certainly fascinat- ments are not well suited, however, to inves- ing and relevant in their own right. One can tigating claims about the prevalence of such argue, however, that such questions eventu- biases. Experiments assume that meaningful ally beget questions about the ultimate impact variation in mass media content exists; estab- of that content on individuals and political lishing the nature and extent of such variation processes and institutions. falls to scholars using other methods, such as As an example of the promise and lim- content analysis (Patterson 1993; Hamilton itation of experimentation in mass media 2004).

201 202 Thomas E. Nelson, Sarah M. Bryner, and Dustin M. Carnahan

Convincing experimental work in social unambiguous. The real knots proved to be science frequently demands a leap of faith to selection effects and omitted variables, and take the stimulus and setting of the experi- these problems ultimately proved intolera- ment as valid instantiations of actual political ble for experimentally minded researchers. phenomena (Aronson 1977; Boettcher 2004). For instance, what about selective exposure? With research on the mass media, that leap is The claim is that people avoid material that more like a hop; the method and the phe- challenges their political preconceptions and nomenon seem made for each other. The instead feast on a diet of ideologically con- institutions of the mass media serve up, on genial media fare (Iyengar and Hahn 2009). a daily basis, morsels of content that the Selective exposure is a potential objection experimenter can excise and transport to the to any claim of long-term media exposure laboratory almost effortlessly. Furthermore, effects, political or otherwise. Many critics although many topics in political scholar- of violent entertainment media claim that ship are plagued by problems of causal order, they foster aggressive behavior in their con- media scholarship has fewer conundrums, at sumers, especially children (Eron 2001). A least with respect to conceptions of short- predictable rebuttal to this criticism, and the term media effects. Few of us doubt that cross-sectional data that support it, is selec- day-to-day variations in the topic, framing, tive exposure: violent television does not cre- style, and other features of news media con- ate aggressive behavior, but simply attracts tent properly belong in the cause column, and aggressive viewers. It takes experimentation all manner of individual-level political param- to answer the selectivity rebuttal and show eters belong in the effect column. conclusively that, yes, indeed, a steady diet Although made for each other, media of violent television can make an erstwhile and experimental research took their time to pacific child more aggressive. get acquainted. The earliest scholarly explo- As for omitted variables, this is a common rations of the political media were observa- shortcoming of observational research exam- tional. This material has been well reviewed ining the effects of media attention, namely, elsewhere; for our purposes, a few impor- agenda setting and priming. Nonexperimen- tant lessons from this era stand out. First is tal and quasi-experimental work indeed show the claim of minimal effects, which argues that media attention to an issue can heighten that the political impact of mass media mes- the public’s concern about it, while also caus- sages pales in comparison to more proxi- ing attitudes about the president to align mate influences such as friends and family. more closely with attitudes about that issue Furthermore, because many media messages (McCombs and Shaw 1972). The potential are open to multiple interpretations, viewers omitted variable is real-world change in the tend to see what they want to see, thereby urgency of that issue. Frequently, these phe- leading, at most, to a reinforcement of prior nomena covary, and it takes Herculean anal- views. Experimental work since this time has yses with fine-grained, time-series data to cast doubt on the blanket claim of minimal sort out cause from effect (Behr and Iyen- effects (or any blanket claim, really). Still, in gar 1985) – or experiments, wherein the methodological parlance, the claim of min- researcher has perfect knowledge of what pre- imal effects resonates as a cautionary tale cedes what, because he or she has designed it about limitations to the generalizability of that way. laboratory findings. A statistically significant Bartels (1993) calls mass media research effect observed in the tightly controlled con- “an embarrassment” because of its repeated ditions of the experimental laboratory might failure to demonstrate convincing effects, and be overwhelmed by multiple competing influ- his point remains relevant. The mass media ences in the real world (Kinder 2007). are, collectively, a huge institution that con- To say that the problems of recipro- sume and metabolize tremendous resources, cal causation are relatively mild for media and yet, clear, unambiguous media effects scholarship is not to say that causality is are difficult to spot. The minimal effects era Media and Politics 203 yielded valuable insights about the cognitive, 1. Media Species social, and institutional forces that check the independent effects of the mass media. The The very term media effects could be seen as media are, after all, but one player in the game an unconscionable overgeneralization: not all of contemporary politics. Still, the paucity media are alike. Still, most experiments con- of positive findings undoubtedly also reflects centrate on the effects of variation within weak research design. a source category (e.g., television news), Philosophers of science talk about the rather than across. With the proliferation importance of establishing a causal frame as of media sources, however, it becomes even a precondition for a properly posed scien- more urgent to address the question of tific question (White 1990). In other words, whether there is an identifiable, unique it is not enough to simply ask whether a cer- impact of a particular modality for transmit- tain variable is consequential; one must locate ting information. Observational studies relat- the potential consequence within a specific ing consumption habits to political outcomes set of background conditions. This leads us can certainly be helpful but eventually raise to question the claim about minimal effects the same thorny questions about selectivity. in circumstances where equal and opposite forces intersect (Zaller 1992). Showing pow- News erful and unambiguous media effects is just as challenging as providing unambiguous evi- In the pre-Internet age, research focused dence that campaigns matter (Vavreck 2009). on differences between newspaper and tele- This is in part because the causal frame is vision media effects. Much of this work vague. If campaigns do not matter, then why was inspired by concerns about differences don’t the talented people who run them sim- in political engagement and sophistication ply unilaterally disarm? They do not do so between those who take in a steady diet of rich because, under a counterfactual causal frame printed news compared to those who subsist of no opposition activity, one would surely on sweet but substanceless television. Neu- find that campaigns matter a lot. Much of the man, Just, and Crigler’s (1992) book Com- same is said about the criticism that research mon Knowledge demonstrates, by using the has exaggerated the political impact of com- intersection of survey and experimental data, munication frames because, in the real world, that although there is more information to the opposition offers its own frame, thereby be had from print sources, many people actu- canceling out the effect of the original frame. ally learn more from television. Television is It is doubtless the case that in many high- easy to understand and to decipher, whereas stakes political contests, two equally pow- print is harder and takes more effort. This is erful and widespread frames will compete not just selection bias – Neuman et al. found for public attention, resulting in little net that medium matters in the lab, a finding that movement in public opinion. Even in such is in line with both work by earlier schol- circumstances, it would be inaccurate to con- ars (Andreoli and Worchel 1978) and more clude that frames do not matter. Two equally recent research (Druckman 2003). powerful locomotives, placed nose to nose Sometimes content differences between and running at full speed, are not going to media are trivial, thus affording a stricter test move very far. It would be a misinterpreta- of modality effects. Stories in the printed ver- tion to claim, however, that neither one of sion of the New York Times are word-for- them is having an effect, as if the chemi- word identical to those in the online version, cal and mechanical forces that supply their but each modality contains something the power cease to exist. Experimentation allows other lacks. The printed version, for instance, us to create that counterfactual causal frame, provides cues to a story’s importance via enabling us to investigate the how and why placement and headline. Readers and view- of social phenomena, not merely their net ers follow the suggestions of editors and pro- impact. ducers and spend a lot more time on material 204 Thomas E. Nelson, Sarah M. Bryner, and Dustin M. Carnahan that is deemed important by the people who In addition, entertainment media – unlike put together the news product. So-called hard news – often seek to elicit sharp emo- indexed news sources instead provide a menu tional reactions in viewers, ranging from of story choices, and the consumer selec- sympathy to rage. Experiments are par- tion of those choices is presumably guided ticularly adept at shedding light on how by more idiosyncratic interests. From the these emotional responses affect public opin- standpoint of normative democratic theory, ion and political evaluations. Holbert and a case could be made that news producers Hansen (2008) offer one example of how should encourage citizens to eat their media emotional reactions – in this case, anger – vegetables because important stories are not can mediate the relationship between debate always entertaining. Absent cues to impor- exposure and perceived candidate perfor- tance, consumers may well adopt a more per- mance. When exposed to the movie Fahren- sonalized news consumption pattern, picking heit 9/11, participants reported increased and choosing items that suit their own indi- levels of anger toward both major party can- vidual whims, which is exactly what Althaus didates in the 2004 presidential election, and Tewksbury (2000) found. which then affected how participants evaluated each candidate’s debate performance. Kim and Vishak’s (2008) experimental Entertainment study presented evidence that the fact-based Television news and newspapers obviously nature of news programming promoted the differ in many ways, but they share the use of memory-based processing – leading goal of informing the consumer. What about to increased recall of factual information the political impact of other kinds of pro- and the use of that information in forming grams that primarily set out to entertain? and expressing political attitudes. Entertain- News magazines and soft news broadcasts, ment media, in contrast, promoted online political talk radio, late night comics, and information processing; that is, individuals even fictional primetime dramas often refer- could recall little factual detail from their ence political affairs and figures. And what program (an episode of The Daily Show) of the rise of feature-length political doc- but still responded to the information in umentaries such as Fahrenheit 9/11 and An their reported impressions. Concerns about Inconvenient Truth? They make no pretense the political enfeeblement of audiences have of neutrality, but seek to deliver a parti- become even more acute with the rise of soft san message with enough panache to hold news. Most of the extant work has been obser- the viewer’s attention for ninety minutes or vational (see Baum [2003] and Prior [2003] more. for an overview of the debate). Experimen- Because audiences welcome entertainment tal work clearly has as much to offer to this programming as a diversion from serious fare, debate as it did to Common Knowledge. they might process such content passively, abandoning the normal activities of delib- New Media eration and counterargument characteristic of the news viewer (Zaller 1992). Despite A truism holds that research lags behind these differences, or perhaps because of them, technological and social change, but a new exposure to entertainment media can exert generation of media scholars has embraced some of the same influences long attributed experimentation as the most appropriate to exposure to news media, specifically prim- method to examine the far-reaching impli- ing and agenda setting. Holbrook and Hill cations of information technology develop- (2005) show that exposing audiences to enter- ments. A notable example is the study of tainment media in the form of a crime drama computer-mediated (or online) discussion. leads participants to emphasize the impor- Political discussion and deliberation has long tance of addressing the issues of crime and been thought of as an asset for democracy, violence in their post-test responses. fostering understanding on divisive issues by Media and Politics 205 allowing citizens to defend their positions and Because the environment for these sources is explicate their reasoning (Goodin 2008). The extremely dynamic, little experimental work Internet provides an unprecedented virtual has yet been conducted, although we can public forum for diverse voices to assemble only suspect how promising work in this area and discuss pressing issues. However, some could be. Is information from these alter- scholars argued that online discussion is sub- native media sources treated with a higher stantively different from face-to-face (FTF) level of skepticism than media from tradi- deliberation in terms of both process and tional sources (see Cassese et al. [2010] for consequence. Does online political discussion an early exploration of this phenomenon)? differ from FTF discussion in ways that make Do people learn as much from these sources? it less valuable for the promotion and main- Does online content’s malleable form force tenance of democracy? us to question what it means to be a journal- For all their merits, observational studies ist? These questions could be explored exper- might not supply the best answers to these imentally, and to great end. questions because they depend on unreliable self-report measures of how people interact in online political conversations or how these 2. Media Effects behaviors differ from traditional FTF inter- Priming and Agenda Setting actions. Experimentation, in contrast, provides investigators with the opportunity to We now turn to phenomena that do not observe precisely how these types of discus- depend on a specific communication modal- sions vary – and therefore differ in terms of ity; that is, they can, in theory, occur whether their deliberative value – through allowing a the medium is newspaper, Internet, or smoke direct comparison between these two forms. signal. We define media effect in this context as Based on much of this experimental evi- a signature consequence of a distinctive prac- dence, deliberative theorists apparently have tice of mass media organizations that can – little to worry about with regard to the perhaps not easily – be separated from the future of political discussion in a new media mere informational content they report. For environment; individuals who participate in example, the mass media did not kill Michael online political discussions are more will- Jackson, but decisions they have made about ing to express their opinions as a result how to report his death – from the sheer vol- of anonymity (Ho and McLeod 2008) and ume of coverage to the balance taken between receive comparable levels of political infor- discussions of his public and private lives – mation through their conversations as FTF will shape his legacy far beyond the mere fact discussants. Still, the ills of democracy will of his passing. not all be solved by online discussion. For Experiments can take us far toward under- example, anonymity in discussion appears standing the consequences of such media to undermine the credibility of the source, decisions. If there is a single book that has thereby making participants less likely to done more than any other to accentuate trust and learn from their fellow discussants the advantages of experimentation, that book (Postmes, Spears, and Sakhel 2001). would be News That Matters (Iyengar and Other areas of research concerning media Kinder 1987). Interestingly, for all the ground effects are also affected by the rise of new broken by this volume, its theoretical claims media, including perhaps what we even are really extensions and qualifications of the choose to call media. An abundance of obser- minimal effects school. The two important vational studies show that younger gener- phenomena the book explores – agenda set- ations increasingly access information from ting and priming – are functions of media new media sources, including social network- attention, not media content. Furthermore, ing sites such as facebook.com, youtube.com, there are no direct effects on political opin- and a myriad of blogs (see Prensky 2001; ions of any sort. In keeping with the “cogni- Kohut 2008; Winograd and Hais 2009). tive miser” model of human social thought, 206 Thomas E. Nelson, Sarah M. Bryner, and Dustin M. Carnahan the book argues that the media do not really Huber and Lapinski’s (2008) null finding change attitudes, they simply stir the mixture does not therefore undermine the qualified of ingredients that constitute our opinions. claim that, for some people, implicit racial cues Research on priming and presidential eval- have meaningfully different consequences uation is largely disinterested in the content than explicit racial cues. By concentrating her of the primed stories. Just about any issue that experiment on a sample of such individu- the news highlights is likely to exert extraor- als, however, Mendelberg (2001) effectively dinary influence on judgments of presiden- exaggerated the generalizability of her find- tial competence. The racial priming litera- ings. This criticism about sample unrepresen- ture has a special concern with content. The tativeness has dogged laboratory experiments landmark study is Mendelberg’s (2001) The for decades (see Druckman and Kam’s chap- Race Card, which offers an implicit–explicit ter in this volume). In her defense, we should (IE) model of racial priming: 1) messages can note that Mendelberg anticipated the critique prime racial attitudes, making them assume a against using college sophomores by literally more prominent role in subsequent political taking her experiment on the road, toting her evaluations; and 2) implicit cues are usually equipment to the homes of her nonstudent more powerful primes than are explicit cues. participants. We hereby offer an extended digression into A third possible explanation for the fail- controversy over the racial priming hypoth- ure of conceptual replications – invoked by esis because it provides a textbook example Mendelberg (2008) in her rebuttal to Huber of experimentation’s vitality in political sci- and Lapinski (2008) – is the lack of correspon- ence, especially for investigating the conditions dence among separate operationalizations of under which a phenomenon occurs and the the key constructs: manipulations, measures, mechanisms responsible for its occurrence. moderators, and mediators. The crucial vari- The critiques of the IE model are largely able in this research is the explicitness of the methodological, not theoretical. In a large- racial cue. In theory, a number of messages scale, survey-based experiment, Huber and might activate racial sentiments; Mendelberg, Lapinski (2008) failed to replicate Mendel- in fact, lists seventeen experiments that rely berg’s (2001) findings that implicit racial cues on a wide range of stimuli. In theory, again, are more effective in activating racial sen- some of these cues will be recognized as mak- timents than are explicit cues. Huber and ing obvious appeals to racial antipathy, in vio- Lapinski’s experiments were conceptual, not lation of the norm of racial equality. Once direct replications of Mendelberg in terms messages rise to the level of explicitness, the of stimuli, procedures, measures, and sample. egalitarian norm allegedly kicks in and the Conceptual replications test the robustness of recipient suppresses any incipient prejudices. a finding, determining whether theoretically Mendelberg (2008), however, argues that irrelevant alterations in design, materials, or Huber and Lapinski’s (2008) experiments procedure moderate the effect. However, a fail to properly operationalize explicitness by conceptual replication failure is an ambigu- using two stimuli that prime racial feelings ous signal. It could mean that the theory is not in equal measure but above and below the an authentic representation of actual political threshold of explicitness. Huber and Lap- processes, or that the phenomenon revealed inski wisely include a manipulation check, by the original research is real, but narrow designed to measure whether the stimuli pro- and trivial. Huber and Lapinski argue the lat- vides a legitimate test of the IE hypothesis, by ter, attributing Mendelberg’s positive find- asking participants whether they believed that ings to her sample, which they argued to be the messages were “good for democracy.” especially likely to exhibit the IE difference. Although participants exposed to explicit ads Huber and Lapinski’s sample, which did not evaluated explicit ads as “somewhat bad” to a display any equivalent moderating effect for relatively greater degree than those exposed the explicitness of the racial signal, was more to the implicit ads (between fifteen and twenty representative of the general population. percent in the implicit condition compared to Media and Politics 207 twenty-five percent in the explicit condition), questions, but they can help us understand Mendelberg complains that the absolute pro- whether such variations make any difference. portion is small and changes little, suggesting The question of whether framing affects opin- that a majority of respondents in the explicit ion and behavior is often followed, in the condition do not feel the irresistible tug of next breath, with the question of when it car- racial egalitarianism. ries such effects (Druckman 2001). This is a question concerning the boundary conditions governing framing effects. Most of us believe Framing framing happens, but surely it does not always Framing research puts media content front happen. and center by claiming it is not simply whether an issue is covered, but how it is covered that matters. As the debate over implicit racial 3. Message Processing cues illustrates, so much of an experiment’s value depends on the translation of an abstract Media scholars are rarely content simply to concept into a concrete treatment – that investigate whether a particular effect hap- is, the operationalization of the independent pens, or even when; they also want to know variable. At a theoretical level, researchers why. If experimentation’s principal value to studying framing must tame an unruly con- the social sciences is investigating causa- cept that, however intuitive it might seem, is tion, then a close second is surely investigat- defined in different ways by different scholars, ing psychological mechanism. Observational both across and within fields (Schaffner and research has made strides in incorporating Sellers 2009). Even if a consensus could be measures that reveal, if crudely, psycholog- obtained on the dictionary definition of fram- ical processes (e.g., reaction time measures ing, there are many variations in the experi- in telephone interviews). Still, the experimental operationalization of this concept. We menter’s kit overflows with specialized tools argue that messages that convey entirely dif- for revealing what takes place between stim- ferent objective information should not be ulus and response (but see Bullock and Ha’s considered alternative frames. By analogy, chapter in this volume). think about two different ways of changing Scholars with a rationalist inclination visual perspective. One could look out in make the unsurprising but still important a certain direction (say, southeast), describe claim that the chief psychological effect of what one sees, and then pivot on one’s foot media consumption is learning. In other to a different compass point and describe that words, the media teach us what we need view. The two descriptions will likely be quite to know to make sensible political deci- different because they will refer to different sions (Lau and Redlawsk 2006). Psycholog- sets of objects. Next, consider slowly circling ical theory points to other processes that around a stationary object and describing it are less obvious, while helping unpack the from different vantage points. Each descrip- generic learning effect. Various dual mode tion will refer to the same object, but will theories suggest two broad categories of still vary because different features of the psychological response to communication, object will come into prominence. Alternative especially to persuasive messages: 1)amore frames do not change the object of descrip- thoughtful, effortful “central,” “systematic,” tion, merely the way it is characterized. or “piecemeal” route; and 2) a quicker, super- We can pose many fascinating questions ficial “peripheral,” “heuristic,” or “stereo- about why one news organization might typical” route (Cacioppo and Petty 1982; cover an issue in one particular way, whereas White and Harkins 1994; Mutz and Reeves another organization might cover the same 2005). Gone are the days when media schol- issue quite differently (Price and Tewksbury ars assumed uniform effects across the general 1997). Experiments would be a very poor public. Even studies employing the proverbial choice of method to gain traction on such college sophomore have shown important 208 Thomas E. Nelson, Sarah M. Bryner, and Dustin M. Carnahan moderator effects. Individual traits and qual- ulating the amount of information, instead ities, such as political knowledge and sophis- manipulate the interactivity of the web site, tication (Nelson, Oxley, and Clawson 1997), finding that more interactive web sites lead trust in media (Gunther 1992), value orienta- to higher levels of processing (Sicilia, Ruiz, tion (Johnson 2007), need for cognition and and Munuera 2005). They suggest that the evaluation (Neuman et al. 1992), and race relationship between number of hyperlinks (White 2007) significantly moderate media and depth of processing may be nonlinear, effects. Braverman (2008) examines involve- and that if individuals are presented with too ment both with the material and the way the much information, then they shut down. message is transmitted, and finds an interactive effect between the two. Experiments thus provide an excellent way to look at indi- 4. External Validity vidual differences in processing across media sources, something that survey research can- A method that maximizes our confidence in not tap in such a controlled manner. causal hypotheses is not much good if all we Psychological theorizing about framing learn about is what happens in our labora- initially posited that it functions much like tory. Generalizability has several dimensions, priming; that is, frames subtly draw our lim- including mundane realism and psychological ited attentional resources toward some con- realism. We can define mundane realism as siderations and away from others. Subsequent verisimilitude: the correspondence between research has expanded the set of psycholog- features and procedures of the experiment ical processes implicated by framing. Price and those prevalent in the real world. Psy- and Tewksbury (1997) argue that frames alter chological realism refers to the engagement the applicability of stored information to the and arousal of similar psychological processes framed issue. Under certain frames, a cogni- to those that prevail in analogous situations in tion may no longer be perceived to fit the the real world. Solomon Asch’s (1955) classic issue. Nelson, Clawson, and Oxley (1997) experiments on conformity resemble nothing believe that frames also operate in a more we know in the real world, and yet it cannot mindful way by affecting judgments of the be denied that conformity pressure was pos- importance or relevance of cognitions. itively suffocating for the subjects in those Research on message processing surfaces famous studies. in scholarship on new media. The distinc- The importance of this distinction is appa- tive qualities of new communication forms – rent in many of the research traditions on dynamism, decentralization, nonlinearity, mass media effects, including scholarship on and the fading boundary between producer modality effects on political learning. To and consumer – suggest a sea change in the provide the proper causal frame, we should acquisition and use of information. Hyper- ask not whether newspaper consumers are media structure is said to mimic cognitive more knowledgeable than, say, those who get structure at the individual level, with seman- their news almost exclusively from television; tic nodes linked by association to form a instead, we need to ask whether the same indi- conceptual web. This distinctive structure, in vidual would learn more or less if he or she theory, facilitates the absorption and reten- got his or her news from a different source. tion of hypermedia content (Eveland and Framed in this manner, experiments Dunwoody 2002). Wise, Bolls, and Schaefer become a natural research choice, but that (2008) manipulate the amount of content is just the first of our decisions. We could, available to a web site visitor and, by taking for example, take representative samples of readings of the heart rates of the par- coverage from two distinct media, randomly ticipants, find that in a richer media assign research participants to receive one set environment (one that displays more stories), or the other, and compare their knowledge participants use higher levels of cognitive and understanding of politics following these resources. Other scholars, rather than manip- treatments. But is this the best way to go Media and Politics 209 about it? That partly depends on what we experiments conducted under less externally mean by a modality effect, and this is where valid conditions? There are, of course, strong questions about generalizability emerge in claims about the superiority of evidence col- high relief. From a strict mundane realism lected under conditions of greater mundane perspective, the aforementioned approach is realism; however, for the most part, these best. But we can take a sharp scalpel and are speculations or common sense bromides. separate questions about the effects of typi- Critics who fret about the verisimilitude of cal coverage across diverse media from ques- social science experiments should take note tions about the inherent differences between that cutting-edge experimental physics uses media. Media differ not only in the obvious laboratory conditions that have not occurred perceptual qualities, but also in characteris- anywhere in the universe since approximately tic ways that journalism is practiced. These half of a second after the commencement differences are not inherent to the perceptual of the Big Bang (Kaku 2008). The little qualities of the media; rather, they amount to systematic evidence we do have does not institutional folkways. As we all know, news- strongly support the case that more real- papers present far more information than a ism equals more validity (Anderson, Lindsay, typical news broadcast. It does not have to be and Bushman 1999). Investments in exter- this way, but it is. nal validity could be costly in ways beyond So should we remain faithful to inher- simply the expenditure of resources. Exper- ent differences or typical differences? Nei- imenters did not repair to the laboratory ther answer is obviously better than the just because it was close to their offices; the other because they represent equally legiti- laboratory setting is simply a natural exten- mate framings of the question about media sion of the logic of experimental control. effects. The former formulation is likely to Isolating the effect of one variable requires appeal more to media researchers with a prag- controlling for the effects of systematic and matic political orientation: they want to know unsystematic error. The former contributes what happens in the world of actual politics, to type I error (false positives) and the lat- and how characteristic practices leave their ter to type II (false negatives). Laboratory mark. Media researchers with deeper psycho- settings, all things being equal, help mini- logical interests will want to know what it is mize the impact of variables that would water about television that leads to differing lev- down the impact of the experimental stim- els of information absorption and refinement uli and lead to a false rejection of the null relative to print or other media, irrespective hypothesis. of the tendency for print media to include a This is the great advantage of Iyengar greater overall volume of information. and Kinder’s (1987) studies, to some. That The external validity gauntlet was their experiments yielded positive evidence thrown down with flourish by Iyengar and of agenda setting and priming, despite the Kinder’s (1987) seminal experiments. These potential distractions of their soft laboratory researchers invested great effort in putting setting, suggests that such phenomena are together stimuli, procedures, and a laboratory likely to have real impact in the real world. environment that closely mimicked the typi- Yet, perhaps even a soft laboratory is not cal real-world news consumption experience. enough. Paul Sniderman and Don Kinder This standard for mundane realism has been are rightly lauded as pioneers in apply- matched, but never exceeded (Brader 2005). ing experimental methods to the study of Was all this effort worth it? One of the most political communication, and yet both have important reasons, in our judgment, that the publicly criticized framing research for its work has had such lasting impact is because excessive reliance on (laboratory) experimen- of the exacting measures taken to anticipate tation. The critiques boil down, once again, to and preempt criticism on external validity mundane realism. Sniderman complains that grounds. Such perceptions matter, but are the typical experimental treatment makes cru- they based on a myth about the weakness of cial departures from real-world frames, and 210 Thomas E. Nelson, Sarah M. Bryner, and Dustin M. Carnahan

Kinder (2007) says that the laboratory setting We hasten to add that confidence in causal is too unreal because it rivets the participant’s inference does not require a laboratory set- attention to stimuli that, outside the labora- ting. The resourceful, inventive researcher tory, might never register with us. can conduct field experiments that have all the Such a view brings us full circle to ques- rigor of a classic experimental design but add a tions about the ultimate scientific purpose of naturalistic setting and/or manipulation. Not experimental work. To paraphrase McGuire only does a well-designed field experiment (1983), experiments are better attuned to negate external validity concerns, but it also investigating what can happen than what provides an avenue toward greater precision will happen. They are singularly excellent in estimating the magnitude of relationships tools for theory building, which is a realm between variables of interest. In other words, of abstraction and ideal. Core concepts and it not only helps answer the “So what?” ques- vital mechanisms are thrown into high relief, tion, but also the “How much?” question. whereas nuisance factors, complications, and Nevertheless, the number of field experi- contingencies are set aside for another day. As ments is dwarfed by that of laboratory inves- a class, experimenters are unconcerned with tigations because the obstacles facing the field achieving an exact calibration of the magni- experimenter are formidable (for an extended tude of the effects of their variables outside discussion, see Gerber’s chapter in this vol- the laboratory. For all the attention given ume). The literature on media effects is as to mundane realism in their studies, Iyen- guilty as any, suggesting that this might be gar and Kinder (1987) never issued any pre- a growth area for enterprising researchers. cise claims about how much the media can One exemplary effort was inspired by the affect the public’s agenda. For example, how civic journalism movement, which seeks to much media emphasis would be needed, in move journalistic practice from the mere terms of amount and prominence of news recounting of facts to the promotion of pub- stories, to move global warming to the top lic discussion and participation. Researchers of the national agenda? We doubt anyone is collaborated with media outlets, who system- prepared to make precise estimates of such atically varied their production to represent quantities on the basis of laboratory findings, “old” and “new” journalistic styles, and con- however realistic the conditions under which sumers of these different journalistic products they were obtained. were later surveyed along various dimensions Furthermore, increasing the external such as intent to vote, engagement in political validity of an experiment can be associated discussion, and involvement with civic groups with an increase in ethical concerns. Many (Denton and Thorson 1998). media experiments involve deceiving, how- Once in a great while, opportunistic inves- ever temporarily, the study participants. For tigators can even take advantage of naturally the experiment to capture how a person might occurring manipulations to conduct a quasi- really react to a piece of media, it seems log- experiment. Such investigations typically lack ical that the person should believe that the random assignment and/or carefully con- treatment is real. However, this brings with trolled manipulations, but they more than it a wealth of other problems – how long last- make up for these shortcomings by supplying ing are media effects, even those done in the systematic observations of the effects of tangi- lab? If they last long enough for the per- ble variation in the phenomena we care about. son to believe that they are real, then we An example is Mondak’s (1995) study of the might expect that the participant’s attitude consequences of the 1992 Pittsburgh newspa- is somehow affected. Given the possibility per strike. Mondak shows that the strike did of potentially long-lasting effects, we believe not cause the good people of Pittsburgh to that researchers should carefully consider the suffer decrements in national or international ethical implications of experiments high in political awareness relative to demographi- external validity. cally comparable residents of Cleveland. Media and Politics 211

5. Conclusion band, blogs, RSS feeds, podcasts, social networking web sites, and the like, they did not The media never sleep, nor does innova- tell us. It does seem safe, if a little cow- tion in communications technology. Today’s ardly, to predict that experimentation will be consumer of the latest in trendy communi- a vital part of the scholarly analysis of such cation toys is tomorrow’s befuddled techno developments. has-been, beseeching his or her teenager to “make this damned thing work.” The media technology universe has changed so drasti- References cally that we will likely witness the demise of the printed daily newspaper within years, Althaus, Scott L., and David Tewksbury. 2000. not decades. We must wonder if media schol- “Patterns of Internet and Traditional News arship’s conventional wisdom will similarly Media Use in a Networked Community.” Polit- obsolesce. ical Communication 17: 21–45. Fortunately, clear thinking about concepts Anderson, Craig A., James J. Lindsay, and Brad 1999 and processes never goes out of style. The J. Bushman. . “Research in the Psy- chological Laboratory: Truth or Triviality?” accelerating pace of change in information Current Directions in Psychological Science 8: technology is unnerving, but we have to be 3–9. careful about confusing superficial with fun- Andreoli, Virginia, and Stephen Worchel. 1978. damental change. What is really changing – “Effects of Media, Communicator, and Mes- the sheer amount of available information sage Position on Attitude Change.” Public Opin- and the ease with which it can be accessed ion Quarterly 42: 59–70. by ordinary people? The size and breadth of Aronson, Elliot. 1977. “Research in Social Psy- the communication networks commanded by chology as a Leap of Faith.” Personality and citizens? The practice of journalism by peo- Social Psychology Bulletin 3: 190–95. 1955 ple who are not professional journalists? And Asch, Solomon E. . “Opinions and Social 193 31 35 what are the consequences of such changes? Pressure.” Scientific American : – . Bartels, Larry M. 1993. “Messages Received: The Just as the Internet has opened up opportu- Political Impact of Media Exposure.” American nities for new forms of criminal behavior, so, Political Science Review 87: 267–85. too, is it likely that the political consequences Baum, Matthew A. 2003. “Soft News and Political of innovations in information technology will Knowledge: Evidence of Absence or Absence not all be beneficial. of Evidence?” Political Communication 20: 173– Sound science does not always require a 90. good theory, but it is not a bad place to start, Behr, Roy L., and Shanto Iyengar. 1985. “Tele- and there are plenty of fascinating and per- vision News, Real-World Cues, and Changes tinent theories of mass communication from in the Public Agenda.” Public Opinion Quarterly 49 38 57 which to choose. Perhaps, with the diversi- : – . 2004 fication and personalization of mass media Boettcher, William A., III . “The Prospects sources, strong priming and agenda-setting for Prospect Theory: An Empirical Evalua- tion of International Relations Applications of effects will become rare. Perhaps the rise of Framing and Loss Aversion.” Political Psychology user-generated content will usher in a new 25: 331–62. era of political trust and involvement, or per- Brader, Ted. 2005. “Striking a Responsive Chord: haps ideological polarization will accelerate How Political Ads Motivate and Persuade Vot- with the proliferation of partisan information ers by Appealing to Emotions.” American Jour- sources. nal of Political Science 49: 388–405. It seems pointless to speculate about the Braverman, Julia. 2008. “Testimonials versus future direction of information technology Informational Persuasive Messages: The Mod- change and the havoc it will create for pol- erating Effect of Delivery Mode and Personal 35 666 itics. If the giants of twentieth-century media Involvement.” Communication Research : – 94. research anticipated YouTube, mobile broad- 212 Thomas E. Nelson, Sarah M. Bryner, and Dustin M. Carnahan

Cacioppo, John T., and Richard E. Petty. 1982. into News. Princeton, NJ: Princeton University “The Need for Cognition.” Journal of Personal- Press. ity and Social Psychology 42: 116–31. Ho, Shirley S., and Douglas M. McLeod. 2008. Cassese, Erin, Christopher Weber, David Hauser, “Social-Psychological Influences on Opinion and Steven Nutt. 2010. “Media, Framing, Expression in Face-to-Face and Computer- and Public Opinion: How Does YouTube Fit Mediated Communication.” Communication In?” Paper presented at the annual meeting Research 35: 190–207. of the Midwest Political Science Association, Holbert, R. Lance, and Glenn J. Hansen. 2008. Chicago. “Stepping beyond Message Specificity in the Dalton, Russell J., Paul A. Beck, and Robert Huck- Study of Emotion as Mediator and Inter- feldt. 1998. “Partisan Cues and the Media: Emotion Associations across Attitude Objects: Information Flows in the 1992 Presidential Fahrenheit 9/11, Anger, and Debate Superior- Election.” American Political Science Review 92: ity.” Media Psychology 11: 98–118. 111–26. Holbrook, Robert A., and Timothy Hill. 2005. Denton, Frank, and Esther Thorson. 1998. “Agenda-Setting and Priming in Prime Time “Effects of a Multimedia Public Journalism Television: Crime Dramas as Political Cues.” Project on Political Knowledge and Attitudes.” Political Communication 22: 277–95. In Assessing Public Journalism, eds. Edmund B. Huber, Gregory A., and John S. Lapinski. 2008. Lambeth, Philip Meyer, and Esther Thorson. “Testing the Implicit–Explicit Model of Racial- Columbia: University of Missouri Press, 143– ized Political Communication.” Perspectives on 78. Politics 6: 125–34. Druckman, James N. 2001. “The Implications Iyengar, Shanto. 1990. “The Accessibility Bias in of Framing Effects for Citizen Competence.” Politics: Television News and Public Opinion.” Political Behavior 23: 225–56. International Journal of Public Opinion Research 2: Druckman, James N. 2003. “The Power of Televi- 1–15. sion Images: The First Kennedy-Nixon Debate Iyengar, Shanto, and Kyu S. Hahn. 2009. “Red Revisited.” Journal of Politics 65: 559–71. Media, Blue Media: Evidence of Ideological Entman, Robert M. 2004. Projections of Power: Selectivity in Media Use.” Journal of Commu- Framing News, Public Opinion, and U.S. For- nications 59: 19–39. eign Policy. Chicago: The University of Chicago Iyengar, Shanto, and Donald R. Kinder. 1987. Press. News That Matters: Television and American Eron, Leonard D. 2001. “Seeing Is Believing: How Opinion. Chicago: The University of Chicago Viewing Violence Alters Attitudes and Aggres- Press. sive Behavior.” In Constructive and Destructive Johnson, Marcia K. 2007. “Reality Monitoring Behavior: Implications for Family, School, and Soci- and the Media.” Applied Cognitive Psychology 21: ety, eds. Arthur C. Bohart and Deborah J. 981–93. Stipek. Washington, DC: American Psycho- Kaku, Michio. 2008. PhysicsoftheImpossible:A logical Association, 49–60. Scientific Exploration into the World of Phasers, Eveland, William P., Jr., and Sharon Dunwoody. Force Fields, Teleportation, and Time Travel.New 2002. “An Investigation of Elaboration and York: Doubleday. Selective Scanning as Mediators of Learning Kim, Young Mie, and John Vishak. 2008. “Just from the Web versus Print.” Journal of Broad- Laugh! You Don’t Need to Remember: The casting and Electronic Media 46: 34–53. Effects of Entertainment Media on Political Goldberg, Bernard. 2001. Bias: A CBS Insider Information Acquisition and Information Pro- Exposes How the Media Distort the News.Wash- cessing in Political Judgment.” Journal of Com- ington, DC: Regnery. munication 58: 338–60. Goodin, Robert E. 2008. Innovating Democracy: Kinder, Donald R. 2003. “Communication and Democratic Theory and Practice after the Delibera- Politics in the Age of Information.” In Oxford tive Turn. New York: Oxford University Press. Handbook of Political Psychology,eds.DavidO. Gunther, Albert C. 1992. “Biased Press or Biased Sears, Leonie Huddy, and Robert Jervis. New Public? Attitudes towards Media Coverage of York: Oxford University Press, 357–93. Social Groups.” Public Opinion Quarterly 56: Kinder, Donald R. 2007. “Curmudgeonly Advice.” 147–67. Journal of Communication 57: 155–62. Hamilton, James. 2004. All the News That’s Fit Kohut, Andrew. 2008. “The Internet Gains in to Sell: How the Market Transforms Information Politics.” Pew Internet and American Life Media and Politics 213

Project. January 11. Retrieved from www. Group Behavior.” Personality and Social Psychol- pewinternet.org/Reports/2008/The-Internet- ogy Bulletin 27: 1243–54. Gains-in-Politics.aspx (November 7, 2010). Prensky, Marc. 2001. “Digital Natives, Digital Lau, Richard R., and David P. Redlawsk. 2006. Immigrants.” On The Horizon 9: 1–6. How Voters Decide: Information Processing dur- Price, Vincent, and David Tewksbury. 1997. ing Election Campaigns. Cambridge: Cambridge “News Values and Public Opinion: A Theoret- University Press. ical Account of Media Priming and Framing.” McCombs, Maxwell E., and Donald L. Shaw. In Progress in the Communication Sciences, eds. 1972. “The Agenda-Setting Function of the G. Barnett and F. J. Boster. Greenwich, CT: Mass Media.” Public Opinion Quarterly 36: 176– Ablex, 173–212. 87. Prior, Markus. 2003. “Any Good News in Soft McGuire, William J. 1983. “A Contextualist The- News? The Impact of Soft News Preference on ory of Knowledge: Its Implications for Innova- Political Knowledge.” Political Communication tion and Reform in Psychological Research.” 20: 149–71. In Advances in Experimental Social Psychology. Schaffner,BrianF.,andPatrickJ.Sellers,eds. vol. 16, ed. Leonard Berkowitz. San Diego: 2009. Winning with Words: The Origins and Academic Press, 1–47. Impact of Framing. New York: Routledge. Mendelberg, Tali. 2001. The Race Card: Cam- Sicilia, Maria, Salvador Ruiz, and Jose L. Munuera. paign Strategy, Implicit Messages, and the Norm 2005. “Effects of Interactivity in a Web Site: of Equality. Princeton, NJ: Princeton Univer- The Moderating Effect of Need for Cogni- sity Press. tion.” Journal of Advertising 34: 31–44. Mendelberg, Tali. 2008. “Racial Priming: Issues in Vavreck, Lynn. 2009. The Message Matters: The Research Design and Interpretation.” Perspec- Economy and Presidential Campaigns. Princeton, tives on Politics 6: 135–40. NJ: Princeton University Press. Mondak, Jeffery J. 1995. “Newspaper and Political White, Ismail. 2007. “When Race Matters and Awareness.” American Journal of Political Science When It Doesn’t: Racial Group Differences 39: 513–27. in Response to Racial Cues.” American Politi- Mutz, Diana C., and Byron Reeves. 2005.“The cal Science Review 101: 339–54. New Videomalaise: Effects of Televised Inci- White, Paul H., and Stephen G. Harkins. 1994. vility on Political Trust.” American Political Sci- “Race of Source Effects in the Elaboration ence Review 99: 1–15. Likelihood Model.” Journal of Personality and Nelson, Thomas E., Rosalee A. Clawson, and Zoe Social Psychology 67: 790–807. M. Oxley. 1997. “Media Framing of a Civil Lib- White, Peter A. 1990. “Ideas about Causation in erties Conﬂict and Its Effect on Tolerance.” Philosophy and Psychology.” Psychological Bul- American Political Science Review 91: 567–83. letin 108: 3–18. Nelson, Thomas E., Zoe M. Oxley, and Rosalee Winograd, Morely, and Michael D. Hais. 2009. A. Clawson. 1997. “Toward a Psychology of Millennial Makeover: MySpace, YouTube, and the Framing Effects.” Political Behavior 19: 221–46. Future of American Politics. New Brunswick, NJ: Neuman, W. Russell, Marion R. Just, and Ann N. Rutgers University Press. Crigler. 1992. Common Knowledge: News and the Wise, Kevin, Paul D. Bolls, and Samantha R. Construction of Political Meaning. Chicago: The Schaefer. 2008. “Choosing and Reading Online University of Chicago Press. News: How Available Choice Affects Cognitive Patterson, Thomas E. 1993. Out of Order.New Processing.” Journal of Broadcasting and Elec- York: A. Knopf. tronic Media 52: 69–85. Postmes, Tom, Russell Spears, and Khaled Sakhel. Zaller, John R. 1992. The Nature and Origins of 2001. “Social Inﬂuence in Computer-Mediated Mass Opinion. New York: Cambridge Univer- Communication: The Effects of Anonymity on sity Press. CHAPTER 15

Candidate Advertisements

Shana Kushner Gadarian and Richard R. Lau

Nowhere can democracy be better seen “in campaign. To simplify, let us assume a can- action” than during political campaigns. As didate’s goal is to win the election, but he or Lau and Pomper (2002) argue, “Democracy she is facing one or more opponents who have is a dialogue between putative leaders and the same goal and therefore want to defeat citizens. Campaigns provide the most obvi- him or her. All candidates must decide what ous and the loudest forums for this dialogue. strategy to follow in order to maximize their Candidates try to persuade voters to cast chances of winning. But all candidates face a ballot and to support their cause. Voters resource limits that put very real constraints respond by coming to the polls and select- on what it is possible to do. Therein lies the ing their preferred candidates” (47). Can- rub. Candidates must decide how they can get didates make their arguments in speeches the biggest bang for their buck. Of the myr- at campaign rallies and on their Web sites, iad strategies they could follow, which is most but those venues are primarily experienced likely to result in electoral victory? For the by the most committed of partisans. It is past ﬁfty years, the medium of choice has been only through their campaign advertisements television for those candidates with sufﬁcient that candidates have any chance of reach- resources to afford televised ads. But even ing uncommitted voters. And in times of limiting attention to television only slightly even very approximate party balance, it is reduces the options available. uncommitted voters who usually determine Now reverse the perspective to that of election outcomes. Hence, political ads are a citizen in a democracy living through an arguably the vehicle through which democ- election campaign. The choices the candi- racy operates. dates collectively make about what campaign Consider the choices facing a political can- strategies they want to follow determine the didate at the outset of a democratic election campaign environment available to the voter. History shapes the electoral context as well: a We thank Kevin Arceneaux and the book’s editors for longtime incumbent will be better known (for comments on earlier drafts of this chapter. better or worse) at the outset of a campaign,

214 Candidate Advertisements 215 whereas a candidate mounting his first cam- number of studies has focused on one ques- paign starts with almost a blank slate. But the tion: the effectiveness of negative, as opposed citizen still has a great deal of control over to positive, political ads. This same question how much of that campaign he or she wants has motivated the great bulk of the nonex- to experience. Some people actively seek out perimental studies as well. We can use this as much information as possible about all can- question to illustrate the difficulty of trying didates running, others do everything they to determine the effectiveness of different can to avoid anything vaguely political, and campaign strategies by studying real polit- still others are so busy taking care of their ical campaigns. We hope that researchers families that they just do not have much time studying political campaigns will begin sys- for anything else. Some people (roughly forty tematically studying many aspects of political percent in the United States) know how they campaigns in addition to their tone, but for are going to vote before the campaign even now, by necessity, we are mostly limited to begins; others (another forty percent, more explorations of this one specific question. or less) know that they will not bother vot- Experiments overcome the most vexing ing. The remainder – approximately twenty difficulties, but they do so by creating an arti- percent – will often vote given the added ficial situation that could, in several impor- stimulation during presidential elections, but tant ways, crucially misrepresent (or limit) will probably not bother to vote during any the very phenomenon the scientist is trying less intense political campaign. How they will to study. At the least, all researchers study- vote is much more uncertain and potentially ing campaign advertisements experimentally open to influence from the campaign. face a number of “practical” challenges that These two sets of factors – the choices determine the nature of the experiment they made by candidates at the outset of a cam- run, affect the causal inferences that can be paign (which can be modified during the drawn from the research, and influence the course of an extended campaign) and the breadth of situations to which the findings can choices made by voters during the course of be generalized. We conclude this chapter by the campaign – are what make it so chal- discussing a number of important questions lenging to study the effects of actual political about candidate advertisements that could be campaigns. Candidate X spends all available explored experimentally but so far have not resources to convince as many citizens as pos- been sufficiently addressed. sible to vote for him or her, whereas candidate Y is simultaneously doing everything he or she can think of (and afford) to counter what 1. Methodological Difficulties in candidate X does and to convince those same Studying Real Campaigns through citizens to vote for him or her. In the mean- Observational Methods time, many citizens are either totally obliv- ious of anything political going on around Fifty years ago when social science research them or are aware of an ongoing campaign on media effects was in its infancy, Carl but are doing their best to avoid it. There is Hovland (1959) noted a “marked differ- little wonder that political scientists might try ence” in the picture of communication to eliminate many of these complications by effects obtained from experimental and sur- turning to experiments to study the effects of vey methods, with experiments indicating candidate advertisements. the possibility of “considerable modifiability This chapter reviews the experimental of attitudes” through exposure to televised literature on the effects of candidate adver- communications, whereas correlational stud- tisements, primarily on vote choice and ies usually find that “few individuals . . . are candidate evaluation.1 By far, the largest affected by communications” (8). Hovland focused on several important methodological 1 1 We concentrate only on ads from candidates running reasons for the different results: ) the audi- for office and ignore interest group ads. ence for many real-world communication 216 Shana Kushner Gadarian and Richard R. Lau efforts such as political campaigns are highly wisdom about negative advertising is that it selected, so the communicator often ends is effective but risky. It can quickly and rela- up preaching to a choir that already agrees tively inexpensively lower evaluations of the with the message; 2) actual exposure to the target of the attacks, but may result in a back- communication is almost guaranteed in the lash that lowers evaluations of the sponsor lab, whereas in the real world even peo- of the attacks as well, leaving the net ben- ple who are exposed to a candidate adver- efit somewhat up in the air (Lau, Sigelman, tisement may well ignore it; 3) subjects in and Brown Rovner 2007). However, candi- experiments typically view communications dates who expect to lose or who find them- in social isolation, whereas real-world com- selves behind in a race, and candidates with munication efforts are experienced in a social fewer resources than their opponent, have few context that usually reinforces prior atti- viable options except to attack. This means tudes and thus resists change; 4) in the lab- that negative advertising is a strategy often oratory, the dependent variable is typically chosen by likely losers, which makes it dif- gathered immediately after the communi- ficult to determine the effectiveness of neg- cation, whereas with observational methods ative campaigning – or any other campaign the dependent variable (e.g., voting) may strategy that candidates choose with some not occur until days or even months after a knowledge of the likely campaign outcome. communication is delivered; and 5) labora- Did a candidate lose an election because he tory research typically studies less involving or she chose to attack his or her opponent, issues, whereas observational studies typically or did a candidate choose to attack his or focus on more important topics. Over the her opponent because he or she was going succeeding fifty years, improved theory and to lose anyway and no other strategy gave methods have provided numerous examples him or her a better chance of reversing his of observational studies documenting rather or her fortune? Maybe the candidate would substantial effects of political communication have lost by more votes had he or she cho- (see Kinder 2003) so that Hovland’s starting sen some other campaign strategy. Table 15.1 point is no longer true, but these method- lists these six problems and their methodolog- ological issues continue to describe important ical consequences. differences between experimental and obser- In statistical parlance, the problem is that vational studies of candidate advertisements. choice of campaign strategy is endogenous Indeed, the advent of television remote con- to the likely outcome of the election. Some trols, the increase in the number of cable unmeasured (unexplained) portion of the channels, and the introduction of the Inter- dependent variable is related to unmeasured net have made Hovland’s first issue of discre- portions of the independent variable, and this tionary exposure to candidate messages more correlation violates one of the basic assump- problematic today than it was fifty years ago. tions of regression analysis. The statistical We would add a sixth point to Hovland’s solution is to find one or more “instrumen- (1959) list, the fact that today’s actual can- tal variables” that are related to the problem- didates can target their messages to particu- atic independent variable (campaign strategy) lar audiences and usually have a good idea but are not related to the dependent variable. of how receptive that audience will be to This is a tall order, and the results are only the messages, whereas in laboratory experi- as good as the quality of the instruments that ments we typically randomly expose subjects can be found (Bartels 1991). But if reason- to different messages. It is difficult to imag- able instruments are available, they are used, ine candidates making decisions about cam- along with any other independent variables in paign strategy without some idea of what their the model, to predict the problematic inde- chances are of winning the election, of whom pendent variable in a first-stage regression. the opponent is likely to be, and of which The predicted scores from this first-stage resources they will have in order to accom- regression, which have been “purged” of any plish their electoral goals. The conventional inappropriate correlation with unmeasured Candidate Advertisements 217

Table 15.1: Methodological Consequences of Differences between Observational and Experimental Studies of Candidate Advertisements

Problem Methodological Consequence

Audience selectivity Politicians preach to the choir, minimizing possibility of further opinion change in observational studies Unequal/uncertain exposure Harder to document opinion change with observational studies, experiments/lab studies overestimating potential effects Social influence Prior attitudes more resistant to change in the real world, experiments/lab studies overestimating potential effects Ephemerality Dependent variable usually measured immediately after treatment in experiments, which maximizes apparent treatment effects from lab studies Ego involvement Opinions on less ego-involving issues are easier to change, but such issues have less real-world importance, and the absence of relevant data usually makes them difficult to study with observational methods Endogeneity of campaign strategies Causal inferences extremely difficult with observational methods, experiments can simulate campaign situations that simply do not occur in actual campaigns

Note: The first five problems listed in Table 15.1 are discussed by Hovland (1959). aspects of the dependent variable, are then newspaper accounts of the campaigns, in Lau used to represent or stand in for the problem- and Pomper). This by necessity treats as iden- atic independent variable in a second-stage tically positive (or negative) a wide range of regression. campaign ads and themes and statements that In their studies of the effect of nega- might have quite different effects.2 Similarly, tive advertising, Ansolabehere, Iyengar, and because they did not have enough data to reli- Simon (1999) and Lau and Pomper (2004; ably measure campaign tone on a daily or even see also 2002) provide good examples of this weekly basis, these authors again implicitly procedure, and because they are based on assume that tone has the same effect across an analysis of virtually every contested Sen- the entire final two months of a campaign – ate election across multiple election years, a dubious assumption at best. Most of these the external validity of the findings are dif- limitations are not inherent in observational ficult to challenge. At the same time, one methods, but in practice they put very real could question exactly what has been learned constraints on what can be learned from any from these studies. Campaigns are dynamic such study. events and are usually observed over a period Candidate advertisement experiments can of time. Both Ansolabehere et al. and Lau resolve some of the inferential difficulties in and Pomper relied on American National estimating the effects of ads. Experiments can Election Studies (ANES) survey data in their avoid the endogeneity of campaign strategy individual-level analyses and thus implicitly that makes determining the causal direction observed those campaigns during the period of effects difficult in observational studies. the ANES was interviewing (typically, the Researchers can also more precisely estimate two months before the November elections). They have one estimate of the tone of each 2 Lau and Pomper (2002, 2004) did distinguish candidate’s campaign over this entire eight- between issue-based or policy-based statements and person-based statements (still, at best, a very gross week period (from each respondent’s mem- distinction), but did not find many differences ory, in Ansolabehere et al.; from a coding of between the two. 218 Shana Kushner Gadarian and Richard R. Lau the effect of the ad itself on turnout or can- and do not follow up with respondents didate evaluation without needing to adjust afterward. Researchers can measure the for voter characteristics that may drive both short-term effect of ads within the span of actual ad exposure and the political outcomes the experiment, but it is not entirely clear researchers care about because experimen- whether the effects captured during an exper- tal exposure to candidate ads is randomly iment are long lasting or a short-term reac- assigned rather than determined by the citi- tion to the experimental stimuli. In addition, zens’ personal characteristics, such as political it is difficult to measure the duration of ad interest, past voting behavior, and state of res- effects within the limited time constraints of 3 idence. Last, experiments allow researchers to most experiments. Nor do we know what test hypotheses about how ads affect behav- the cumulative effects of a political ad on atti- ior in ways that prove difficult to isolate using tudes might be (Gerber et al. 2007; Chong purely observational methods. and Druckman 2010). Experiments solve a number of issues However serious these concerns are, raised by observational studies, but they though, the benefit of using experiments is also come with limitations. Political cam- that they clearly establish the causal effect paigns involve at least two candidates vying of an ad on dependent variables of interest. to persuade voters to support their candi- Establishing this causal effect may be the first dacy and use multiple means of persuasion step to observing the effect in a broader politi- based on the closeness of the race and can- cal context. In other words, if researchers candidate resources. But experiments by their not show that a campaign ad affects candidate very nature lend themselves more naturally evaluations or turnout within an experiment, to studying discrete events – that is, single then it seems quite unlikely that they will be ads rather than comprehensive ad campaigns. able to observe an effect of that same ad using Even the most comprehensive experiments observational methods. We use the experi- that create a campaign environment (Lau mental literature on negative advertising to and Redlawsk 2006) cannot completely re- review how experiments determine what we create the “blooming, buzzing” chaos that know about the effects of campaign ads. is a political campaign, and thus the experimental environment is a simplification of the real-world campaign environment. Although 2. Negative Ads and the Likelihood experiments can isolate how campaign ads of Voting affect the public, they may over- or underestimate the effect of these ads in a full informa- A politician’s decision to produce and run tion campaign environment. The effect of any negative ads – ads that portray an opponent’s one particular ad may be washed away in a real campaign by the cumulative impact of candi- 3 Although we may be interested in how long the exper- date visits or debates, and thus experiments imental treatments last, it is worth noting that it is that only include single ads may overestimate unclear how long campaign messages more broadly last. Lodge, Steenbergen, and Brau (1995) suggest how ads affect the public. Yet, if the impact of that the half-life of campaign messages is typically ads depends on relevant policy information or less than a week. We have no reason to believe that the personal characteristics of the candidates the effects of campaign ads are substantially different from other types of campaign messages. One con- within a campaign that are absent from a more cern is that experiments may overstate how effective controlled but sterile experimental environ- ads are in shaping candidate evaluations because sub- ment, then using experiments to isolate how jects often forget which ad they saw, even at the end of a relatively short experiment. Yet, we agree with ad tone or content affects evaluations may Lodge et al.’s (1995) contention that “recall is not a underestimate their effects. necessary condition for information to be influential” A second potential limitation of study- (317–18), meaning that even if experimental subjects later forget their exposure to a particular ad, the expo- ing candidate ads experimentally is that most sure may continue to affect candidate evaluation and experiments occur at a single point in time vote choice. Candidate Advertisements 219 positions as wrong or that cast doubt on an whether the ads came from Bush or Gore, opponent’s character – may influence voters and whether the ads were positive or nega- to support the sponsor of the attack and thus tive. Overall, the authors found no evidence increase the probability of turning out to vote. that negative ads systematically increased or Yet, negative ads may also backfire, lower- decreased voters’ probability of turning out, ing voters’ probability of voting for the attack suggesting that exposure to one or two ads sponsor and demobilizing the public. In this in the middle of an ongoing campaign may section, we explore experimental findings on not systematically affect the decision to vote. how candidate advertising affects the proba- Clinton and Lapinski’s use of multiple ad bility of voting. treatments that varied tone and the source In the paradigmatic study of demobiliza- of the negative ad more closely mimic real- tion, Ansolabehere et al. (1994) demonstrate world campaigns than do experiments with that exposure to a single negative ad embed- single ads, although by using actual candidate ded in a fifteen-minute newscast decreased ads their tone manipulation inevitably varied experimental subjects’ intention to turn out more than simply tone. in the next election by five percentage points Krupnikov (2009) argues that exposure to compared to respondents who saw a posi- negative advertising may be demobilizing for tive ad with the same audiovisual script. The some voters depending on the timing of expo- many strengths of this experiment include 1) sure. She argues that campaign negativism the treatment conditions varied on only two only affects turnout when exposure comes dimensions (tone and sponsoring candidate) after a voter selects a candidate but before while being identical on all other dimensions he or she can implement the voting deci- (visuals, voiceover, issue focus); 2) the studies sion – a hypothesis that, in practice, would be occurred during ongoing political campaigns; very difficult to test with observational meth- and 3) the experimental setting approximated ods. In an online experiment using a repre- the home environment where ads might actu- sentative sample of Americans, respondents ally be viewed. By varying only the tone who received a negative ad after choosing between the treatment and control condi- a favored candidate were six percent more tions, the authors had much greater control likely to say that they would put no effort into over the nature of the experimental treatment turning out than those who received the neg- and were able to conclude that it is the nega- ative message before choosing their favorite tivism of the ads per se that decreases respon- candidate, suggesting a relatively strong dents’ intention to turn out. In addition, by demobilization effect but only for some setting the experiments within an actual cam- respondents. It is worth noting, however, paign, the researchers could choose salient ad that the experimental design uses candidates content and use real candidates, increasing devoid of names, written rather than audiovi- the realism of the lab experience. sual treatments, and ads of similar length but In the aforementioned example, the exper- different policy areas – and perhaps some of iments tested the effect of a single candi- these other differences rather than the timing date ad on candidate evaluations, but rarely of ads per se may affect the decision to turn do voters receive only one-sided informa- out. Overall, the Ansolabehere et al. (1994) tion in a campaign. Whether negative ads experiments provide the most controlled test mobilize or demobilize may depend on how of the hypothesis that negativity demobilizes many negative ads voters see or hear. Using the electorate, but other studies’ use of multi- an experiment with 10,200 eligible voters in ple ads more convincingly proxies the dynam- the Knowledge Networks panel during the ics and complexity of real campaigns. 2000 election, Clinton and Lapinski (2004) So, do negative ads demobilize or mobilize varied three factors to test whether negative the electorate? The results across observa- ads mobilized or demobilized the public: how tional and experimental studies are mixed. A many ads respondents saw (one vs. two ads), meta-analysis of fifty-seven experimental and 220 Shana Kushner Gadarian and Richard R. Lau observational studies of the so-called demobi- the effect of attack ads on more interested lization hypothesis demonstrated no consis- or less sensitive voters. Yet, when forced to tent effect of negative advertising on turnout confront negative ads in an experiment, these (Lau et al. 2007), which suggests that negative same voters may be turned off by the nega- ads may matter for a variety of reasons that tivism and want to disengage from politics. vary over different campaigns and/or for cer- If this is particularly the case for uncom- tain individuals at different times. There may mitted voters, then an experiment may find be at least three reasons why the turnout an overall demobilization effect, even when findings differ across experimental and obser- these voters would never encounter these ads vational studies: 1) cumulative effects, 2) in a real campaign. Although any of these pos- timing, and 3) selective versus broad expo- sibilities may explain the differences between sure. Most experimental studies of advertising observational and experimental research, if consider the effects of only one or two neg- there is one crucial factor that determines ative ads on turnout, although well-funded whether ad exposure will mobilize or demo- campaigns can run multiple ads per day for bilize, researchers have yet to identify it. weeks up to an election. Ansolabehere et al.’s (1994) experiments looked at the effect of exposure to a single ad for one candidate 3. Negative Ads and Candidate within campaigns that actually featured a high Evaluation/Vote Choice volume of competing ads. To the extent that the effects identified in the lab are cumula- Presumably, candidates decide to use negative, exposure to multiple negative ads should tive ads because they believe that the ads will strengthen the demobilizing effects identi- either decrease the likelihood of voting for fied. But if exposure to competing mes- the opposing candidate or at least lower the sages with varying tones could cancel each public’s evaluations of the opposing candi- other out, or if there is a relatively low ceil- date. Yet, experimental research on the effects ing effect for exposure to repeated attack of ads demonstrates that although negative ads, one-shot experimental designs cannot ads may decrease evaluations of the target of identify these longer-term effects. Observa- the attack, negative ads may have a variety tional studies are better suited to pick up the of other consequences, including 1) a back- effect of advertising volume on turnout, but lash against the attacking candidate who loses tone measures for individual ads are more popularity as a result of sponsoring the attacks difficult to obtain. The timing of advertis- (Matthews and Dietz-Uhler 1998); 2)a“vic- ing exposure may influence turnout in ways tim syndrome,” where the target of the attack that experimental designs and cross-sectional actually becomes more favorably viewed after observational studies may account for dif- a negative ad (Haddock and Zanna 1997); or ferently. Experiments are not typically run 3) a “double impairment,” where both the during campaigns, whereas observational source of the negative ad and the target are studies tend to occur right before elections. viewed more negatively. To the extent that voters are influenced by How negative ads affect candidate evalua- different factors over the course of a cam- tion may depend on the relationship between paign; for example, after they have made up the target of the ad and the voter. Negative their minds (Krupnikov 2009)orwhenit ads may reinforce partisan loyalties and affect looks like their favored candidate is going to those who share the partisanship of the ad lose, then how far in advance of an election a sponsor (Ansolabehere and Iyengar 1995), or, study is done and the competitiveness of the alternatively, negative ad exposure may lead race may determine whether voters decide to to what Matthews and Dietz-Uhler (1998) turn out at all. Last, negativism of any form call a “black sheep effect,” whereby nega- is uncomfortable for some voters, so, in the tive ads that come from a liked group cause real world, those voters may ignore negative respondents to downgrade a liked candidate ads, meaning that studies may only pick up (one who shares the voter’s partisanship), Candidate Advertisements 221 whereas negative ads that come from a less 4. Competing Messages liked candidate do not have this effect. In Matthews and Dietz-Uhler’s experiment, 123 Experiments that consider single ads in isola- undergraduates read either a positive or neg- tion may miss the dynamics that occur in real ative mock advertisement about family val- campaigns, when competing candidates pro- ues that was said to be sponsored by either vide contending considerations that compli- an in-group (same party) or out-group can- cate how voters evaluate candidates. Clinton didate. An in-group sponsor of a positive ad and Owen (2009) used a large-scale Knowl- was evaluated more positively than an out- edge Networks experiment during the 2000 group member, regardless of the type of ad. election to test the effect of competing Bush However, an in-group sponsor of a negative and Gore ads on how decided and undecided ad was evaluated more negatively than either subjects make a candidate choice. Respon- an in-group sponsor with a positive message dents with an initial predisposition toward a or an out-group sponsor of either type of ad. candidate solidified their vote intention after Does this black sheep effect apply to viewing the competing ads, but undecided all types of in-groups? Schultz and Pancer subjects were less likely to designate a candi- (1997) found that when the in-group/out- date preference after ad exposure, suggesting group characteristic is gender rather than that competing messages may inhibit some party, the black sheep effect is absent. In voters from making a vote choice. their experiment, 134 students read positive In a study of 274 undergraduates, Roddy or negative statements about the integrity and Garramone (1988) tested the impact of and personality of female or male candidates’ the type of negative ad respondents see (issue opponents. When judging a candidate of their vs. image) and the tone of a response to the own gender, subjects rated the candidate as attack (positive vs. negative). The authors cre- having greater integrity when the candidate ated positive and negative television ads for attacked his or her opponent than when he two fictional candidates that focused on either or she did not. Yet, when judging a candidate the candidate’s issue positions or the candi- of the opposite gender, participants tended date’s character. Each respondent saw one of to rate the candidate who attacked his or her the negative ads paired with either a negative opponent as having less integrity than those or positive response ad from the candidate who did not attack. It may be the case that targeted by the initial attack. Roddy and Gar- gender is not a strong or salient enough polit- ramone find that when candidates strike first, ical characteristic to make subjects feel bad negative issue ads significantly increase evalu- when a group member attacks another candi- ations of the sponsor’s character and decrease date. Alternately, undergraduate samples may the probability of voting for the target of the be particularly aware of social norms about attack more than negative image ads do. Yet, gender equality and want to reflect these when the target of an attack responds with a norms by not judging women candidates positive ad rather than a negative ad, it sig- differently from male candidates. Although nificantly lowers the probability of voting for either explanation is a possibility, whether a the sponsor of the original negative ad. “black sheep” effect occurs broadly or only During the 1996 presidential campaign, with partisanship is an open question. The Kaid (1997) conducted an experiment with relative dearth of actual female candidates a total of 1,128 undergraduates that exposed and their tendency to be disproportionately subjects to both positive and negative Dole Democratic makes these competing explana- and Clinton ads at two points during the cam- tions almost impossible to tease apart with paign – late September and early October. observational methods, but it would be a rel- Respondents received a mix of four negative atively simple matter to simultaneously vary and positive ads from the two candidates in both candidate gender and partisanship in an the September session and four different ads experiment to reconcile these contradictory in October. In the September round of exper- findings. iments, partisans increased evaluations for 222 Shana Kushner Gadarian and Richard R. Lau in-party candidates and lowered their eval- lying psychological mechanisms of those uation of out-party candidates. These find- effects. ings suggest that candidate ads affect individuals differentially based on their partisan identification and that experiments that use 6. Practical Challenges in nonpartisan candidates may overestimate the Designing Experiments on effectiveness of ads on the entire electorate. Candidate Advertisements A major benefit of Kaid’s (1997) experimental design is that it tests the impact Any researcher adopting experimental meth- of competing candidate ads over time. Kaid ods to study the effects of candidate advertise- shows that the effect of Dole messages ments faces a number of practical challenges changed significantly during the 1996 cam- that have to be addressed in designing the paign. In the first wave of the experiment, research. In this section, we explicate some exposure to Dole ads increased evaluations practical challenges that scholars face. of Dole among both Republicans and independents, but by October, all respondents Actual Ads or Ads Created Just took a more negative view of Dole after see- for the Experiment? ing the Dole spots. However, because this experiment was conducted in the context of One of the first questions any researcher plan- an ongoing presidential campaign, it is not ning an experiment must decide is whether to clear if it is time per se or some other fac- study ads actually produced and used by can- tor confounded with time in the 1996 cam- didates in a real campaign or to employ ads paign (i.e., it became increasingly clear that created explicitly for the experiment. “Bor- Dole was going to lose) that explains these rowing” ads from a real campaign is cheap and results. easy and has a great deal of external validity, but researchers give up a good deal of control by doing so. Many combinations of strategies 5. Other Campaign Ad Effects that we want to study experimentally simply do not occur in practice. The obvious In a series of experiments, Brader (2005) solution to this problem is to run an experi- demonstrates that campaign ads affect vot- ment and create your own political ads, ran- ers by appealing to emotions – particularly, domly assigning different campaign strategies enthusiasm and fear. In the experiments, to different candidates. The researcher gets a respondents saw either positive ads or neg- specifically desired type of ad, but with the ative ads as they watched a newscast. Half added expense of having to create the ad and of the positive ads included enthusiasm cues, the commensurate loss of external validity whereas half of the negative ads included resulting from employing ads that were not fear cues; these emotional appeals came from created by an actual candidate. A compromise the addition of music and evocative imagery. solution is to employ the video from an actual Brader shows that ads affect voters by appeal- ad (thus claiming some degree of external ing to their emotions, but that they do not validity) but to manipulate the “voice-over” simply move voters toward one candidate or that accompanies the video, so that all other away from another. Rather, emotions affect aspects of the manipulated ad are identical, voters by changing the way that voters make except for the verbal message. decisions. Fear appeals lead voters to make decisions based on contemporary informa- Real Candidates or Mock Candidates? tion, whereas enthusiasm appeals encourage decisions based on prior beliefs. This Another problem with making inferences study represents a growing body of exper- from real elections is that many candidates imental work that examines not only how (e.g., all incumbents) are familiar to many ads affect the public, but also the under- voters from previous elections. Thus, any Candidate Advertisements 223 new information that might be learned about experiments do not use party labels is that a familiar candidate during the course of a respondents may interpret other cues, such as campaign is interpreted against a background gender (King and Matland 2003) or person- of whatever prior information citizens have ality traits (Rapoport, Metcalf, and Hartman stored about that candidate in their memo- 1989), to infer candidates’ partisanship and ries. Again, the experimental solution is obvi- policy positions. Thus, experimental effects ous: use candidates with whom subjects are that researchers associate with these other not familiar, either by “creating” your own characteristics may actually be a function of mock candidates or by using real candidates candidates’ partisanship even when partisan- from another state with whom few subjects ship is absent. are familiar. But then any knowledge we gain as a result of this experiment should be lim- College Sophomores versus Real ited to situations where the candidates are People as Subjects? new and unfamiliar – either open seat races without any incumbent or lower-level offices Another question that any researcher where the candidates are just starting their employing experimental methods must face is political careers. A lot of elections are like where to get subjects. The easiest and cheap- this, and there is nothing wrong with study- est solution for many (academic) researchers ing them. We suspect, however, that most is undergraduates – a young, bright, cap- experimentalists using such a design do not tive, and generally compliant population. For imagine that they are primarily studying open many topics social scientists study (e.g., basic seat or lower-level elections. cognitive processes), undergraduates are perfectly good subjects. But as Sears (1986) warned, undergraduates are a homogenous Include Party Affiliation or Ignore It? population in terms of their age, life expe- Admittedly, a large store of information about rience, education/intelligence, heightened an incumbent politician is surely limited to political awareness, and “less crystallized the most politically interested subset of the social and political attitudes” (22); this homo- population. But for many people, a candi- geneity may restrict our ability to study date’s party affiliation substitutes for a great factors that could be of great interest to polit- deal of more specific information. However, ical campaigns. Druckman and Kam’s chap- if we include party affiliation in the descrip- ter in this volume discusses methods that tion of a stimulus candidate, then a subject’s may increase the generalizability of results own party identification could largely deter- produced by student samples, and we do mine candidate evaluations independent of not make strong recommendations about whatever advertisements from that candidate whether campaign ad experiments should are shown, thus greatly reducing the power choose adult samples over student samples. of any experimental manipulation to detect Whether researchers want to use a broad or significant effects. McGraw’s chapter in this narrow database (Sears 1986) should be deter- volume raises similar concerns. If party affil- mined by their theoretical expectations, in iation is avoided in the description of a stim- addition to practical considerations about the ulus candidate, then it will be much easier expense and difficulty of subject recruitment. to push around evaluations of that person. It is not impossible to bring “real people” But would whatever we learn from such an into a laboratory, but it does add consider- experiment be generalizable only to non- ably to the financial costs and time required partisan elections? Surely that would not be to run an experiment. To recruit nonstu- the intent of most researchers. Indeed, how dent experimental samples, researchers can campaign ads are perceived and interpreted include college employees in the sample depends crucially on whether the ad comes pools of on-campus labs (Kam, Wilking, and from my guy or their guy (Lipsitz et al. 2005). Zechmeister 2007); recruit community mem- An additional complication that arises when bers through newspaper advertising, flyers, or 224 Shana Kushner Gadarian and Richard R. Lau

Internet message boards (Brader 2005;Lau vote choice with treatments that may or may and Redlawsk 2006); bring the experiment to not reflect candidates’ actual policy or per- subjects face-to-face (Ansolabehere and Iyen- sonal positions. Respondents do often for- gar 1995; Mendelberg 2001); or use online get treatments relatively soon after exposure, panels. High-quality online samples such as and debriefing should lower concerns that those in the YouGov/Polimetrix or Knowl- respondents will believe that the treatments edge Networks panels may provide a less are real. However, we know of few stud- expensive way to include adults in experimen- ies that follow up with respondents to verify tal samples than sending research assistants that treatments do not have long-term conse- to the homes of respondents or paying “real quences. In addition to the extent that exper- people” to come to the lab. However, when imental subjects use online processing rather experimentalists rely on online experiments, than memory-based processing, experiments this inevitably shortens the duration of an may affect political attitudes and behaviors experiment and may limit the types of candi- even if respondents forget the details of treat- date ad experiments that are possible. Shorter ments, which may be particularly troubling online experiments are more common than during the course of a campaign. longer, more in-depth experiments that may allow researchers to embed candidate commercials within newscasts or otherwise make 7. Conclusion the experimental treatment less obvious, or to include multiple ads as treatments. Online As experimental research on campaign ads subjects may feel less pressure to comply with continues to increase in popularity in the directions and be more likely to opt out of discipline, we should consider whether Hov- the experiment by switching to something in land’s (1959) critiques of experimental stud- their home or on their computer that is more ies of media effects still hold or whether the interesting. Although online panels can pro- current literature has overcome these con- vide more representative samples of the vot- cerns. It is worth noting that many of Hov- ing public than can on-campus labs, online land’s complaints against the external valid- experiments may be only a partial solution ity of experimental studies of media effects to concerns over external validity if studies (unrealistic exposure, social isolation of the themselves are more limited. experimental subject, probable but unknown ephemerality of the demonstrated effects, use of trivial and less ego-involving issues in Experiment during Campaigns versus studies) are not inherent in the experimen- Outside Campaigns? tal method, but the choices that researchers Another decision facing researchers is whe- make when designing experiments may either ther to run experiments during the course alleviate or exacerbate these issues. Although of an ongoing campaign or at another time. some current studies directly address issues Running studies during actual campaigns may such as the ephemerality of campaign effects increase external validity by using real candi- (Clinton and Lapinski 2004; Chong and dates and salient issues, but there are prac- Druckman 2009) and use real candidates to tical and ethical challenges to waiting for a make studies more ego involving (Kaid 1997; campaign to begin. On the practical side, Clinton and Owen 2009), several of the other limiting studies to campaign season signifi- critiques are overlooked in the current litera- cantly limits how often researchers can con- ture and thus provide opportunities for new – duct field studies and requires tight coordi- and very important – scholarship. nation with the scheduling of pre-testing and In a world of increasing media choice, the human subjects approval. On the ethical side, problem of audience selectivity is probably when researchers use real candidates during more serious now than in Hovland’s (1959) an actual campaign, there is a possibility of era and may make demonstrating effects of affecting candidate evaluations and possibly campaign ads more difficult in observational Candidate Advertisements 225 studies than in experimental ones. That is, in condition the effect of campaign communi- the real world, less interested or less partisan cation will be an increasingly important issue citizens can tune out, turn off, or altogether for researchers to consider. avoid candidate ads, whereas in the lab some Hovland’s (1959) last several concerns percent of subjects receive a political message about the external validity of experiments, that they would never otherwise receive. The ephemerality and ego involvement, are not advent of new technologies and collection of completely diminished by current research, new data that more precisely track when and but several studies do directly address these how many times campaign ads air (Franz et al. concerns. Again, there is nothing inherent in 2007) can help both observational and exper- the experimental method that demands that imental researchers estimate when campaign scholars design one-shot studies that mea- exposure is probable and what types of indi- sure reactions to ads or candidate evalua- viduals are likely to be most affected by expo- tions immediately after ad exposure. Rather, sure. We know of few experimental studies for practical and financial reasons, many that contend with the issue of selectivity or researchers choose not to follow up with sub- allow respondents to opt out of experimen- jects later to test how long campaign ads may tal treatments (Arceneaux and Johnson 2007; affect evaluations or intended vote. Like Clin- Gaines and Kuklinski’s chapter in this vol- ton and Lapinski’s (2004) study with Knowl- ume); however, with a more complex media edge Networks, studies using online panels environment, questions about the types of allow following up with subjects more eas- citizens who may be affected by campaign ily than bringing people back into the lab. communications may suggest more complex Without following up with subjects, it is experimental designs in the future. unclear whether ephemerality of experimen- Political campaigns take place in a social tally induced effects should concern politi- context. Voters do not experience campaigns cal scientists. In addition, we believe that the in social isolation, but rather they talk about concern that experiments often involve triv- events and ads with friends, family, and ial or less ego-involving issues is less seri- coworkers, many of whom share their politi- ous in campaign experiments, and particularly cal identities. The makeup of one’s social net- less worrisome in studies that use real can- work may affect how ads influence candidate didates, because partisanship provides infor- evaluation or vote choice. When voters are mation about the candidate and acts as a surrounded by like-minded compatriots, they strong affective tie between subject and candi- may be less likely to accept counterattitudi- date (Kaid 1997; Clinton and Lapinski 2004; nal arguments than when faced with a more Brader 2006; Clinton and Owen 2008). heterogeneous social network. Yet, experi- Campaigns are strategic in their use of ments of campaign ads rarely take these types resources (Shaw 2007) and are now more of social dynamics into account, confirm- able to target particular audiences with dif- ing Hovland’s (1959) concerns over social ferent messages or to microtarget likely vot- isolation. Unlike social psychology experi- ers. Without good observational studies that ments that often introduce elements of group describe when and why campaigns deploy dif- discussion and decision making, most cam- ferent campaign ad strategies such as going paign experiments take place at the individual negative versus staying positive, experimental level and only include interactions between studies may produce political situations that the experimenter and the subject. Although do not occur in reality, thus harming exter- researchers lose an element of control, exper- nal validity. Shaw’s study of the 2000 and imenters could invite subjects into the lab 2004 presidential campaigns illuminates the in groups rather than individually or could strategic logic of where and when presiden- snowball sample respondents from existing tial candidates advertise, but we know of few social networks. With political campaigns’ studies that outline other strategies employed increasing use of social networking sites such by candidates. Because campaign strategy as Facebook, the issue of how social networks is endogenous to the political situations in 226 Shana Kushner Gadarian and Richard R. Lau which candidates find themselves, experi- Clinton, Joshua, and John Lapinski. 2004. “‘Tar- mental researchers should seriously consider geted’ Advertising and Voter Turnout: An when they would expect to see the results that Experimental Study of the 2000 Presidential 66 69 96 they demonstrate in their studies. Election.” Journal of Politics : – . 2009 Experimentation provides researchers Clinton, Joshua, and Andrew Owen. .“An with significant control over design in order Experimental Investigation of Advertising Per- suasiveness: Is Impact in the Eye of the to test causal claims about, for example, Beholder?” Working paper, Vanderbilt Uni- whether negative advertising demobilizes versity. voters. Experiments are invaluable in demon- Franz, Michael M., Paul Freedman, Kenneth strating the mechanism at work. As with M. Goldstein, and Travis N. Ridout. 2007. all research, however, experiments require Campaign Advertising and American Democracy. trade-offs between what types of questions Philadelphia: Temple University Press. researchers can answer and how to create Gerber, Alan, James Gimpel, Donald Green, and environments that, on some level, resemble Daron Shaw. 2007. “The Influence of Tele- real elections. Yet, despite these challenges, vision and Radio Advertising on Candidate experimental research on candidate adver- Evaluations: Results from a Large Scale Ran- tising illuminates how voters contend with domized Experiment.” Working paper, Yale University. ads in forming evaluations of candidates and Haddock, Geoffrey, and Mark Zanna. 1997. deciding whom to choose at election time. “Impact of Negative Advertising on Evalua- tions of Political Candidates: The 1993 Cana- dian Federal Elections.” Basic and Applied Social References Psychology 19: 205–23. Hovland, Carl. 1959. “Reconciling Conflicting Ansolabehere, Stephen, and Shanto Iyengar. 1995. Results Derived from Experimental and Survey Going Negative: How Attack Ads Shrink and Studies of Attitude Change.” American Psychol- Polarize the Electorate. New York: Free Press. ogist 14: 8–17. Ansolabehere, Stephen, Shanto Iyengar, and Adam Kaid, Lynda Lee. 1997. “Effects of the Television Simon. 1999. “Replicating Experiments Using Spots on Images of Dole and Clinton.” Ameri- Aggregate and Survey Data: The Case of Neg- can Behavioral Scientist 40: 1085–94. ative Advertising and Turnout.” American Polit- Kam, Cindy, Jennifer Wilking, and Elizabeth ical Science Review 93: 902–9. Zechmeister. 2007. “Beyond the ‘Narrow Data Ansolabehere, Stephen, Shanto Iyengar, Adam Base’: Another Convenience Sample for Exper- Simon, and Nicholas Valentino. 1994.“Does imental Research.” Political Behavior 29: 415– Attack Advertising Demobilize the Elec- 40. torate?” American Political Science Review 88: Kinder, Donald R. 2003. “Communication and 829–38. Politics in the Age of Information.” In Oxford Arceneaux, Kevin, and Martin Johnson. 2007. Handbook of Political Psychology, eds. David “Channel Surfing: Does Choice Reduce Video- O. Sears, Leonie Huddy, and Robert Jervis. malaise?” Paper presented at the annual New York: Oxford University Press, 357– meeting of the Midwest Political Science Asso- 393. ciation, Chicago. King, David C., and Richard E. Matland. 2003. Bartels, Larry. 1991. “Instrument and ‘Quasi- “Sex and the Grand Old Party: An Experimen- Instrumental’ Variables.” American Journal of tal Investigation of the Effect of Candidate Sex Political Science 35: 777–800. on Support for a Republican Candidate.” Amer- Brader, Ted. 2005. “Striking a Responsive Chord: ican Politics Research 31: 595–612. How Political Ads Motivate and Persuade Vot- Krupnikov, Yanna. 2009. “Who Votes? How and ers by Appealing to Emotions.” American Jour- When Negative Campaigning Affects Voter nal of Political Science 49: 388–405. Turnout.” Paper presented at the annual meet- Chong, Dennis, and James Druckman. 2010. ing of the Midwest Political Science Associa- “Dynamic Public Opinion: Communication tion, Chicago. Effects over Time.” American Political Science Lau, Richard R., and Gerald Pomper. 2002. Review 104: 663–680. “Effectiveness of Negative Campaigning in Candidate Advertisements 227

U.S. Senate Elections, 1988–98.” American Journal of Applied Social Psychology 28: 1903– Journal of Political Science 46: 47–66. 15. Lau, Richard R., and Gerald Pomper. 2004. Neg- Mendelberg, Tali. 2001. The Race Card: Cam- ative Campaigning: An Analysis of U.S. Senate paign Strategy, Implicit Messages, and the Norm Elections. Lanham, MD: Rowman Littleﬁeld. of Equality. Princeton, NJ: Princeton Univer- Lau, Richard R., and David Redlawsk. 2006. How sity Press. Voters Decide: Information Processing in Election Rapoport, Ronald, Kelly Metcalf, and Jon Hart- Campaigns. New York: Cambridge University man. 1989. “Candidate Traits and Voter Press. Inferences: An Experimental Study.” Journal of Lau, Richard R., Lee Sigelman, and Ivy Politics 51: 917–32. Brown Rovner. 2007. “The Effects of Neg- Roddy, Brian, and Gina Garramone. 1988. ative Political Campaigns: A Meta-Analytic “Appeals and Strategies of Negative Political Reassessment.” Journal of Politics 69: 1176– Advertising.” Journal of Broadcast and Electronic 209. Media 32: 415–27. Lipsitz, Keena, Christine Trost, Matthew Gross- Schultz, Cindy, and S. Mark Pancer. 1997.“Char- man, and John Sides. 2005. “What Voters acter Attacks and Their Effects on Percep- Want from Campaign Communication.” Polit- tions of Male and Female Political Candidates.” ical Communication 22: 337–54. Political Psychology 18: 93–102. Lodge, Milton, Marco R. Steenbergen, and S. Sears, David O. 1986. “College Sophomores in Brau. 1995. “The Responsive Voter: Campaign the Laboratory: Inﬂuence of a Narrow Data Information and the Dynamics of Candidate Base on Social Psychology’s View of Human Evaluation.” American Political Science Review Nature.” Journal of Personality and Social Psy- 89: 309–26. chology 51: 515–30. Matthews, Douglas, and Beth Dietz-Uhler. 1998. Shaw, Daron. 2007. TheRaceto270:TheElec- “The Black-Sheep Effect: How Positive and toral College and the Campaign Strategies of 2000 Negative Advertisements Affect Voters’ Per- and 2004. Chicago: The University of Chicago ceptions of the Sponsor of the Advertisement.” Press. CHAPTER 16

Voter Mobilization

Melissa R. Michelson and David W. Nickerson

Civic participation is an essential component mobilization, the problems faced by such of a healthy democracy. Voting allows citi- work, and opportunities for future study. zens to communicate preferences to elected officials and influence who holds public office. At the same time, deficiencies and asymme- 1. Observational Studies tries of participation in the United States call into question the representativeness of Nonexperimental studies have primarily reelected officials and public policies.1 Yet, lied on survey research to demonstrate cor- although political activity is crucial for the relations between self-reported mobilization equal protection of interests, participation and civic-minded behaviors, while also con- is often seen by individuals as irrational or trolling for demographic characteristics (e.g., excessively costly, and it is well known that age, education, income) that are known to turnout in the United States lags well behind be significant predictors of turnout.2 The that of other democracies. Scholars have con- conclusion usually reached is that mobi- sistently found that participation is linked lization efforts are generally effective (e.g., to socioeconomic variables, psychological Rosenstone and Hansen 1993; Verba, Schloz- orientations, and recruitment. Candidates, man, and Brady 1995). However, four major parties, and organizations thus spend con- empirical hurdles render this conclusion siderable effort mobilizing electoral activity. suspect. This chapter highlights contributions made by field experiments to the study of voter 2 Most of the critique that follows is equally applicable to the few studies using selection models. Selec- 1 For example, California’s population has not been tion models acknowledge the problem with strategic majority Anglo (non-Latino white) for some time, targeting by campaigns and attempt to model the yet, the electorate is more than two-thirds Anglo. process; however, such models rely on strong Thus, elections and ballot measures are decided by assumptions that may not be warranted in many an electorate that is not necessarily representative of instances, so the problem of strategic selection is not state opinion. fully solved (see Sartori 2003).

228 Voter Mobilization 229

First, campaigns strategically target indi- “Did anyone from one of the political par- viduals likely to vote, donate money, or ties call you up or come around and talk to volunteer, thereby creating a strong corre- you about the campaign? (IF YES:) Which lation between the behavior or attitude to be party was that?” Yet, experiments using var- studied and campaign contact. Because con- ious types of outreach clearly show that the tacted individuals are more likely to partic- method and quality of mobilization matters, ipate than noncontacted individuals – even generating turnout effects that range from in the absence of mobilization – strategic negligible to double digits (Green and Ger- targeting causes researchers to overestimate ber 2008). Coarse, catch-all survey measures the effect of mobilization. That is, observa- obscure the object of estimation by lump- tional samples use an inappropriate baseline ing heterogeneous forms of campaign contact for comparison. together. Second, individuals who are easier to con- Controlled experiments directly solve tact are also more likely to vote. Arceneaux, each problem associated with observa- Gerber, and Green (2006) analyze experi- tional studies. Random assignment eliminates mental data as if they were observational by selection problems and constructs a valid matching contacted individuals in the treat- baseline for comparison. By directly manipu- ment group to people in the control group lating the treatment provided to the subjects, with exactly the same background charac- researchers can avoid relying on overbroad teristics. They find that matching overesti- survey questions and the vagaries of self- mates the effect of mobilization. Matching reported behavior. Field experiments using fails to account for unobserved differences official lists of registered voters can also max- between treatment and control subjects (e.g., imize external validity by including a wide residential mobility, health, free time, mortal- range of subjects and by using official voter ity, social behavior), leading to inflated esti- turnout records to measure the dependent mates of the power of the treatment. Thus, variable of interest for both the treatment and the treatment of interest (i.e., mobilization) is the control groups. likely to be correlated with unobserved causes of participation. Pioneering Experiments A third drawback to survey-based research is that respondents often exhibit selective The experimental literature on mobiliza- recall. Politically aware individuals are more tion dates back to Gosnell (1927). Following likely to report contact from campaigns and Gosnell, experiments were used sporadically organizations because they pay more atten- throughout the next several decades (Elder- tion to political outreach and are more likely sveld 1956; Adams and Smith 1980; Miller, to place the event into long-term memory Bositis, and Baer 1981). These small n studies (Vavreck 2007). Because politically interested tested a range of techniques (mail, phone, and people are more likely to participate, the door-to-door canvassing), and all reported correlation between mobilization and behav- double-digit increases in voter turnout. ior could be a function of selective mem- Unfortunately, these early studies provided ory. Thus, the key independent variable in biased estimates of campaign contact by plac- observational studies relying on self-reported ing uncontacted subjects assigned to the campaign contact is likely to suffer from mea- treatment group in the control group. That surement error. is, these pioneering studies undercut the ana- Finally, survey questions used to collect lytic benefits of randomization by focusing self-reported campaign contact offer cat- only on the contacted individuals and turned egories too coarse to estimate treatment the experiments into observational studies. effects. Standard survey questions tend to The real flowering of the experimental treat all forms of campaign contact as iden- study of campaign effects came at the turn tical. For example, the American National of the millennium with the 1998 New Haven Election Studies item on mobilization asks, experiment (Gerber and Green 2000). A large 230 Melissa R. Michelson and David W. Nickerson number of subjects were drawn from a list age points, volunteer telephone calls by two to of registered voters and randomly assigned five percentage points, and indirect methods to nonpartisan treatments (mail, phone, or such as mail generally not at all (with some door) or to a control group. Gerber and notable exceptions). Green then had callers and canvassers carefully record whether each subject in the treat- Extensions to the First Experiments ment group was successfully contacted and referenced official records to verify voter The initial studies have been extended in a turnout for both the treatment and control large number of ways. Some studies have groups.3 The failure-to-treat problem was examined previously tested techniques for addressed by using random assignment as heterogeneity. One notable contribution fol- an instrument for contact, thereby providing lowed up on initial findings that commer- an unbiased estimate of the effect of con- cial phone banks were generally ineffective, tact. The experimental design and analysis whereas volunteer phone banks were usually disentangled the effect of mobilization from successful. Nickerson (2007) trained volun- the effects of targeting and selective memory teer callers to behave like commercial phone and was very clear about the nature of con- bank staff, giving them quotas of individuals tact provided to individuals, thereby avoiding to reach during each shift, while paying com- measurement error. They concluded that mercial canvassers to behave like volunteers, face-to-face contact raised turnout by nine urging them to take their time and engage percentage points, mail boosted turnout by voters in conversation. The result was a rever- half a percentage point, and phone calls did sal of the general trend: commercial phone nothing to increase participation. bankers trained to act like volunteers were It is curious that the logic of experimenta- able to move voters to the polls, while rushed tion took so long to take root in the study volunteers were ineffective. Thus, Nicker- of campaigns. Fisher (1925) laid the intel- son concluded that it was the quality of the lectual groundwork for experiments during phone bank that mattered, not the identity the 1930s. There were few technological of the canvasser or whether canvassers were hurdles to the process because randomiza- paid. tion could be performed manually (e.g., coin Other experiments have examined other flip) and the analysis could be done through campaign tactics for contacting voters, such as card sorting. Examples of laboratory experi- radio and television advertisements, leaflets, ments studying the effects of television adver- e-mail, and text messaging. In general, the tisements on attitudes and vote intention pattern has been that personalized outreach had been published in leading journals (see is more effective than indirect outreach, but Gadarian and Lau’s chapter in this volume). there are notable exceptions. For example, Regardless of the cause of the delay, the Dale and Strauss (2007) find that text mes- past decade has seen an explosion of inter- sages are effective at moving young people est in the use of field experiments to explore to the polls. Whether this is a counterex- voter mobilization, with increasing atten- ample or evidence that cell phones are con- tion to other facets of electoral campaigns, sidered personal objects is open to debate, such as vote choice and campaign contribu- and further research is needed to confirm tions. In a meta-analysis of the more than and further explore their findings. Similarly, 100 field experiments replicating their ini- the effectiveness of television advertisements tial study, Green and Gerber (2008) conclude (Vavreck 2007; Green and Vavreck 2008)may that well-conducted, door-to-door visits gen- be evidence that not all indirect methods erally increase turnout by six to ten percent- of reaching out to voters are ineffective, or may say something about the power of visual images. Paradoxically, the same rapid growth 3 Ironically, a merging error that did not substantively alter the results casts doubt on the initial findings in field experiments that allowed for precise with respect to phone calls (Imai 2005). estimates of the effectiveness of mobilization Voter Mobilization 231 techniques has also complicated the theoret- year was transferred to turnout in the general ical picture, necessitating more experiments. election. Gerber et al.(2003) hypothesize that A third line of extensions from the ini- individuals successfully encouraged to vote tial New Haven experiment has focused on may, in the future, feel more self confident subpopulations with below-average rates of about their ability to negotiate the voting pro- voter turnout. To the extent that low rates cess, or may have shifted their self-identity of participation bias the electorate, focus- to include civic participation rather than ing on groups with the lowest rates of voter abstention. turnout is a priority. Research in other areas Mobilization experiments have also ex- of political science suggests that civic engage- plored the effect of social networks. Nickerson ment strategies that are effective with Anglos (2008) examines two canvassing efforts that (non-Latino whites) will not necessarily work spoke with one of the voters in two-voter for African Americans, Latinos, and Asians. households, allowing for measurement of However, a lengthy series of recent exper- both the effect on contacted voters and their iments demonstrates that these subgroups housemates. The study used a unique placebo generally respond to requests to vote in a design, wherein individuals assigned to the similar manner as do high-propensity voters control group were contacted but received a (Michelson, Garcıa´ Bedolla, and Green 2007, message encouraging them to recycle. Both 2008, 2009). Indeed, each population faces experiments found that sixty percent of the its unique challenges. The residential mobil- propensity to vote was passed along to the ity of young and poor voters makes them other member of the household. Yet, other harder to contact (Nickerson 2006). Cam- experiments using social networks have pro- paigns targeting Latinos need to be bilingual duced mixed results (see Nickerson’s chapter in most instances, and efforts aimed at Asian in this volume). Americans need to be multilingual. Despite Electoral context is also an important fac- these challenges, field experiments proved tor; even if all individuals in a treatment that these groups can be effectively moved group are successfully contacted, not all will to the polls. be moved to vote. This is a reflection of A fourth set of analyses extends the exper- the ongoing real-world context from which imental project by considering the dynamics experimental subjects are taken (see Gaines, of voter mobilization. Contact is more effec- Kuklinski, and Quirk 2007). Arceneaux and tive as Election Day approaches, yet, thirty Nickerson (2009) argue that mobilization has to fifty percent of the mobilization effect the strongest effect on voters who are indif- on turnout in one election is carried into ferent about turning out, but these indifferent future elections (Gerber, Green, and Shachar voters are not the same from one election to 2003). That is, blandishments to vote are the next. Only low-propensity voters can be less effective when made earlier in an elec- mobilized in high-salience elections, whereas tion campaign, suggesting that contact has a high-propensity voters are more likely to limited shelf-life, but those individuals who respond in low-salience elections, and occa- are effectively moved to the polls continue sional voters are best targeted during mid- to be more likely to vote in future elections, level salience elections. suggesting the act of voting is transfor- Since the baseline effectiveness of various mative. Using data from fourteen experi- treatments has been established, voter mobi- ments targeting low-propensity communities lization experiments provide an excellent of color conducted previous to the Novem- real-world setting by which to test social psy- ber 2008 election, Michelson, Garcıa´ Bedolla, chological theories. Researchers know how and Green (2009) find a similar habit effect in much turnout is elicited using various tech- low-propensity communities of color. Across niques. By embedding psychological theo- fourteen separate mobilization experiments ries into messages encouraging turnout, the conducted during 2008, one third of the strength of the role the psychological con- mobilization effect generated earlier in the structs play in voter mobilization can be 232 Melissa R. Michelson and David W. Nickerson measured. One of the first efforts to link Monitoring has been found to enhance social psychology to voting behavior through compliance to social norms in the laboratory field experiments was conducted by a team (e.g., Rind and Benjamin 1994). Consistent of researchers at the Ohio State University with those findings, field experiments have prior to the 1984 presidential election. Stu- found that the mobilization effect is enhanced dents predicting that they would vote were by messages that signal to voters that their in fact more likely to do so (Greenwald et al. behavior is being observed. Gerber, Green, 1987). Efforts to replicate the finding on a and Larimer (2008) sent mailers to targeted larger scale, however, have failed to uncover individuals that indicated to varying degrees reliable treatment effects on representative that they were being monitored. Some mail- samples of voters (e.g., Smith, Gerber, and ers noted only that researchers were watching Orlich 2003). Gollwitzer’s (1999) theory of the election, others also included the recipi- implementation intentions, which holds that ent’s own voting history or the voting his- articulating explicit plans for action increases tories of the recipient’s neighbors, and some follow-through, has been found to more than also included a promise to send an updated double the effect of mobilization phone calls chart after the election. The more intru- by simply asking subjects about when they sive and public the information provided, the will vote, where they will be coming from, and larger the effect on turnout. The final treat- how they will get to the polling place (Ger- ment arm in the experiment raised turnout by ber and Rogers 2009; Nickerson and Rogers 8.1 percentage points, exceeding the effect of 2010). many door-to-door efforts. Psychological theories have also been used Without existing benchmarks to compare to explain apparent paradoxes in the litera- the results, the 1998 New Haven experi- ture. For instance, contacting people more ment simply constituted proof that campaigns than once, either by phone or in person, can mobilize voters and overcome the col- does not increase turnout significantly more lective action problem inherent in political than a single phone contact. However, an participation. It also provided an invaluable important caveat to this finding is that follow- example of how campaigns could be stud- up calls made to individuals who indicate in ied experimentally. That template has been an initial contact that they intend to vote expanded on to answer questions of increas- has a powerful and large effect on turnout ing nuance and detail about who can be (Michelson, Garcıa´ Bedolla, and McConnell mobilized, the dynamics of mobilization, and 2009). In a series of experiments, Michel- the psychology of mobilization. Despite the son et al. asked youth, Latinos, and Asian wealth of insights gained from the past decade Americans who were contacted during an ini- of experiments, the experimental study of tial round of telephone calls whether they mobilization behavior faces a large number of intended to vote. Restricting follow-up calls potential problems. The next section dis- to voters who indicated that they intended cusses the practical problems of carrying to vote resulted in double-digit treatment out experiments, concerns about the external effects, most of which can be attributed to the validity of the findings, and ethical concerns second call. To explain this finding, Michel- about studying campaign activities. son et al. turn to Sherman’s (1980) theory of the self-erasing nature of errors of prediction, which posits that asking people to 2. Problems Facing Field predict their future behavior increases the Experiments: Implementation likelihood of them engaging in the predicted behavior, and Fishbein and Ajzen’s (1975) The major attraction of experiments as a theory of reasoned action, which holds that methodology is that randomization assures subjects respond to treatment if a social that the treatment and control groups are norm is cued and subjects care what others comparable. By gathering theoretically ideal think. datasets, researchers can offer transparent Voter Mobilization 233 and straightforward analysis without the need is mobilized just like the treatment group. for control variables or complicated model- Nickerson (2005) offers various scalable pro- ing. However, constructing these data is diffi- tocols to conserve statistical efficiency in the cult. Not only is the process time consuming, face of problems implementing a treatment but problems can also arise when working in regime. the field removes control from the researcher. Even if an organization makes a good faith In particular, treating the correct people and effort to adhere to the prescribed protocol, documenting the contact can be difficult. simply applying the treatment to assigned Although the heads of organizations may subjects can be objectively difficult. When agree to participate in experiments, faithful working door to door, canvassers must nego- execution of the protocol is not always a given. tiate unfamiliar streets and, in rural neighbor- Mistakes can be made by managers when pro- hoods, may find themselves in areas without viding lists. Volunteers may make mistakes street signs or house numbers. Physical bar- when knocking on doors, they may speak with riers such as locked gates and apartment everyone encountered on a block in enthusi- buildings, the presence of dogs, or the lack asm for the campaign, or avoid blocks entirely of sidewalks may prevent canvassers from because they do not think the campaign will accessing doors. Even when canvassers have be well received. Treating members of the access, targeted voters are often not at home. control group is mathematically equivalent Young people and low-income individuals to failing to apply the treatment to mem- are likely to have moved since registering, bers of the treatment group and does not and older individuals are often at work or necessarily invalidate the experiment. The otherwise away when canvassers are avail- assignment can still be used as an instru- able. These factors will cause contact rates ment for actual contact to purge the estimate to be less than 100 percent (in fact, contact of the nonrandom determinants of contact, rates in the high single digits or low teens but statistical power will suffer dramatically are not uncommon for a single pass through (Nickerson 2005). a neighborhood or call sheet). Low contact Carefully training managers and can- rates reduce statistical power, and the pri- vassers can help mitigate these problems, as mary solution is to revisit the neighborhood can active involvement by the researcher in or phone list repeatedly. This added labor providing lists and monitoring the campaign. can decrease the number of subjects covered. Randomizing at the precinct level, rather than Thus, researchers should structure their ran- at the household level, can prevent many domizations in such a way that unattempted errors by managers and volunteers. The seri- people can be placed into the control group or ous downside of this strategy is that the power omitted from the analysis (Nickerson 2005). of the experiment decreases. The power of Experiments in some minority areas pose an experiment comes, in large part, from special challenges because of naming conven- the number of random decisions made. Ran- tions. Latino families often use the same first domly assigning ten precincts to treatment names but with suffixes (e.g., Junior, Senior) and control groups, rather than 5,000 house- or with different middle names (e.g., Maria A. holds, yields vastly fewer possible outcomes Garcia and Maria E. Garcia). Hmong share of the process. Statistical power is decreased the same twenty last (clan) names and also when subjects in a group share characteris- have similar first names. Canvassers work- tics and tendencies (i.e., intracluster correla- ing in these communities must be particularly tion is high). Whether this decrease in power attentive to the details of the names (and per- offsets problems in implementation depends haps ages or other identifying information) on the extent of anticipated problems and in order to ensure that they are contacting the degree of subject homogeneity within the targeted individual. Those preparing walk precincts. Last-minute changes in strategies lists or call sheets must be attentive to these have caused a number of experiments to go issues as well, for example, by not deleting the in the dumpster because the control group middle or suffix name columns to save space 234 Melissa R. Michelson and David W. Nickerson and by drawing attention to these issues dur- and impersonal are less effective than those ing canvasser training. Furthermore, match- that are measured and conversational (Nick- ing names to voter files after the election can erson 2007; Ha and Karlan 2009; Michel- be complicated, and multiple matches will be son, Garcıa´ Bedolla, and Green 2009). The likely. Once again, collecting and retaining as talent and charisma of individual volunteers much identifying information as possible will will vary in large campaigns, and subjects mitigate these problems. may be given qualitatively different treat- A final logistical problem (and a challenge ments depending on their canvasser or caller. for internal validity) is defining what consti- Researchers can work to minimize vari- tutes contact from the campaign. Contact is ance in treatment by carefully training not a problematic definition for impersonal workers and crafting scripts that anticipate forms of outreach such as mail, leaflets, and deviations and questions, thereby equipping e-mail. Incorrect addresses and spam filters canvassers to provide consistent answers. may prevent some materials from reaching However, the researcher should keep in mind their intended targets, but most mailed and that the quantity to be estimated is always e-mailed get out the vote (GOTV) messages an average treatment effect. This average can be safely assumed to have been delivered. conceals variation in how subjects respond However, it is difficult to know how much and variation in the treatment provided. of a script must be completed on the phone Researchers can take two steps to capture or in person to consider a subject treated. this variation. Canvassers and callers can be This coding decision makes no difference for randomly assigned phone numbers or can- intent-to-treat analysis that relies solely on vassing areas, and researchers can record assignment to treatment conditions (and is which canvasser contacts each targeted sub- most useful for program evaluation), but it ject. Combined, these two design principles poses a large problem for attempts to mea- allow researchers to measure the extent of the sure the effect of a campaign on individuals variation across canvassers. (the quantity in which political scientists are typically interested). If treatment is defined as a respondent listening to the entire script, 3. Problems Facing Field but there is an effect of listening to half of the Experiments: External Validity script and hanging up, then estimates of the treatment effect will be biased. An alternative The chief reason to study campaign effects is to define treatment more loosely, includ- in the field rather than in the laboratory is ing any individual with whom any contact is to more accurately capture the experience of made. This allows for more reasonable adop- typical registered voters receiving contact in tion of the assumption that noncontact has real-world settings with the associated dis- zero effect, but may also dilute the measured tractions and outside forces acting on the effect of the intended treatment. interaction. That is, the whole point of field A related problem is heterogeneity in the experiments is external validity. However, treatment applied by canvassers and callers. field experiments themselves can only draw Again, variance in the treatment provided inferences about compliers, campaigns sub- is not a problem for indirect tactics, but jecting themselves to experimentation, and it is a concern when campaign workers are techniques that campaigns are willing to interacting with subjects. In laboratory set- execute. tings, variance in treatment is typically solved Researchers can attempt to include all reg- by limiting oversight and implementation of istered voters in an experiment and make the experiment to one or two people. This assignments to treatment and control groups. solution is not practical in large voter mobi- However, as discussed, the treatment will not lization experiments where hundreds of thou- be applied to all subjects. Subjects can be use- sands of households can be included in the fully divided into those who are successfully experiment. Conversations that are rushed treated (compliers and always-takers) and Voter Mobilization 235 those who are not (noncompliers) (Angrist, scope of an experiment and add verisimili- Imbens, and Rubin 1996). As an epistemolog- tude, but organizations have competing goals ical matter, it is impossible to know the effect that compromise research design. Because of of the treatment on noncompliers because objections from the organization being stud- they do not accept the assigned treatment ied, theories are rarely tested cleanly. Experi- by definition. Thus, conditioning on contact ments can be designed to minimize the direct only provides researchers with the average and indirect cost to campaigns, but the trade- treatment effect on those contacted. People off is nearly unavoidable. For example, groups who cannot be contacted are likely to be regularly resist removing a control group different from people who can be contacted from their target pool of potential voters, (Arceneaux et al. 2006), so the extent to which either because they overestimate their abil- the results apply to the uncontacted is an open ity to gather enough volunteers and reach all question. Raising contact rates can address voters in a particular community or because some concerns about external validity, but they believe it will hurt their reputation if without 100-percent compliance it is impossi- they do not reach out to all individuals who ble to know what would happen if all targeted would expect to be contacted. Organizations individuals were successfully contacted. also resist trying new techniques proposed by Researchers are also limited by the types of researchers and prefer to use familiar tech- campaigns that agree to cooperate with them. niques used by the group in past campaigns. Specifically, campaigns are likely to agree to Some of these objections can be overcome by randomize their contacts only when they have offering additional resources in exchange for limited resources or if they do not believe the cooperation, encouraging groups to provide experiment will influence the outcome of the an honest estimate of organizational capac- election. Given the high level of uncertainty ity, and designing experiments to minimize of most political candidates, as well as the the bureaucratic burden on managers. Still, contradictory and expensive advice of cam- cutting-edge research is difficult to orches- paign consultants, this generally has meant trate with existing campaigns and organiza- that political parties and candidates have tions. declined to participate in field experiments.4 Researchers constructing their own cam- To date, only one high-profile campaign, that paign have more freedom; that is, they are of Rick Perry in the 2006 Texas guberna- still limited by internal institutional review torial race, has agreed to participate in a board (IRB) requirements and federal law, nonproprietary experimental study (Gerber but their efforts may not mimic actual cam- et al. 2007). The bulk of experiments has paign behavior. For example, researchers are been conducted by nonpartisan 501(c)3 civic likely to be constrained by tax laws prevent- organizations, many of which have a strong ing research dollars from pursuing partisan incentive to cooperate because funders aims, thereby limiting much of their research increasingly want such efforts to include to nonpartisan appeals. Conducting free- experimental evaluation components. If well- standing campaigns also opens researchers funded and highly salient campaigns behave to a host of ethical considerations that differently and/or voters respond differently are largely not present when working with to outreach from brand name organizations, an organization already intervening in the then external validity is a real concern for community. much of the mobilization literature. A primary tension in the experimental mobilization literature is between theory 4. Problems Facing Field and authenticity. Working with an actual Experiments: Ethical Concerns campaign or organization can expand the Voter mobilization scholars interacting with 4 Most partisan experiments conducted to date have real-world politics and political campaigns been proprietary in nature. have the potential to change real-world 236 Melissa R. Michelson and David W. Nickerson outcomes. Thus, they face ethical obliga- eral ends, violate privacy, or cause psycholog- tions that likely exceed limits that might be ical distress. For instance, flyers announcing imposed by internal IRBs. The first ethical that elections are held on Wednesday may concern is that conducting experiments in be an effective campaign tactic, but testing actual electoral environments can present a such a tactic violates the democratic norm situation where a researcher could swing a of broad participation. Similarly, voter files close election. Most high-profile elections are make accessible to scholars massive amounts decided by large margins, but even here there of personal information. As with all research are well-known exceptions, such as the nar- that involves human subjects, the privacy row victories of George W. Bush in Florida of individuals must be respected. Scholars in 2000, Christine Gregoire in the Washing- should take steps to anonymize data as thor- ton gubernatorial election of 2004, and Al oughly as possible when sharing with other Franken in the 2008 Senate race in Min- academics and research assistants. nesota. Local elections are much more fre- Even when information is used legally, quently decided by small margins, many by it can cause private citizens to feel that only a few dozen votes. Thus, even a non- their privacy has been violated and gener- partisan voter mobilization campaign could ate adverse consequences, as illustrated by a swing an election by increasing voter turnout recent set of experiments conducted by Ger- in one neighborhood but not another. Avoid- beretal.(2008) and Panagopoulos (2009b). In ing experiments that could potentially alter both cases, researchers indicated to treatment electoral outcomes may well reduce allega- group individuals that their voting history tions of tampering; however, such a strategy was public knowledge and that they would be may limit the external validity and useful- broadcasting their election behavior – either ness of GOTV research.5 Without research via mailings or newspaper advertisements – in tightly contested partisan settings, scholars to their neighbors. No laws were broken, yet are limited in the conclusions they can draw individuals in the treatment groups were hor- about when mobilization works and which rified to learn that their private voting behav- types of messages are most persuasive. ior might be made public, to the extent that, In contrast, working in cooperation with in the latter case, they contacted their local real campaigns does mitigate some ethical district attorneys and the researcher was con- concerns. Working with campaigns means tacted by law enforcement. Regardless of the that an experiment simply systematizes an legality of such experiments, scholars might activity that would take place in any case. think twice about trying to replicate or build A control group or ineffective experimen- on this sort of work. As data about people tal treatment to be tested could swing an become increasingly available for purchase election, but the decision is ultimately made or harvest from the web, researchers should by the candidate or civic group studied, not by limit what may be considered violations of the researcher. Working with organizations privacy, even if they are using public data. engaged in campaigns immunizes researchers to some extent from ethical concerns about election outcomes. 5. Future Directions Yet, much as doctors and psychologists face dilemmas on whether to monitor govern- This chapter focuses on voter turnout in ment torture, researchers must consider care- particular because it is the best developed fully whether they want to be involved in and experimental literature with regard to mobi- lend validity to campaigns that pursue illib- lization, yet much remains to be explored. New technologies will need to be tested, such 5 Ideally, researchers could work with groups on both as interactive text messaging, nanotargeting sides of the partisan divide to avoid appearances of advertisements, and the numerous peer-to- bias. In practice, partisan organizations are generally suspicious, and researchers are likely to be forced to peer activities pioneered by MoveOn.org (see specialize on one side or the other. Middleton and Green 2008). More theories Voter Mobilization 237 from related fields such as psychology (e.g., at stimulating turnout than are nonpartisan cognitive load), economics (e.g., prospect appeals. theory), and sociology (e.g., social cohesion) The reason for the dearth of studies in this can be applied to the voter mobilization set- area is the difficulty in measuring the depen- ting. More can be learned about the dynamics dent variable. Whereas voter turnout is a pub- of information flow in campaigns. The avail- lic record in the United States, vote choice is ability of inexpensive mobile computing plat- private. Thus, researchers must either ran- forms (e.g., Palm Pilot, BlackBerry, iPhone) domize precincts and measure precinct-level will afford researchers the luxury of better vote choice or survey individuals after the data and the ability to execute more sophis- election. The precinct-level strategy has two ticated experiments. It is also likely that the primary downsides. First, treating a sufficient effectiveness of tactics will vary over time, and number of precincts to draw valid inference these shifts should be documented. requires very large experiments that are often Moving beyond turnout to vote choice is beyond the budget of experimenters. Second, another area where future research is likely randomizing at the precinct level precludes to make major inroads. Some nonpropri- the analysis of subgroups of interest because etary research has been done on partisan or precinct-level vote totals cannot be disag- persuasive campaigns, but the results from gregated. Although surveying subjects after these experiments differ wildly. For exam- an election solves the subgroup problem, it ple, Gerber (2004) examines the results of introduces problems of its own. Such sur- several field experiments conducted in coop- veys are expensive, and nonresponse rates are eration with actual candidates to estimate often high, leading to problems with exter- the effect of mailings. Preferences are mea- nal validity and concerns that subject attri- sured by examining ward-level returns for tion may not be equal across treatment and two experiments randomized at the ward level control groups. and by conducting postelection surveys for Civic participation is much broader than three experiments randomized at the house- the act of voting. Citizens (and noncitizens) hold level. For the ward-level experiments, attend meetings, volunteer for organizations, mailings sent by the incumbent had a signifi- donate to campaigns, lobby elected officials, cant effect on vote choice in the primary but and engage in a host of activities. In princi- not in the general election. In contrast, for ple, these topics are amenable to experimen- the three household-level experiments, in- tal study. For example, several experiments cumbent mail did not affect vote choice, have explored charitable giving. Han (2009) whereas challenger mail had statistically sig- randomly changed an appeal to buy a $1 nificant and politically meaningful effects. bracelet to support Clean Water Action (a Similarly intriguing results are reported by national environmental group) by adding two Arceneaux (2007), who found that canvassing sentences of personal information about the by a candidate, or by a candidate’s supporters, requester, meant to trigger a liking heuris- increased support for that candidate (as mea- tic. Individuals who were asked to donate sured by a postelection survey), but did not and who received the appeal with the added alter voters’ beliefs about the candidate. It is personal information were twice as likely to not even clear whether partisan or nonparti- donate. Miller and Krosnick (2004) randomly san campaigns are better at mobilizing voters. varied the text of a letter soliciting donations To convincingly answer the question, parti- to the National Abortion and Reproductive san and nonpartisan messages must be tested Rights Action League (NARAL) of Ohio. head to head; such experiments are rare and The control letter included the same sort inconclusive (Michelson 2005; Panagopoulos of language usually found in such fundrais- 2009a). In short, the field is wide open for ing letters, a “policy change threat” letter ambitious scholars to understand what factors warned that powerful members of Congress influence individual vote choice and whether were working hard to make abortions more partisan appeals are more or less effective difficult to obtain, and a “policy change 238 Melissa R. Michelson and David W. Nickerson opportunity” letter claimed powerful mem- experiments have expanded our understand- bers of Congress were working hard to make ing of when and how voter mobilization cam- abortions easier to obtain. Recipients of the paigns work to move individuals to the polls. letters, all Democratic women, were asked Despite real-world hazards such as threaten- to make a donation to NARAL Ohio and to ing dogs, contaminated control groups, and sign and return a postcard addressed to Pres- uneven canvasser quality, hundreds of efforts ident Clinton. Only the threat letter had a have replicated and extended the initial find- significant effect on financial contributions, ings offered by Gerber and Green (2000). whereas only the opportunity letter had a sig- Experiments have been conducted in a vari- nificant effect on returned postcards. Future ety of electoral contexts, with a variety of tar- experiments could expand on these results to geted communities, and exploring a variety study campaign donations. of psychological theories. This volume offers Another emerging area of field experi- additional details about experiments in voter ments explores how citizen lobbying affects mobilization, including Chong’s chapter in roll call votes in state legislatures. Bergan this volume on work with minority voters and (2007) conducted an experiment in cooper- Sinclair’s chapter in this volume on the power ation with two public health–related groups of interpersonal communication. Yet, much aiming to win passage of smokefree work- work remains to be done. We look forward place legislation in the lower house of the to the next generation of experiments, which New Hampshire legislature. Group members in addition to refining existing results will were sent an e-mail asking them to send an include more new technologies, richer the- e-mail to their legislators; e-mails intended oretical underpinnings, more work on parti- for legislators selected for the control group san and persuasive campaigns, and behaviors were blocked, whereas e-mails intended for beyond turnout. legislators selected for the treatment group were sent as intended. Controlling for past votes on tobacco-related legislation, the e- References mails had a statistically significant effect on two pivotal votes. This form of political mobi- Adams, William C., and Dennis J. Smith. 1980. lization is increasingly common among grass- “Effects of Telephone Canvassing on Turnout roots organizations and worthy of further and Preferences: A Field Experiment.” Public 44 53 83 study. Opinion Quarterly : – . Angrist, Joshua D., Guido W. Imbens, and Don- Nearly every civic behavior could be 1996 studied using experiments if enterprising ald B. Rubin. . “Identification of Causal Effects Using Instrumental Variables.” Journal researchers were to partner with civic orga- of the American Statistical Association 91: 444– nizations. In exchange for randomly manipu- 55. lating the appeals to members of the group Arceneaux, Kevin T. 2007. “I’m Asking for Your (or the broader public) and measurement Support: The Effects of Personally Delivered of the outcome of interest (e.g., meet- Campaign Messages on Voting Decisions and ing attendance), organizations could learn Opinion Formation.” Quarterly Journal of Polit- how to maximize the persuasiveness of their ical Science 2: 43–65. appeals to attract the largest possible set of Arceneaux, Kevin, Alan S. Gerber, and Donald 2006 volunteers, donors, or activists. The work on P. Green. . “Comparing Experimental and Matching Methods Using a Large-Scale Voter voter turnout can serve as a useful template 14 for these types of studies. Mobilization Experiment.” Political Analysis : 1–36. Arceneaux, Kevin, and David W. Nickerson. 2009. “Who Is Mobilized to Vote? A Re-Analysis 6. Conclusion of 11 Field Experiments.” American Journal of Political Science 53: 1–16. Since the modern launch of the subfield Bergan, Daniel E. 2007. “Does Grassroots Lob- less than a decade ago, hundreds of field bying Work?: A Field Experiment Measuring Voter Mobilization 239

the Effects of an e-Mail Lobbying Campaign on Gosnell, Harold F. 1927. Getting-Out-the-Vote: An Legislative Behavior.” American Politics Research Experiment in the Stimulation of Voting. Chicago: 37: 327–52. The University of Chicago Press. Dale, Allison, and Aaron Strauss. 2007. “Text Green, Donald P., and Alan S. Gerber. 2008. Get Messaging as a Youth Mobilization Tool: An Out the Vote: How to Increase Voter Turnout. Experiment with a Post-Treatment Survey.” 2nd ed. Washington, DC: Brookings Institu- Paper presented at the annual meeting of tion Press. the American Political Science Association, Green, Donald P., and Lynn Vavreck. 2008. Chicago. “Analysis of Cluster-Randomized Field Exper- Eldersveld, Samuel J. 1956. “Experimental Pro- iments.” Political Analysis 16: 138–52. paganda Techniques and Voting Behavior.” Greenwald, Anthony G., Catherine G. Carnot, American Political Science Review 50: 54–65. Rebecca Beach, and Barbara Young. 1987. Fishbein, Martin, and Icek Ajzen. 1975. Belief, “Increasing Voting Behavior by Asking Peo- Attitude, Intention and Behavior: An Introduction ple if They Expect to Vote.” Journal of Applied to Theory and Research. Reading, MA: Addison Psychology 72: 315–18. Wesley. Ha, Shang E., and Dean S. Karlan. 2009.“Get- Fisher, Ronald A. 1925. Statistical Methods for Out-the-Vote Phone Calls: Does Quality Mat- Research Workers. London: Oliver & Boyd. ter?” American Politics Research 37: 353–69. Gaines, Brian J., James H. Kuklinski, and Paul Han, Hahrie C. 2009. “Does the Content of Polit- J. Quirk. 2007. “The Logic of the Survey ical Appeals Matter in Motivating Participa- Experiment Reexamined.” Political Analysis 15: tion? A Field Experiment on Self-Disclosure 1–20. in Political Appeals.” Political Behavior 31: 103– Gerber, Alan S. 2004. “Does Campaign Spending 16. Work? Field Experiments Provide Evidence Imai, Kosuke. 2005. “Do Get-Out-the-Vote Calls and Suggest New Theory.” American Behavioral Reduce Turnout? The Importance of Statisti- Scientist 47: 541–74. cal Methods for Field Experiments.” American Gerber, Alan S., and Donald P. Green. 2000. Political Science Review 99: 283–300. “The Effects of Canvassing, Direct Mail, and Michelson, Melissa R. 2005. “Meeting the Chal- Telephone Contact on Voter Turnout: A Field lenge of Latino Voter Mobilization.” Annals of Experiment.” American Political Science Review Political and Social Science 601: 85–101. 94: 653–63. Michelson, Melissa R., Lisa Garcıa´ Bedolla, and Gerber, Alan S., Donald P. Green, and Christo- Donald P. Green. 2007. “New Experiments pher W. Larimer. 2008. “Social Pressure and in Minority Voter Mobilization: A Report on Voter Turnout: Evidence from a Large-Scale the California Votes Initiative.” In Insight.San Field Experiment.” American Political Science Francisco: The James Irvine Foundation. Review 102: 33–48. Michelson, Melissa R., Lisa Garcıa´ Bedolla, and Gerber, Alan S., Donald P. Green, and Ron Donald P. Green. 2008. “New Experiments Shachar. 2003. “Voting May Be Habit Form- in Minority Voter Mobilization: Second in a ing: Evidence from a Randomized Field Exper- Series of Reports on the California Votes Ini- iment.” American Journal of Political Science 47: tiative.” In Insight. San Francisco: The James 540–50. Irvine Foundation. Gerber, Alan, James G. Gimpel, Donald P. Green, Michelson, Melissa R., Lisa Garcıa´ Bedolla, and and Daron R. Shaw. 2007. “The Inﬂuence of Donald P. Green. 2009. “New Experiments in Television and Radio Advertising on Candidate Minority Voter Mobilization: Final Report on Evaluations: Results from a Large-Scale Ran- the California Votes Initiative.” In Insight.San domized Experiment.” Paper presented at the Francisco: The James Irvine Foundation. annual meeting of the Midwest Political Sci- Michelson, Melissa R., Lisa Garcıa´ Bedolla, and ence Association, Chicago. Margaret A. McConnell. 2009.“Heedingthe Gerber, Alan S., and Todd Rogers. 2009. Call: The Effect of Targeted Two-Round “Descriptive Social Norms and Motivation to Phonebanks on Voter Turnout.” Journal of Pol- Vote: Everybody’s Voting and So Should You.” itics 71: 1549–63. Journal of Politics 71: 178–91. Middleton, Joel A., and Donald P. Green. 2008. Gollwitzer, Peter M. 1999. “Implementation In- “Do Community-Based Voter Mobilization tentions: Strong Effects of Simple Plans.” Campaigns Work Even in Battleground States? American Psychologist 54: 493–503. Evaluating the Effectiveness of MoveOn’s 240 Melissa R. Michelson and David W. Nickerson

2004 Outreach Campaign.” Quarterly Journal Panagopoulos, Costas. 2009b. “Turning Out, of Political Science 3: 63–82. Cashing In: Extrinsic Rewards, Extrinsic Miller, Joanne A., and Jon A. Krosnick. 2004. Rewards, Intrinsic Motivation and Voting.” “Threat as a Motivator of Political Activism: A Paper presented at the annual meeting of Field Experiment.” Political Psychology 25: 507– the Midwest Political Science Association, 23. Chicago. Miller, Roy E., David A. Bositis, and Delise L. Rind, Bruce, and Daniel Benjamin. 1994. “Effects Baer. 1981. “Stimulating Voter Turnout in of Public Image Concerns and Self-Image on a Primary: Field Experiment with a Precinct Compliance.” Journal of Social Psychology 134: Committeeman.” International Political Science 19–25. Review 2: 445–60. Rosenstone, Steven J., and John Mark Hansen. Nickerson, David W. 2005. “Scalable Protocols 1993. Mobilization, Participation, and Democracy Offer Efﬁcient Design for Field Experiments.” in America. New York: Macmillan. Political Analysis 13: 233–52. Sartori, Anne E. 2003. “An Estimator for Some Nickerson, David W. 2006. “Hunting the Elusive Binary-Outcome Selection Models without Young Voter.” Journal of Political Marketing 5: Exclusion Restrictions.” Political Analysis 11: 47–69. 111–38. Nickerson, David W. 2007. “Quality Is Job Sherman, Steven J. 1980. “On the Self-Erasing One: Volunteer and Professional Phone Calls.” Nature of Errors of Prediction.” Journal of Per- American Journal of Political Science 51: 269– sonality and Social Psychology 39: 211–21. 82. Smith, Jennifer K., Alan S. Gerber, and Anton Nickerson, David W. 2008. “Is Voting Conta- Orlich. 2003. “Self Prophecy Effects and Voter gious? Evidence from Two Field Experiments.” Turnout: An Experimental Replication.” Polit- American Political Science Review 102: 49–57. ical Psychology 24: 593–604. Nickerson, David W., and Todd Rogers. 2010. Vavreck, Lynn. 2007. “The Exaggerated Effects of “Do You Have a Voting Plan? Implementation Advertising on Turnout: The Dangers of Self- Intentions, Voter Turnout, and Organic Plan Reports.” Quarterly Journal of Political Science 2: Making.” Psychological Science 21: 194–99. 287–305. Panagopoulos, Costas. 2009a. “Partisan and Non- Verba, Sidney, Kay Lehman Schlozman, and partisan Message Content and Voter Mobi- Henry E. Brady. 1995. Voice and Equality: Civic lization: Field Experimental Evidence.” Polit- Voluntarism in American Politics. Cambridge, ical Research Quarterly 62: 70–76. MA: Harvard University Press. Part V

INTERPERSONAL RELATIONS

CHAPTER 17

Trust and Social Exchange

Rick K. Wilson and Catherine C. Eckel

Trust and its complement, trustworthiness, The definition of trust is muddied by the are key concepts in political science. Trust fact that two distinct research methods have is seen as critical for the existence of sta- been used to explore it. Early research treats ble political institutions, as well as for the trust as a perception of norms in a soci- formation of social capital and civic engage- ety, assessed using survey questions about ment (Putnam 1993, 2000; Stolle 1998). the trustworthiness or fairness of others. For It also serves as a social lubricant that forty years, the General Social Survey (GSS), reduces the cost of exchange, whether in World Values Survey (WVS), and Ameri- reaching political compromise (Fenno 1978; can National Election Studies (ANES) have Bianco 1994) or in daily market and nonmar- relied on the same questions to evaluate trust. ket exchange (Lupia and McCubbins 1998; In contrast, recent research has turned to Sztompka 1999;Knight2001). Researchers in behavioral assessments of trust using incen- this area face three key challenges. First, the tivized, economics-style laboratory experi- concept of trust has been used in a multiplicity ments; this work is the focus of this chapter. of ways, leaving its meaning unclear. Second, For the most part, behavioral research uses it is used to refer both to trust in government the investment game (Berg, Dickhaut, and and trust among individuals (interpersonal McCabe 1995), where an individual decides trust). Third, it is sometimes seen as a cause whether to trust another by deciding to put and sometimes as an effect of effective politi- his or her financial well-being into the other’s cal institutions, leaving the causal relationship hands. The relationship between these two between trust and institutions unclear. concepts of trust – the survey measure of perceived trustworthiness of others and the Seonghui Lee and Cathy Tipton provided research assis- decision by a laboratory participant to trust tance. Comments from Salvador Vazquez´ del Mercado, his or her counterpart – is relatively weak, David Llanos, Skip Lupia, and Jamie Druckman signifi- yet both are used to assess the levels of cantly improved the chapter. We gratefully acknowledge support by the National Science Foundation (grant nos. trust and reciprocity in a group or soci- SES-0318116 and SES-0318180). ety. In Section 1, we present a framework

243 244 Rick K. Wilson and Catherine C. Eckel for categorizing concepts of trust, placing 1. A Framework for our discussion of behavioral trust in a richer Interpersonal Trust context. Second, there is a difference between trust Interpersonal trust goes by many names: in government and trust among individu- generalized trust, moral trust, particularized als. Trust in government is addressed in a trust, encapsulated trust, dyadic trust, and number of studies, including Miller (1974), so on. Nannestad (2008) provides a useful Citrin (1974), Hetherington (1998), and Hib- framework for considering the broad range bing and Theiss-Morse (2002), who note how of research on trust. He argues that trust political trust varies with assessments of, for can be organized along two dimensions: the example, the presidency and Congress. Levi first ranging from general to particular, and and Stoker (2000) provide an overview of the second ranging from rational to moral. this work. Although trust among citizens is On the general/particular dimension, gener- interrelated with political trust, the two are alized trust is represented by the GSS, WVS, conceptually distinct. Behavioral research has and ANES questions, which ask individuals exclusively considered trust between indi- to assess the degree of fairness and trust- viduals, and only rarely dealt directly with worthiness of “most people.” At the other trust in political or governmental institutions. extreme, particularized trust refers to situ- Because of our focus on behavioral trust, we ations where an individual decision to trust leave the evaluation of trust in institutions to has a specific target and content (individual A others. trusts individual B with respect to X). On the A third issue is the complex causal rela- rational/moral dimension, concepts such as tionship between societal levels of trust and Hardin’s (2002) “encapsulated trust” fall close the effectiveness and legitimacy of political to the rational end of the spectrum. Here, institutions. Although the argument has been trust is based on a calculation of expected made that trust affects and is affected by polit- return and depends on the assessed trust- ical institutions, and that it plays an important worthiness of others. Uslaner (2002), who role in limiting or enhancing the effectiveness conceptualizes trust and trustworthiness as of those institutions, the causal relationship moral obligations akin to norms that have has been difficult to disentangle from survey- been shaped by childhood experiences, falls based data. We argue that behavioral research on the other end of the spectrum. can play an important role in addressing this Survey research is perhaps best suited to important problem. assess trust in the general/moral quadrant of This chapter thus focuses on the con- Nannestad’s (2008) typology, whereas exper- tributions of behavioral research – in par- imental studies of dyadic trust are in the particular, economics-style experimental studies ticular/rational quadrant. This helps clarify of trust in a dyadic exchange transaction – why these two approaches to measuring trust to an understanding of trust. We ask what are so weakly related. Glaeser et al. (2000), laboratory-based studies of dyadic trust can for example, find little statistical correlation contribute to answering the aforementioned between an individual’s answers to the GSS questions. Section 1 presents a framework questions and his or her behavior in the for understanding dyadic trust. In Section 2, investment game. Both approaches to mea- we turn to the canonical trust experiment, suring trust have value, but they address dif- and in Section 3, we examine the individual- ferent aspects of the concept. We believe that level correlates of trust. Section 4 discusses laboratory experiments are uniquely situated the strategic aspects of trust, and Section to answer questions in the particular/rational 5 details cross-cultural research on trust. quadrant noted by Nannestad. We do not Section 6 evaluates the link between trust claim that experiments are a panacea, but and institutions. Finally, Section 7 concludes rather that they provide a useful tool to add to with what we consider to be the unanswered what we already know from the rich literature questions. on trust. Trust and Social Exchange 245

In its simplest form, trust involves a strate- protocols, allowed researchers to examine the gic relationship between two actors, where correlation between behavior and individual reciprocated trust can improve the well-being characteristics (including neuroscience inno- of both members of the relationship. In polit- vations), provided an environment to study ical science, this includes negotiations among stereotyping and discrimination, and served legislators to trade votes, the decision of a as a platform for cross-cultural compari- voter to give latitude to a representative, or son. In addition, it has allowed researchers the willingness of a citizen to comply with to examine existing institutional mechanisms the decision of a public official. This concept and test bed new institutions. In the remain- of dyadic trust is similar to Hardin’s (2002) der of this chapter, we examine these aspects view of “encapsulated” trust, where two actors of trust experiments. know something about one another, the context of exchange is clear, and what is being entrusted is well defined. In the case of leg- 2. The Trust Game islators engaged in vote trading, for example, one legislator faces the problem of giving up The trust game consists of a sequence of a vote with the future promise of reciprocity. moves between two actors, where both are Knowing the reputation of the other party is fully informed about its structure and payoffs. critical to this choice. To illustrate, suppose there are two actors, Both parties in a dyadic trust relation- players A and B. Both are endowed with $10 ship have important problems to solve. The by the experimenter. Player A has the right truster faces a strategic problem: whether to move first and can choose to keep the $10 and how much to trust the trustee. His or or pass any part of it to the second player. her decision depends critically on expecta- Any amount that is passed is tripled by the tions about the trustee. The problem for the experimenter and then delivered to player trustee is to decide, if trusted, whether and B. (The tripling plays the part of a return how much to reciprocate that trust. His or on investment in the game.) Player B now her problem is arguably easier because reci- has his or her original $10 and the tripled procity is conditional on the revealed trust amount passed to him or her and is given of the first mover. Focusing on dyadic trust the option to send money back to player A. illustrates an element that is often missing The amount can range from $0 to the full in discussions of interpersonal trust: trust tripled value. Player A’s move is “trust,” in depends on whom (or what) one is dealing that by sending a positive amount, he or she with. Individuals do not trust in the abstract, entrusts his or her payoff to player B; player but rather with respect to a specific target and B’s move is “trustworthiness” or reciprocity. in a particular context. Although the deci- From a game-theoretic perspective, a naive, sion about whether and how to reciprocate payoff-maximizing player B would retain any- does not carry the same level of risk for the thing sent to him or her; player A, knowing trustee, the decision is made in a specific con- this, will send him or her nothing. Thus, the text, with attendant norms of responsibility or equilibrium of the game (assuming payoff- obligation. This strategic interaction is diffi- maximizing agents) is for player A to send $0, cult to explore in the context of survey-based rightly failing to believe in player B’s trust- observational studies but is well suited to the worthiness. laboratory. The canonical implementation of this Since the mid-1990s, more than 150 exper- game has the following characteristics: imental studies have examined dyadic trust. r The standard trust experiment, originally Subjects are recruited from the general known as the “investment game” (Berg et al. student population and paid a nominal fee 1995), has proven to be a valuable vehicle for (usually $5) for attending. r subsequent research. It has given rise to new Subjects are randomly assigned to the role methodological innovations in experimental of player A or player B. 246 Rick K. Wilson and Catherine C. Eckel r Brief instructions are read aloud, followed Wilson 2004); sixty-two percent send half or by self-paced computerized instructions more of this endowment. Although subjects and a comprehension quiz. are sensitive to size of stakes, the data clearly r Each player is endowed with an equal show that people trust – and trust pays – even amount of money (usually $10). when the stakes are high. r Partners are kept anonymous. Another common complaint is that stu- r A brief questionnaire collects demogra- dents coming into the laboratory are friends phic and other information. and/or anticipate postgame repeated play. r Subjects are paid their actual earnings in High levels of trust may simply be due cash, in private, at the end the experi- to subjects investing in reputation. Indeed, ment. Anderhub, Englemann, and Guth (2002) and Engle-Warnick and Slonim (2004) find In contrast to the Nash equilibrium, a meta- reputational effects when subjects repeat- analysis of results by Johnson and Mislin edly play the trust game. Isolating reci- (2008) shows that, on average, trusters send procity from an investment in reputation is 50.8 percent of their endowment (based on important, and experimenters address this by eighty-four experiments). Trust pays (barely), taking considerable care to ensure that sub- in that 36.5 percent of what is sent is jects do not know one another in the same returned (based on seventy-five experiments), experimental session. To induce complete just more than the 33.3 percent that com- anonymity, Eckel and Wilson (2006) conduct pensates player A for what was sent. Con- experiments over the Internet, with subjects trary to game-theoretic expectations, trust is matched with others at another site more than widespread and is reciprocated. 1,000 miles away. The subsequent play of the game is within the range observed in other studies. Methodological Issues A specific complaint is that the game does As others in this volume note, political sci- not really measure trust, but rather another entists are sometimes skeptical of what labo- element such as other-regarding preferences ratory experiments can tell us. Experiments (altruism). Glaeser et al. (2000) ask subjects seem contrived, the sample is too limited, to report the frequency of small trusting and the motivations of subjects often seem acts – leaving a door unlocked, loaning money trivial (see, e.g., Dickson’s, Druckman and to a friend – and find positive correlations Kam’s, and McDermott’s chapters in this between these actions and the trust game. At volume). A common complaint about labo- the same time, they include the standard bat- ratory experiments is that, even if subjects tery of survey questions and find that they are paid, the stakes are insufficient to mimic are uncorrelated with trust, but instead are natural settings. Johansson-Stenman, Mah- positively associated with trustworthiness. In mud, and Martinsson (2005) ask whether the same vein, Karlan (2005) finds that the stakes affect behavior in the trust game con- repayment of microcredit loans is positively ducted in Bangladesh, with the highest-stakes correlated with trustworthiness in the trust game being twenty-five times greater than game, but not with trust. Cox (2004) explicitly the lowest-stakes game. The high-stakes set- tests the role of altruism as a motive for trust ting has the U.S. dollar price parity equiv- and trustworthiness and finds positive levels alent of $1,683. They find that, as the size of trust, even when controlling for individual- of stakes increases, a somewhat smaller per- level altruism. In sum, the survey measures of centage of the money is sent (thirty-eight per- trust have weak correlation with behavioral cent in the high-stakes condition compared to trust, but they seem to predict trustworthi- forty-six percent in the middle-stakes condi- ness, indicating that the surveys may be more tion). In experiments carried out in Tatarstan accurate measures of beliefs about trustwor- and Siberia, subjects are given an endowment thiness in society. Evidence from variations equivalent to a full day’s wage (Bahry and on the games supports the idea that they Trust and Social Exchange 247 constitute valid measures of trust and trust- between expectations about trust and real- worthiness. ized trust. Sutter and Kocher (2007) obtain a similar finding using six age cohorts ranging from eight-year-old children to sixty- 3. Correlates of Trust eight-year-old subjects. They find two clear effects. Trusting behavior is nonlinear with Observational studies point to heterogeneity age, with the youngest and oldest cohorts in generalized trust within a given popula- trusting the least, and the twenty-two- and tion. Uslaner (2002), for example, finds that thirty-two-year-old cohorts contributing the generalized trust is positively correlated with most. However, reciprocity is almost linearly education and that African Americans report related to age, with the oldest cohort return- lower levels of trust. Experimenters also ask ing the most. These age cohort effects are what factors are correlated with trust and similar to those reported by Uslaner (2002) reciprocity between individuals and corrob- using survey data. orate several of these findings. We first detail Several studies examine religion and trust. results about observable individual character- Anderson, Mellor, and Milyo (2010) report istics and then turn to a separate discussion of little effect on trust or trustworthiness of reli- underlying neural mechanisms. gion, regardless of denomination. This is contrary to the findings by Danielson and Holm (2007), who find that churchgoers in Tanza- Individual Characteristics nia reciprocate more than does their student Trust experiments have examined the rela- sample. Johansson-Stenman, Mahmud, and tionship between personal characteristics, Martinsson (2009) match Muslims and Hin- such as gender and ethnicity, and behavior dus, both within and across religion, and find in the games. In a comprehensive survey of no difference in any of their matching condi- gender differences in experiments, Croson tions; they cautiously conclude that religious and Gneezy (2009) find considerable varia- affiliation does not matter. Together these tion across twenty trust game studies. Many findings show that there are small effects for demonstrate no difference in the amount standard socioeconomic status variables on sent, but among the twelve that do, nine behavioral trust. show that men trust more than do women. Some have conjectured that trust is a risky Among the eight studies demonstrating a decision and that observed heterogeneity in difference in trustworthiness, six show that trust may be due in part to variations in risk women reciprocate more. They argue that tolerance. In our own work, we directly test the cross-study variation is due to women’s this conjecture by supplementing the canon- greater response to subtle differences in the ical experiment with several different mea- experimental protocols. sures of risk tolerance, ranging from survey Trust is also rooted in other aspects of measures to behavioral gambles with stakes socioeconomic status. Although experiments that mirror the trust game (Eckel and Wilson with student subjects rarely find an effect of 2004). None is correlated with the decision to income on behavior (although see Gachter, trust (or to reciprocate). In contrast, Bohnet Herrmann, and Thoni 2004), several recent and Zeckhauser (2004) focus on the risk of studies use representative samples and find betrayal. They use a simplified trust game positive relationships between income and with a limited set of choices and implement trust behavior (Bellemare and Kroger¨ 2007; a mechanism that elicits subjects’ willingness Naef et al. 2009). Age is also related to to participate in the trust game, depending trust and reciprocity. Bellemare and Kroger¨ on the probability that their partner is trust- (2007) find that the young and the elderly worthy. They find considerable evidence for have lower levels of trust, but higher levels betrayal aversion, with trusters sensitive to of reciprocity than do middle-aged individu- the potential actions of the population of als, a result that they attribute to a mismatch trustees. Indeed, trusters are less willing to 248 Rick K. Wilson and Catherine C. Eckel accept a specified risk of betrayal by trustees trust will be reciprocated. The neural system than to risk a roll of the dice with the same that they isolate is clearly related to process- probability of attaining a high payoff. In a ing social behavior and not simply due to later paper, Bohnet et al. (2008) extend this internal rewards. Tomlin et al. (2006) report study to six countries and find variation in similar results when subjects are simultane- betrayal aversion across societies, with greater ously scanned in an fMRI while playing the betrayal aversion associated with lower lev- trust game. els of trust. Whether trust is a risky decision Several research groups show that the hor- seems to depend importantly on how risk is mone oxytocin (OT) is an important basis measured. Clearly, this is an area in which for cementing trust. It is proposed that OT more work is needed. is stimulated by positive interactions with a In addition to the preceding studies, lab- specific partner. Zak, Kurzban, and Matzner oratory experiments and survey-based stud- (2005) focus on a design to test for changes in ies reach similar conclusions with respect to OT levels for subjects playing the trust game several factors. Trust is positively associated with another human or playing with a ran- with level of education and income, a point dom device. They find elevated levels of OT noted by Brehm and Rahn (1997). Genera- for trustees assigned to the human condition. tional differences also emerge, a point that Behaviorally, they also observe an increase in scholars such as Inglehart (1997) and Put- reciprocation for those in the human con- nam (2000) offer as a cultural explanation for dition. There is no difference in OT for trust. trusters, indicating that the effect is absent for the trust decision. In contrast, Kosfeld et al. (2005) use a nasal spray to administer either Contributions from Biology OT or a placebo. They find that OT enhances and Neuroscience trust, but it is unrelated to trustworthiness. A promising arena for understanding indi- There is also evidence for a genetic basis vidual correlates of trust is linked with of trust. Cesarini et al. (2008) report on trust biological and neurological mechanisms. In experiments conducted with monozygotic principle, observational studies are equally (MZ) and dizygotic (DZ) twins in the United capable of focusing on these mechanisms. States and Sweden. Although the distribution However, most scholars focusing on such of trust and trustworthiness is heterogeneous, issues have a laboratory experimental bent. they find that MZ twins have higher corre- Several research teams focus on the neu- lations in behavior than do their DZ coun- rological basis of trust (for an overview, terparts. The estimated shared genetic effect see Fehr, Kosfeld, and Fischbacher 2005). ranges from ten to twenty percent in their McCabe et al. (2001) find differences in samples. As the authors admit, it is not all brain activation in the trust game when sub- about genes. A significant component of the jects play against a human partner as com- variation is explained by the twins’ environ- pared to a computer. They speculate what ments. the neural underpinnings might be for trust- The jury is still out concerning the biolog- ing behavior. Rilling et al. (2004) examine ical and neural mechanisms that drive trust neural reward systems for a setting similar and trustworthiness. Trust and reciprocation to the trust game, in which there is the pos- involve complex social behaviors, and the sibility of mutual advantage. Delgado, Frank, capacity to test the mechanisms that cause and Phelps (2005) focus on both reward and these behaviors remains elusive. learning systems that follow from iterated play with multiple partners in the trust game. King-Casas et al. (2005) also use an iter- 4. Trust and Stereotypes ated trust game and find not only reward and learning processes, but also anticipatory sig- Individuals vary systematically in their pro- nals in the brain that accurately predict when pensities to trust and to reciprocate trust, but Trust and Social Exchange 249 another source of behavioral heterogeneity truster is penalized and less is reciprocated results from the differences in how individuals (Wilson and Eckel 2006). are treated by others. Those who have stud- In another study (Eckel and Wilson 2008), ied campaigns (Goldstein and Ridout 2004; we show that skin shade affects expectations Lau and Brown Rovner 2009) or ethnicity about behavior. Darker-skinned trusters are and social identity (Green and Seher 2003; expected to send less, but send more than McClain et al. 2009) understand how impor- expected, and they are rewarded for their tant it is to control for specific pairings of unexpectedly high trust. The insight we gain individuals or groups. Voters respond differ- is not just from the expectations, but from the ently when they have information about a response to exceeded or dashed expectations. candidate – such as gender, race, or age – than Our findings concerning stereotypes are when they have abstract information about a not unusual. For example, trusters prefer to candidate. This is partly because beliefs about be paired with women, believing that women others are based on stereotypes. Stereotyp- will be more trustworthy. On average, they ing is the result of a natural human ten- are, but not to the extent expected (Cro- dency to categorize. Two possibilities arise. son and Gneezy 2009). Trusters send more First, the stereotype may accurately reflect to lighter-skinned partners, trusting them at average group tendencies, and so provide a higher rates, and beliefs about darker-skinned convenient cognitive shortcut for making partners are weakly supported (see also Fer- inferences about behavior. However, stereo- shtman and Gneezy 2001; Haile, Sadrieh, and types can be wrong, reflecting outdated or Verbon 2006; Simpson, McGrimmon, and incorrect information, and can subsequently Irwin 2007; Eckel and Petrie in press; Naef bias decisions in a way that reduces accu- et al. 2009). These findings are often masked racy. If trust leads to accumulating social in survey-based studies because subjects dis- capital, then decisions based on stereotypes play socially acceptable preferences when it will advantage some groups and disadvantage costs them nothing to do so. others, and negative stereotypes may become In our current work (Eckel and Wilson self-fulfilling prophesies. 2010), we introduce another change to the Although most of the experimental studies game by allowing subjects to select their have gone through great efforts to ensure that partners. Trusters view the photographs of subjects know nothing about one another, we potential partners after the trust experiment provide visual information to subjects about is explained to them, but before any decisions their partners. In one design, we randomly are made. They then rank potential counter- assign dyads and allow counterparts to view parts according to their desirability as a part- one another’s photograph. This enables us ner, from most to least desirable. To ensure to focus on the strategic implications of the that the ordering task is taken seriously, one joint attributes of players. To eliminate rep- truster is randomly drawn and given his or utation effects, we use subjects at two or her first choice, then a second truster is ran- more laboratories in different locations. Pho- domly drawn and given his or her first choice tographs are taken of each subject and then from those remaining, and so on (follow- displayed to their counterparts. For exam- ing Castillo and Petrie 2010). Not surpris- ple, to study the effect of attractiveness on ingly, when trusters choose their partners, the trust and reciprocity, we look at pairings in overall level of trust is higher. At the same which the truster is measured as more (or time, when subjects know they have been less) attractive than the trustee. We show chosen, they reciprocate at higher rates. Giv- that expectations are higher for more attrac- ing subjects some control over the choice of tive trusters and trustees: attractive trusters counterpart has a strong positive impact on are expected to send more, and attractive trust and trustworthiness. trustees are expected to return more. The Experiments that focus on the joint char- attractive truster inevitably fails to live up acteristics of subjects and that allow for choice to high expectations; as a consequence, the among partners are moving toward answering 250 Rick K. Wilson and Catherine C. Eckel questions about the importance of expecta- villages, usually in classrooms or libraries, and tions in strategic behavior and how expec- the quality and size of the facilities varied (as tations are shaped by characteristics of the did the temperature). pairing. Experiments are well suited to answer Their findings reveal high levels of trust these questions because of the ability to con- and reciprocity in these republics, despite trol information about the pairings of the sub- the fact that the political institutions are jects. regarded with suspicion. On average, 51.0 percent of the truster’s endowment was sent, and trustees returned 38.3 percent of what 5. Cross-Cultural Trust was received, a result very close to average behavior among U.S. students. These find- An ongoing complaint about experiments is ings indicate that trust is widespread in an that they lack external validity. The concern environment where it is unexpected (e.g., see is that the behavior of American university Mishler and Rose 2005). More important, students is not related to behavior in the gen- Bahry and Wilson (2004) point to strong general population, within or across different erational differences in norms that lead to cultures. Several recent experimental stud- distinct patterns of trust and trustworthiness – ies tackle the question of external validity by a finding that would have been unexplored looking at population samples, and consider- without a population sample. able work has taken place cross-culturally in Others have also generated new insights recent years. when conducting trust experiments outside The impetus to behaviorally measure trust university laboratories. Barr (2003) finds across cultures is partly driven by findings by considerable trust and little variation across Knack and Keefer (1997) and Zak and Knack ethnic groups in Zimbabwean villages. Car- (2001), who find that the level of generalized penter, Daniere, and Takahashi (2004), using trust in a country is correlated with economic a volunteer sample of adults in Thailand and growth. These findings, derived from surveys Vietnam, find that trust is correlated with for- and aggregate-level measures, mirror those mation of social capital, measured as own- by Almond and Verba (1963), who provide ing a home, participating in a social group, evidence that trust is correlated with demo- and conversing with neighbors. Karlan (2005) cratic stability. Researchers using trust exper- obtains a similar finding in Peru. He notes iments have entered this arena as well. that trust is related to social capital and shows Several studies focus on cross-cultural that trustworthiness predicts the repayment comparisons of trust using volunteer student of microcredit loans. Cronk (2007) observes subjects. These studies replicate the high lev- low levels of trust among the Maasai in a neu- els of trust found among U.S. students, while tral (unframed) experimental condition, and finding some variability across cultures (see, even less trust when the decision is framed e.g., Yamagishi, Cook, and Watabe 1998; to imply a long-term obligation. Numer- Buchan, Croson, and Dawes 2002; Ashraf, ous other studies have focused on trust in Bohnet, and Piankov 2006). Bangladesh (Johansson-Stenman, Mahmud, Bahry and Wilson (2004) are the first to and Martinsson 2006), Paraguay (Schechter extend these studies to representative pop- 2007), Kenya (Greig and Bohnet 2005), the ulation samples in two republics in Russia. United States, and Germany (Naef et al. They draw from a large sample of respon- 2009), and across neighborhoods in Zurich dents who were administered lengthy face- (Falk and Zehnder 2007). These studies are to-face interviews. A subset of subjects was important in that they aim at linking the trust randomly drawn to participate in laboratory- game to ethnic conflict, repaying loans, and like experiments in the field. Although some- the risk and patience of individuals. thing was gained in terms of confidence in Studies using culturally different groups external validity, a price was paid in terms of have moved beyond student samples and are a loss of control. Sessions were run in remote beginning to assure critics that concerns with Trust and Social Exchange 251 external validity are misplaced. As measured that institutions are crucial for fostering cit- by the trust game, trust and trustworthiness izen trust, then the legitimacy of the insti- permeate most cultures. These studies are tutions within which individuals make trust beginning to give us insight into cultural vari- decisions should be directly related to the ation. How these studies are linked to key degree of trust and trustworthiness. Labora- questions of support for democratic institu- tory experiments are especially well suited to tions or increasing political participation have examine these causal relations. not routinely been addressed. One type of institutional mechanism introduces punishment into the exchange. We consider two punishment mechanisms. The 6. Trust and Institutions first allows for punishment by one party of the other in the trust game – second-party pun- Political scientists have long been concerned ishment. Fehr and Rockenbach (2003) allow with the relationship between interpersonal the first movers in the game to specify an trust and political institutions. This tradition amount that should be returned when making extends back to Almond and Verba (1963), their trust decision. In one treatment, there who claim that there is a strong correla- is no possibility of punishment; in the other tion between citizen trust and the existence treatment, the first mover has the ability to of democratic institutions. Subsequent work punish the second mover. The first mover examined the nature and directional causality specifies a contract that states the amount that of this relationship. Rothstein (2000) argues he or she wants to be returned and whether that there are two approaches to under- he or she will punish noncompliance. They standing how trust among citizens is pro- find that the highest level of trustworthiness is duced. The first, largely advocated by Putnam observed when sanctions are possible but not (1993), takes a bottom-up approach. Trust implemented by the first mover, and the low- emerges when citizens participate in many est level of trustworthiness is observed when different environments and, in doing so, sanctions are implemented. A similar result experience trust outside their own narrow is seen in Houser et al. (2008), where the groups. This, in turn, provides for democratic threat of punishment backfires by reducing stability in that citizens develop tolerance reciprocity when the first mover asks for too for one another, which ultimately extends to much. confidence in governmental institutions. The A second punishment mechanism intro- second approach holds that institutions miti- duces an additional player whose role is to gate the risk inherent in a trust relationship, punish the behavior of the players in the trust thereby encouraging individuals to trust one dyad – third-party punishment. Bohnet, Frey, another. As Rothstein (2000) puts it, “In a and Huck (2001) focus on whether an ex post civilized society, institutions of law and order mechanism that is designed to reinforce trust- have one particularly important task: to detect worthiness is effective in doing so. In this and punish people who are ‘traitors’, that is, study, the punishment is implemented auto- those who break contracts, steal, murder and matically, using a computerized robot. The do other such noncooperative acts and there- experiment uses a simplified trust game with fore should not be trusted” (490–91). In this binary choices and adds treatments whereby view, institutions serve to monitor relation- failure to reciprocate can trigger a fine, which ships, screen out the untrustworthy, and pun- is imperfectly implemented with a known ish noncooperative behavior. For institutions probability. All actors know the associated to effectively accomplish this objective, they thresholds that trigger enforcement and the must be perceived as legitimate. costs of any associated penalties. Thus, the If citizen trust precedes (or is independent institution is transparent and is invoked auto- of) institutions, then individual trust ought to matically, but imperfectly. They find that be insensitive to the institutions within which trust thrives when the institution is weak they interact. If the causal relationship is such and the probability of enforcement is low. 252 Rick K. Wilson and Catherine C. Eckel

However, higher levels of punishment crowd Finally, there has been interest in whe- out trust. When punishment is punitively ther mechanisms that facilitate information large, trust is again observed, but the punish- exchange, especially cheap talk, enhance ment is so large that it is arguably no longer trust. Does a trustee’s unenforceable promise trustworthiness that motivates the trustee. that trust will be reciprocated have any effect Trust only thrives when third-party enforce- on trust? In principle, it should not. Char- ment is absent or low. Using different experi- ness and Dufwenberg (2006) allow trustees mental designs, Kollock (1994) and Van Swol an opportunity to send a nonbinding message (2003) reach similar conclusions. to the truster. The effect of communication Rather than using a robot, Charness and leads to increased amounts of trust, which is Cobo-Reyes (2008) introduce an explicitly then reciprocated: the promise appears to act selected third party who is empowered to as a formal commitment by the trustee. As punish trustees and/or reward trusters. They Charness and Dufwenberg explain, trustees find that more is sent and more is returned who make promises appear “guilt averse,” and when there is the possibility of punishment. so carry out their action. Ben-Ner, Putter- Interestingly, even though the third party man, and Ren (2007) allow both trusters and gains nothing from the exchange between the trustees to send messages. In a control con- truster and trustee, the third party is will- dition, no one was allowed to communicate; ing to bear the cost of punishing. In contrast, in one treatment, subjects were restricted to Banuri et al. (2009) adapt the trust game to numerical proposals about actions; and in study bribery, and show that third-party sanc- the second treatment, subjects could send tions reduce the incidence of and rewards to text plus numerical proposals. Communi- trust (as bribery), essentially by reducing the cation enhanced trust and reciprocity, with trustworthiness of the bribed official. They the richer form of communication yielding argue that high levels of trust and trustwor- the highest returns for subjects. Sommer- thiness are necessary for bribery to be effec- feld, Krambeck, and Milinski (2008) focus tive because parties to the transaction have no on gossip, which is regarded as the intersec- legal recourse if the transaction is not com- tion of communication and reputation for- pleted. mation. In their experiment, play is repeated A second type of institutional mechanism within a group and gossip is allowed. The uses group decision making for the trust and effect is to increase both trust and reciproca- reciprocity decisions themselves. In Kugler tion, largely through reputational enhance- et al. (2007), subjects make trust and reci- ment (see also Keser [2003] and Schotter and procity decisions under group discussion and Sopher [2006]). Communication matters for consensus to decide how much to trust or enhancing trust and trustworthiness, a find- how much to reciprocate. Compared with ing that is widespread in many bargaining control groups using the standard dyadic games. trust game, they find that trust is reduced An implication of these findings is that when groups decide, but reciprocation is not trust is malleable. Perversely, it appears that affected. Song (2008) divides subjects into institutional mechanisms involving monitor- three-person groups, has individuals make a ing and sanctions can crowd out trust and decision for the group, and then randomly trustworthiness. Although the wisdom of selects one individual’s choice to implement groups might be expected to enhance trust, the decision for the group. She also finds that it does not. Only communication reliably levels of trust are reduced and that reciprocity enhances trust and reciprocity, and the more is reduced. Both studies point to trust or communication, the better. None of these reciprocity declining when groups make a studies pursues why sanctions are sometimes decision, a result that echoes similar group ineffective. One yet unexamined possibility is polarization results in other game settings that the legitimacy of the institutions has not (Cason and Mui 1997). Why collective choice been taken into account. For example, select- mechanisms depress trust is left unexplored. ing a third-party punisher, as in Charness and Trust and Social Exchange 253

Cobo-Reyes (2008), may endow the sanctions a repeated-play game with large numbers of with greater legitimacy, thereby making them individuals. A combination of Internet exper- more effective. Further work is warranted to iments and a longer time period could provide determine the basis for effective institutions. a vehicle for such studies. Fourth, we still do not know the extent to which the trust game measures political 7. Unanswered Questions behaviors that are important in natural settings. Although we see that the trust game is Despite extensive experimental research on correlated with some individual characteris- trust, there are a number of significant ques- tics and predicts small trusting acts such as tions left unanswered. Using these questions lending money to friends, as well as larger as a guide, we suggest areas for future research actions such as repayment of micro loans, it is in this section. unclear how trust and reciprocity are related First, there is still no clear answer to to issues of concern to political scientists. the causal relationship between interper- For example, how well does the trust game sonal trust and effective political institu- predict leadership behavior? How well does tions. Because experimental methods have an it predict political efficacy? Is it correlated advantage in testing for causality, we believe with corrupt behavior of elected or appointed it would be fruitful to tackle this question officials? in the lab. As noted previously, some work Fifth, how much insight will biological and has examined the relationship between trust neural studies provide into the complex social and specific institutions, but a more general relationship underlying trusting and recipro- understanding of the characteristics of insti- cal behaviors? Experimental studies show that tutional mechanisms that promote trust, or trust is sensitive to the context in which the when trust will breed successful institutions, decision is made. Can these biological studies is paramount. provide explanatory power beyond what we Second, we do not know enough about can learn by observing behavior? Their likely why monitoring and sanctioning institutions contribution will be through a better under- crowd out trust. Institutions with strong standing of the neural mechanisms behind rewards and punishment may indeed act as the general perceptual and behavioral biases substitutes for norms of trust and reciprocity. common to humans. If trust is a fragile substitute for institutional monitoring and sanctioning, then knowing which institutions support and undermine References trust is important. It appears that punishment itself is seldom productive, but only the Almond, Gabriel A., and Sidney Verba. 1963. The unused possibility of punishment enhances Civic Culture. Princeton, NJ: Princeton Uni- responsible behavior. versity Press. Third, little is known about the rela- Anderhub, Vital, Dirk Engelmann, and Werner tionship between trust and social networks. Guth. 2002. “An Experimental Study of the We do not understand whether dyadic trust Repeated Trust Game with Incomplete Infor- relationships build communities. Do trusting mation.” Journal of Economic Behavior and Orga- 48 197 216 individuals initiate widespread networks of nization : – . reciprocated trust? Or do trusting individuals Anderson, Lisa R., Jennifer Mellor, and Jeffrey Milyo. 2010. “Did the Devil Make Them Do turn inward, limiting the number of partners, It? The Effects of Religion in Public Goods and thereby segregating communities of trusters? Trust Games.” Kyklos 63: 163–75. Our own evidence suggests that subjects use Ashraf, Nava, Iris Bohnet, and Nikita Piankov. skin shade as a basis for discriminating trust. 2006. “Decomposing Trust and Trustworthi- If this persists over time, then it is easy to see ness.” Experimental Economics 9: 193–208. how segregation can result. To understand Bahry, Donna, and Rick K. Wilson. 2004. “Trust this dynamic, it is important to study trust as in Transitional Societies: Experimental Results 254 Rick K. Wilson and Catherine C. Eckel

from Russia.” Paper presented at the annual Cason, Timothy, and Vai-Lam Mui. 1997.“ALab- meeting of the American Political Science oratory Study of Group Polarization in the Association, Chicago. Team Dictator Game.” Economic Journal 107: Banuri, Sheheryar, Rachel Croson, Catherine C. 1465–83. Eckel, and Reuben Kline. 2009. “Towards an Castillo, Marco, and Ragan Petrie. 2010.“Dis- Improved Methodology in Analyzing Corrup- crimination in the Lab: Does Information tion.” CBEES Working Paper, University of Trump Appearance?” Games and Economic Texas, Dallas. Behavior 68: 50–9. Barr, Abigail. 2003. “Trust and Expected Cesarini, David, Christopher T. Dawes, James H. Trustworthiness: Experimental Evidence from Fowler, Magnus Johannesson, Paul Lichten- Zimbabwean Villages.” Economic Journal 113: stein, and Bjorn Wallace. 2008. “Heritability 614–30. of Cooperative Behavior in the Trust Game.” Bellemare, Charles, and Sabine Kroger.¨ 2007.“On Proceedings of the National Academy of Sciences of Representative Social Capital.” European Eco- the United States of America 105: 3721–26. nomic Review 51: 183–202. Charness, Gary R., and N. Jimenez Cobo-Reyes. Ben-Ner, Avner, Louis G. Putterman, and Ting 2008. “An Investment Game with Third Party Ren. 2007. “Lavish Returns on Cheap Talk: Intervention.” Journal of Economic Behavior and Non-Binding Communication in a Trust Organization 68: 18–28. Experiment.” SSRN Working paper. Retrieved Charness, Gary, and Martin Dufwenberg. 2006. from http://ssrn.com/abstract=1013582 on “Promises and Partnership.” Econometrica 74: February 22, 2011. 1579–601. Berg, Joyce E., John W. Dickhaut, and Kevin Citrin, Jack. 1974. “Comment: The Political Rele- McCabe. 1995. “Trust, Reciprocity, and Social vance of Trust in Government.” American Polit- History.” Games and Economic Behavior 10: 122– ical Science Review 68: 973–88. 42. Cox, James C. 2004. “How to Identify Trust and Bianco, William T. 1994. Trust: Representatives and Reciprocity.” Games and Economic Behavior 46: Constituents. Ann Arbor: University of Michi- 260–81. gan Press. Cronk, Lee. 2007. “The Inﬂuence of Cultural Bohnet, Iris, Bruno S. Frey, and Steffen Huck. Framing on Play in the Trust Game: A Maa- 2001. “More Order with Less Law: On Con- sai Example.” Evolution and Human Behavior 28: tract Enforcement, Trust, and Crowding.” 352–58. American Political Science Review 95: 131–44. Croson, Rachel, and Uri Gneezy. 2009. “Gender Bohnet, Iris, Fiona Greig, Benedikt Herrmann, Differences in Preferences.” Journal of Economic and Richard Zeckhauser. 2008. “Betrayal Aver- Literature 47: 448–74. sion: Evidence from Brazil, China, Oman, Danielson, Anders J., and Hakan J. Holm. 2007. Switzerland, Turkey, and the United States.” “Do You Trust Your Brethren? Eliciting Trust American Economic Review 98: 294–310. Attitudes and Trust Behavior in a Tanzanian Bohnet, Iris, and Richard Zeckhauser. 2004. Congregation.” Journal of Economic Behavior “Trust, Risk and Betrayal.” Journal of Economic and Organization 62: 255–71. Behavior and Organization 55: 467–84. Delgado, Mauricio R., Robert H. Frank, and Eliz- Brehm, John, and Wendy Rahn. 1997. “Individ- abeth A. Phelps. 2005. “Perceptions of Moral ual Level Evidence for the Causes and Conse- Character Modulate the Neural Systems of quences of Social Capital.” American Journal of Reward During the Trust Game.” Nature Neu- Political Science 41: 999–1023. roscience 8: 1611–18. Buchan, Nancy R., Rachel T. A. Croson, and Eckel, Catherine C., and Ragan Petrie. in press. Robyn M. Dawes. 2002. “Swift Neighbors and “Face Value.” American Economic Review. Persistent Strangers: A Cross-Cultural Inves- Eckel, Catherine C., and Rick K. Wilson. 2004. tigation of Trust and Reciprocity in Social “Is Trust a Risky Decision?” Journal of Economic Exchange.” American Journal of Sociology 108: Behavior and Organization 55: 447–65. 168–206. Eckel, Catherine C., and Rick K. Wilson. 2006. Carpenter, Jeffrey P., Amrita G. Daniere, and “Internet Cautions: Experimental Games with Lois M. Takahashi. 2004. “Cooperation, Trust, Internet Partners.” Experimental Economics 9: and Social Capital in Southeast Asian Urban 53–66. Slums.” Journal of Economic Behavior and Orga- Eckel, Catherine C., and Rick K. Wilson. 2008. nization 55: 533–51. “Initiating Trust: The Conditional Effects of Trust and Social Exchange 255

Sex and Skin Shade among Strangers.” Work- July. Retrieved from http://ssrn.com/abstract= ing paper, Rice University. 807364 (November 7, 2010). Eckel, Catherine C., and Rick K. Wilson. 2010. Haile, Daniel, Abdolkarim Sadrieh, and Harrie A. “Shopping for Trust.” Paper presented at the Verbon. 2006. “Cross-Racial Envy and Under- annual meeting of the Midwest Political Sci- investment in South Africa.” Working paper, ence Association, Chicago. Tilburg University. Engle-Warnick, Jim, and Robert L. Slonim. 2004. Hardin, Russell. 2002. Trust and Trustworthiness. “The Evolution of Strategies in a Repeated New York: Russell Sage Foundation. Trust Game.” Journal of Economic Behavior and Hetherington, Marc J. 1998. “The Political Rele- Organization 55: 553–73. vance of Political Trust.” American Political Sci- Falk, Armin, and Christian Zehnder. 2007.“Dis- ence Review 92: 791–808. crimination and In-Group Favoritism in a Hibbing, John R., and Elizabeth Theiss-Morse. Citywide Trust Experiment.” Zurich IEER 2002. Stealth Democracy: American’s Beliefs about Working Paper No. 318. April. Retrieved from How Government Should Work. New York: http://ssrn.com/abstract=980875 (November Cambridge University Press. 7, 2010). Houser, Daniel, Erte Xiao, Kevin McCabe, and Fehr, Ernst, Michael Kosfeld, and Urs Fisch- Vernon Smith. 2008. “When Punishment Fails: bacher. 2005. “Neuroeconomic Foundations of Research on Sanctions, Intention and Non- Trust and Social Preferences.” IEW Work- Cooperation.” Games and Economic Behavior 62: ing Paper No. 221. June. Retrieved from 509–32. http://ssrn.com/abstract=747884 (November Inglehart, Ronald. 1997. Modernization and Post- 7, 2010). modernization: Cultural, Economic, and Political Fehr, Ernst, and Bettina Rockenbach. 2003. Change in 43 Societies. Princeton, NJ: Princeton “Detrimental Effects of Sanctions on Human University Press. Altruism.” Nature 422: 137–40. Johansson-Stenman, Olof, Minhaj Mahmud, and Fenno, Richard F., Jr. 1978. Home Style: House Peter Martinsson. 2005. “Does Stake Size Mat- Members in Their Districts. Boston: Little, ter in Trust Games?” Economics Letters 88: 365– Brown. 69. Fershtman, Chaim, and Uri Gneezy. 2001.“Dis- Johansson-Stenman, Olof, Minhaj Mahmud, and crimination in a Segmented Society: An Peter Martinsson. 2006. “Trust, Trust Games Experimental Approach.” Quarterly Journal of and Stated Trust: Evidence from Rural Economics 116: 351–77. Bangladesh.” Working paper, Goteborg Uni- Gachter, Simon, Benedikt Herrmann, and versity. Christian Thoni. 2004. “Trust, Voluntary Johansson-Stenman, Olof, Minhaj Mahmud, and Cooperation, and Socio-Economic Back- Peter Martinsson. 2009. “Trust and Religions: ground: Survey and Experimental Evidence.” Experimental Evidence from Bangladesh.” Journal of Economic Behavior and Organization Economica 76: 462–85. 55: 505–31. Johnson, Noel D., and Alexandra Mislin. 2008. Glaeser, Edward L., David I. Laibson, Jose A. “Cultures of Kindness: A Meta-Analysis Scheinkman, and Christine L. Soutter. 2000. of Trust Game Experiments.” December “Measuring Trust.” Quarterly Journal of Eco- 12. Retrieved from http://ssrn.com/abstract= nomics 115: 811–46. 1315325 (November 7, 2010). Goldstein, Kenneth, and Travis N. Ridout. 2004. Karlan, Dean. 2005. “Using Experimental Eco- “Measuring the Effects of Televised Politi- nomics to Measure Social Capital and Predict cal Advertising in the United States.” Annual Financial Decisions.” American Economic Review Review of Political Science 7: 205–26. 95: 1688–99. Green, Donald P., and Rachel L. Seher. 2003. Keser, Claudia. 2003. “Experimental Games for “What Role Does Prejudice Play in Ethnic the Design of Reputation Management Sys- Conﬂict?” Annual Review of Political Science 6: tems.” IBM Systems Journal 42: 498–506. 509–31. King-Casas, Brooks, Damon Tomlin, Cedric Greig, Fiona E., and Iris Bohnet. 2005.“Is Anen, Colin F. Camerer, Steven R. Quartz, There Reciprocity in a Reciprocal-Exchange and P. Read Montague. 2005. “Getting to Economy? Evidence from a Slum in Nairobi, Know You: Reputation and Trust in a Two- Kenya.” Kennedy School of Government Person Economic Exchange.” Science 308: 78– (KSG) Working Paper No. RWP05-044. 83. 256 Rick K. Wilson and Catherine C. Eckel

Knack, Stephen, and Philip Keefer. 1997.“Does Ethnic Trust Differences.” Working paper, Social Capital Have an Economic Payoff? A University of Zurich. Cross-Country Investigation.” Quarterly Jour- Nannestad, Peter. 2008. “What Have We Learned nal of Economics 112: 1251–88. about Generalized Trust, If Anything?” Annual Knight, Jack. 2001. “Social Norms and the Rule Review of Political Science 11: 413–36. of Law: Fostering Trust in a Socially Diverse Putnam, Robert D. 1993. Making Democracy Work. Society.” In Trust in Society, ed. K. S. Cook. Princeton, NJ: Princeton University Press. New York: Russell Sage Foundation, 354–73. Putnam, Robert D. 2000. Bowling Alone: The Col- Kollock, Peter. 1994. “The Emergence of Ex- lapse and Revival of American Community. New change Structures – An Experimental-Study of York: Simon & Schuster. Uncertainty, Commitment, and Trust.” Amer- Rilling, James K., Alan G. Sanfey, Jessica A. ican Journal of Sociology 100: 313–45. Aronson, Leigh E. Nystrom, and Jonathan D. Kosfeld, Michael, Markus Heinrichs, Paul Zak, Cohen. 2004. “Opposing Bold Responses to Urs Fischbacher, and Ernst Fehr. 2005.“Oxy- Reciprocated and Unreciprocated Altruism in tocin Increases Trust in Humans.” Nature 435: Putative Reward Pathways.” NeuroReport 15: 673–76. 2539–43. Kugler, Tamar, Gary Bornstein, Martin G. Rothstein, Bo. 2000. “Trust, Social Dilemmas Kocher, and Matthias Sutter. 2007. “Trust and Collective Memories.” Journal of Theoreti- between Individuals and Groups: Groups Are cal Politics 12: 477–501. Less Trusting Than Individuals but Just as Schechter, Laura. 2007. “Traditional Trust Mea- Trustworthy.” Journal of Economic Psychology 28: surement and the Risk Confound: An Exper- 646–57. iment in Rural Paraguay.” Journal of Economic Lau, Richard R., and Ivy Brown Rovner. 2009. Behavior and Organization 62: 272–92. “Negative Campaigning.” Annual Review of Schotter, Andrew, and Barry Sopher. 2006. “Trust Political Science 12: 285–306. and Trustworthiness in Games: An Experimen- Levi, Margaret, and Laura Stoker. 2000. “Political tal Study of Intergenerational Advice.” Experi- Trust and Trustworthiness.” Annual Review of mental Economics 9: 123–45. Political Science 3: 475–507. Simpson, Brent, Tucker McGrimmon, and Kyle Lupia, Arthur, and Mathew D. McCubbins. 1998. Irwin. 2007. “Are Blacks Really Less Trusting The Democratic Dilemma: Can Citizens Learn Than Whites? Revisiting the Race and Trust What They Need to Know? New York: Cam- Question.” Social Forces 86: 525–52. bridge University Press. Sommerfeld, Ralf D., Hans-Juergen Krambeck, McCabe, Kevin, Daniel Houser, Lee Ryan, Ver- and Manfred Milinski. 2008. “Multiple Gos- non Smith, and Theodore Trouard. 2001.“A sip Statements and Their Effect on Reputation Functional Imaging Study of Cooperation in and Trustworthiness.” Proceedings of the Royal Two-Person Reciprocal Exchange.” Proceedings Society B-Biological Sciences 275: 2529–36. of the National Academy of Sciences 98: 11832–35. Song, Fei. 2008. “Trust and Reciprocity Behav- McClain, Paula D., Jessica D. Carew John- ior and Behavioral Forecasts: Individuals versus son, Eugene Walton, and Candis S. Watts. Group-Representatives.” Games and Economic 2009. “Group Membership, Group Identity, Behavior 62: 675–96. and Group Consciousness: Measures of Racial Stolle, Dietlind. 1998. “Bowling Together, Bowl- Identity in American Politics?” Annual Review ing Alone: The Development of Generalized of Political Science 12: 471–85. Trust in Voluntary Associations.” Political Psy- Miller, Arthur H. 1974. “Political Issues and Trust chology 19: 497–525. in Government: 1964–1970.” American Political Sutter, Matthias, and Martin G. Kocher. 2007. Science Review 68: 951–72. “Trust and Trustworthiness across Different Mishler, William, and Richard Rose. 2005. “What Age Groups.” Games and Economic Behavior 59: Are the Political Consequences of Trust? A 364–82. Test of Cultural and Institutional Theories in Sztompka, Piotr. 1999. Trust: A Sociological Theory. Russia.” Comparative Political Studies 38: 1050– Cambridge: Cambridge University Press. 78. Tomlin, Damon, M. Amin Kayali, Brooks King- Naef, Michael, Ernst Fehr, Urs Fischbacher, Casas, Cedric Anen, Colin F. Camerer, Steven Jurgen¨ Schupp, and Gert G. Wagner. 2009. Quartz, and P. Read Montague. 2006.“Agent- “Decomposing Trust: Explaining National and Speciﬁc Responses in the Cingulate Cortex Trust and Social Exchange 257

During Economic Exchanges.” Science 312: Yamagishi, Toshio, Karen Cook, and Motoki 1047–50. Watabe. 1998. “Uncertainty, Trust, and Com- Uslaner, Eric M. 2002. The Moral Foundations mitment Formation in the United States and of Trust. Cambridge: Cambridge University Japan.” American Journal of Sociology 104: 165– Press. 94. Van Swol, Lyn M. 2003. “The Effects of Regula- Zak, Paul, and Stephen Knack. 2001. “Trust tion on Trust.” Basic and Applied Social Psychology and Growth.” Economic Journal 111: 295– 25: 221–33. 321. Wilson, Rick K., and Catherine C. Eckel. 2006. Zak, Paul J., Robert Kurzban, and William “Judging a Book by Its Cover: Beauty and T. Matzner. 2005. “Oxytocin Is Associated Expectations in the Trust Game.” Political with Human Trustworthiness.” Hormones and Research Quarterly 59: 189–202. Behavior 48: 522–27. CHAPTER 18

An Experimental Approach to Citizen Deliberation

Christopher F. Karpowitz and Tali Mendelberg

Deliberation has become, in the words of Our strategy for this chapter is to highlight one scholar, “the most active area of polit- the strengths of experiments that have already ical theory in its entirety” (Dryzek 2007, been completed and to point to some aspects 237). Our exploration of the relationship of the research that need further improve- between experiments and deliberation thus ment and development. We aim to discuss begins with normative theory as its starting what experiments can do that other forms of point. Experiments can yield unique insights empirical research cannot and what experi- into the conditions under which the expecta- ments need to do in light of the normative tions of deliberative theorists are likely to be theory. approximated, as well as the conditions under Proceeding from normative theory is not which theorists’ expectations fall short. Done without its difficulties, as deliberative theo- well, experiments demand an increased level rists themselves admit (see, e.g., Chambers of conceptual precision from researchers of 2003; Thompson 2008). One difficulty is all kinds who are interested in deliberative that theories of deliberation offer a wide- outcomes. However, perhaps most impor- ranging, sometimes vague, and not always tant, experiments can shed greater scholarly consistent set of starting points for experi- light on the complex and sometimes conflict- mental work – as Diana Mutz (2008) ruefully ing mechanisms that may drive the outcomes observes, “It may be fair to say that there are of various deliberative processes. In other as many definitions of deliberation as there words, experiments allow researchers to bet- are theorists” (525). We recognize, too, the ter understand the extent to which, the ways inevitable slippage between theory and praxis in which, and under what circumstances it is that will lead almost every empirical test to be, actually deliberation that drives the outcomes in some sense, “incomplete” (Fishkin 1995). deliberative theorists expect. Finally, we agree that empirical researchers should avoid distorting the deeper logic of We thank Lee Shaker for invaluable research assistance. deliberative theory in the search for testable

258 An Experimental Approach to Citizen Deliberation 259 hypotheses (Thompson 2008). Experiments 2003). Although theories of deliberation do cannot “prove” or “disprove” theories of not agree about each of deliberation’s con- deliberation writ large. The critical ques- stituent aspects or about all of its expected tion for experimental researchers, then, is not outcomes (Macedo 1999), it is possible to dis- “Does deliberation work?” but rather under till a working set of empirical claims about what conditions does deliberation approach deliberation’s effects (see Mendelberg 2002; theorists’ goals or expectations? Mutz 2008). The literature on deliberation is too large Hibbing and Theiss-Morse (2002)sum- to allow us to provide full coverage here. A marize three broad categories of effects: variety of additional research traditions from deliberation should lead to “better citizens,” social psychology and from sociology can use- “better decisions,” and a “better (i.e., more fully inform our attempts to explore delib- legitimate) system.” Benefits for individual erative dynamics (see Mendelberg [2002] citizens may include: for an overview), but we focus on politi- 1 r cal discussion. Because we aim to explore increased tolerance or generosity and a deliberation as practiced by ordinary citizens, more empathetic view of others (War- we do not consider the literature on elites ren 1992; Gutmann and Thompson 1996, (Steineretal.2005). We set aside, too, a 2004); r valuable research tradition focused on dyadic a decrease in the set of pathologies of exchanges within social networks (Huckfeldt public opinion documented extensively and Sprague 1995;Mutz2006) to focus on since Converse’s (1964) seminal work, discussions among groups, not dyads. Our leading to more political knowledge, an focus on discussion of political issues and enhanced ability to formulate opinions, topics means that we do not cover the vast greater stability opinions, and more coher- and influential research on deliberation in ence among related opinions (Fishkin juries (see, e.g., Hastie, Penrod, and Pen- 1995); r nington 1983; Schkade, Sunstein, and Kah- a better understanding of one’s own inter- neman 2000; Devine et al. 2001), although ests; an increased ability to justify pref- we emphasize the value and importance of erences with well-considered arguments that research. Finally, we cannot do justice (Warren 1992; Chambers 1996, 2003); r to experiments derived from formal theories a better awareness of opponents’ argu- (see, e.g., Hafer and Landa 2007 or Meirowitz ments and an increased tendency to recog- 2007). nize the moral merit of opponents’ claims (Habermas 1989, 1996; Chambers 1996; Gutmann and Thompson 1996, 2004); r 1. The Substantive Issues: a sense of empowerment, including among Independent and Dependent those who have the least (Fishkin 1995; Variables Bohman 1997); r a greater sense of public spiritedness (War- One of the challenges of empirical research ren 1992) and an increased willingness to on deliberation is the multiplicity of potential recognize community values and to com- definitions (Macedo 1999). Still, many defi- promise in the interest of the common nitions of deliberation share a commitment good (Mansbridge 1983; Chambers 1996; to a reason-centered, “egalitarian, recipro- but see Sanders 1997 and Young 2000); cal, reasonable, and open-minded exchange and r of language” (Mendelberg 2002, 153; see a tendency to participate more in public also Gutmann and Thompson 1996; Burkhal- affairs (Barber 1984; Gastil, Deess, and ter, Gastil, and Kelshaw 2002; Chambers Weiser 2002;Gastiletal.2008).

1 See Gaertner et al. (1999) for a particularly informa- Benefits for the quality of decisions flow tive study. from many of these individual benefits and 260 Christopher F. Karpowitz and Tali Mendelberg include the idea that collective decisions part in deliberation to feel that their voices or outcomes of deliberating groups will be were better heard, that the deliberating group grounded in increased knowledge, a more functioned well as a collectivity, or that the complete set of arguments, a fuller under- process was more collaborative than other standing of the reasons for disagreement, and forms of interaction. In addition, researchers a more generous aggregate attitude toward can investigate the effects of this sort of delib- all groups in society, especially those who erative interaction for subsequent levels and have the least (Chambers 1996; Gutmann and forms of participation (see, e.g., Karpowitz Thompson 2004). 2006; Gastil et al. 2008). As to the benefits for democratic systems, The variety of approaches to delibera- they center on the rise in support for the tive theory provides a rich set of procedural system that can follow from deliberation. and substantive conditions, characteristics, Increased legitimacy for the system is a com- and mechanisms that can be explored experi- plex concept (Thompson 2008); among other mentally in a systematic way. Theorists differ, things, it can include a heightened level of for example, on the desirability of consensus trust in democratic processes and a greater in deliberative procedures. However, because sense of confidence that the process has been the requirement of producing consensus is fairly carried out (see also Manin 1987). This something that can be experimentally manip- sense becomes particularly important when ulated, empirical researchers can help spec- the ultimate decision does not correspond ify the relationship between the potential well to an individual’s predeliberation prefer- characteristics of deliberation (the indepen- ences or when there is a deep or long-standing dent variables) and the positive outcomes (in conflict at issue (Mansbridge 1983; Benhabib other words, the dependent variables) theo- 1996; Chambers 1996). rists hope to see. These laudable outcomes for citizens, Of course, key independent variables rele- decisions, and systems are rooted in a fourth vant to deliberative outcomes may not only claim: that deliberation is a better decision- emerge from theory, but they can also be making process, one that is more public- found through careful attention to the real- spirited, more reasonable, more satisfying, world context of ongoing deliberative reform and ultimately more just than adversarial and effort, where the variety of practices that aggregative forms of decision making (Mans- might be subsumed under the broad heading bridge 1983; Chambers 1996; Gutmann and of deliberation is extraordinary. Mansbridge Thompson 2004). The process is a cru- (1983), Mansbridge et al. (2006), Cramer cial mediating variable in deliberation. In Walsh (2006, 2007), Polletta (2008), and other words, normative theory leads empir- Gastil (2008) are a few examples of scholars ical investigators to ask what aspects of dis- who contribute to our understanding of delib- course and linguistic exchange leads to the eration’s effects by insightful observation of individual-level outcomes we describe, and real-world practices. what in turn causes those aspects of interaction. Examples of mediating variables include the content and style of interaction, such 2. The Role of Experiments as whether deliberators use collective vocab- ulary such as “us” and “we” (Mendelberg Although sometimes styled as “experiments and Karpowitz 2007), the number of argu- in deliberation,” much of the research to date ments they make (Steiner et al. 2005), and has been purely observational – most often, the extent to which deliberators engage in a these are case studies of specific delibera- collaborative construction of meaning rather tive events (e.g., Mansbridge 1983; Fishkin than speaking past each other (Rosenberg 1995; Eliasoph 1998; Fung 2003; Gastil and 2007). Focusing on such mediating variables Levine 2005; Cramer Walsh 2006, 2007;Kar- allows scholars to investigate which aspects powitz 2006; Warren and Pearse 2008). The of the discourse cause those who have taken methods for evaluating and understanding An Experimental Approach to Citizen Deliberation 261 deliberation are also diverse, including par- trol because it need not take place within an ticipant observation (Eliasoph 1998), survey official or even a public context. Ordinarily, research ( Jacobs, Cook, and Delli Carpini investigators are quite limited by the inti- 2009), and content analysis of discussion (see mate link between the behavior we observe Gamson 1992; Conover, Searing, and Crewe and fixed features of the political system. In 2002; Hibbing and Theiss-Morse 2002; contrast, citizens can deliberate with a set Schildkraut 2005; Cramer Walsh 2006), just of strangers with whom they need have no to name a few. Although our emphasis here prior connection – in contexts removed from is on experiments, other research designs are organizational structures, official purviews, valuable in an iterative exchange with experi- or public spaces – and can reach decisions ments and have a value of their own separate directed at no official or public target. In from experiments. that sense, they present the experimenter a Experimentation is, as Campbell and Stan- wide degree of control. Ultimately, deliber- ley put it, “the art of achieving interpretable ation often does exist in some dialogue with comparisons” (quoted in Kinder and Pal- the political system. However, unlike other frey 1993, 7). We follow the standard view forms of participation, it also exists outside that effective experimentation involves a high public and political spaces, and it can there- level of experimenter control over settings, fore be separated and isolated for the study. treatments, and observations so as to rule Experiments also allow us to measure the out potential threats to valid inference. Like outcomes with a good deal more precision Kinder and Palfrey, we see random assign- than nonexperimental, quasi-experimental, ment to treatment and control groups as or field experimental studies. All groups “unambiguously desirable features of experi- assigned to deliberate do in fact deliberate, mental work in the social sciences” (7). In the- all groups assigned to deliberate by a par- ory, random assignment may not be strictly ticular rule do in fact use that rule, all indi- necessary to achieve experimental control, viduals assigned to receive information do but it does “provide a means of comparing in fact receive information, and so on. We the yields of different treatments in a manner can measure actual behavior rather than self- that rule[s] out most alternative interpreta- reported behavior. Under the fully controlled tions” (Cook and Campbell 1979, 5). circumstances afforded by the experimental Much deliberation research is quasi- study of deliberation, we need not worry experimental (Campbell and Stanley 1963): that people self-select into any aspect of the it involves some elements of experimen- treatment. Intent-to-treat problems largely tation – a treatment, a subsequent out- disappear. come measure, and comparison across treated This boosts our causal inference consid- and untreated groups – but not random erably. We note that the causal inference is assignment. The lack of random assignment not a consequence of the simulation that con- in quasi-experiments places additional bur- trol provides. Unlike observational studies or dens on the researchers to sort out treat- quasi-experimental studies, experiments can ment effects from other potential causes and do go beyond simple simulation in two of observed differences between groups. respects. First, the control can help verify In other words, “quasi-experiments require that it is deliberation and not other influences making explicit the irrelevant causal forces creating the consequences that we observe. hidden within the ceteris paribus of random Second, the control afforded by experiments assignment” (Cook and Campbell 1979, 6; see allows us to test specific aspects of the deliber- Esterling, Fung, and Lee [2010] for a sophis- ation, such as heterogeneous or homogenous ticated discussion of nonrandom assignment group composition; the presence or absence to conditions in a quasi-experimental setting). of incentives that generate or dampen conflict The hallmark of experiments, then, is of interests; the proportions of women, eth- causal inference through control. Citizen nic, and racial groups; group size; the group’s deliberation lends itself quite well to con- decision rule; the presence or absence of a 262 Christopher F. Karpowitz and Tali Mendelberg group decision; the presence or absence of ion change, which is presumed to be the result facilitators and the use of particular facilita- of the deliberative poll. tion styles (Rosenberg 2007); the availability The first deliberative polls were criticized of information; and/or the presence of experts heavily on a variety of empirical grounds, with (Myers 2009). critics paying special attention to whether the In sum, the goal of experimental research deliberative poll should qualify as an exper- should be to isolate specific causal forces iment (Kohut 1996;Merkle1996). Mitofsky and mediating or moderating mechanisms in (1996), for example, insists that problems order to understand the relationship between with panel attrition in the response rates those mechanisms and deliberative outcomes in the postdeliberation surveys made causal as described by normative theorists. In addi- inferences especially difficult and that the tion, experiments allow us to reduce measure- lack of a control group made it impossi- ment error. This should lead to insights about ble to know whether any change in indi- how to better design institutions and deliber- vidual opinion “is due to the experience of ative settings. being recruited, flown to Austin, treated like a celebrity by being asked their opinions on national television and having participated 3. Some Helpful Examples in the deliberations, or just due to being interviewed twice” (19). Luskin, Fishkin, and To date, the experimental work on deliber- Jowell (2002) admit that their approach fails ation has been haphazard (Ryfe 2005, 64). to qualify as a full experiment by the stan- Our aim here is not to provide an exhaustive dards of Campbell and Stanley (1963) “both account of every experiment, but to show the because it lacks the full measure of control strengths and weaknesses of some of the work characteristic of laboratory experiments and that has been completed. It is rare for a study because it lacks a true, i.e. randomly assigned, to use all elements of a strong experimen- control group” (Luskin et al. 2002, 460). tal design. We find a continuum of research As the number of deliberative polls has designs, with some studies falling closer to the proliferated, researchers have pursued a vari- gold standard of random assignment to con- ety of innovations. For example, subsequent trol and treatments, and others falling closer work has included both pre- and postdeliber- to the category of quasi-experiments. ation interviews of those who were recruited The best known and, arguably, the most to be part of the deliberating panel but who influential investigation of deliberation is chose not to attend and postevent inter- Fishkin’s (1995) deliberative poll. In a delib- views of a separate sample of nondeliberators. erative poll, a probability sample of citizens These additional interviews function as a type is recruited and questioned about their pol- of control group, although random assign- icy views on a political issue. They are sent ment to deliberating or nondeliberating con- a balanced set of briefing materials prior to ditions is not present. Although still not qual- the deliberative event in order to spark some ifying as a full experiment, these additions initial thinking about the issues. The sample function as an untreated nonequivalent control is then brought to a single location for sev- group design with pre-test and post-test and as a eral days of intensive engagement, including post-test-only control group design, as classified small-group discussion (with assignment to by Campbell and Stanley (1963). When such small groups usually done randomly), infor- additions are included, the research design mal discussion among participants, a chance of the deliberative poll does have “some of to question experts on the issue, and an the characteristics of a fairly sophisticated opportunity to hear prominent politicians quasi-experiment” (Merkle 1996, 603), char- debate the issue. At the end of the event (and acteristics that help eliminate some important sometimes again several weeks or months threats to valid inference. afterward), the sample is asked again about Still, the quasi-experimental deliberative their opinions, and researchers explore opin- poll does not exclude all threats to valid An Experimental Approach to Citizen Deliberation 263 inference, especially when the problem of ation. Simon and Sulkin (2002), for exam- self-selection into actual attendance or nonat- ple, insert deliberation into a “divide the dol- tendance at the deliberative event is con- lar” game in which participants were placed sidered (see Barabas 2004, 692). Like that in groups of five and asked to divide $60 of recent deliberative polls, Barabas’ analy- between them. A total of 130 participants sis of the effects of a deliberative forum about took part in one of eleven sessions, with mul- social security is based on comparing control tiple game rounds played in each session. In groups of nonattenders and a separate sam- the game, each member of the group could ple of nonattenders. To further reduce the make a proposal as to how to divide the potential for problematic inferences, Barabas money, after which a proposal was randomly makes use of propensity score analysis. Quasi- selected and voted on by the group. A bare experimental research designs that make majority was sufficient to pass the proposal. explicit the potential threats to inference or Participants were randomly assigned to one that use statistical approaches to estimate of three conditions – no discussion, discus- treatment effects more precisely are valuable sion prior to proposals, and discussion after advances (see also Esterling, Fung, and Lee proposals. In addition, participants were ran- 2010). These do not fully make up for the domly assigned to either a cleavage condi- lack of randomization, but they do advance tion in which players were randomly assigned empirical work on deliberation. to be in either a three-person majority or We note one additional important chal- a two-person minority and proposals were lenge related to deliberative polling: the required to divide money into two sums – complexity of the deliberative treatment. As one for the majority and one for the minor- Luskin, Fishkin, and Jowell (2002) put it, ity – or a noncleavage condition in which the deliberative poll is “one grand treatment” no majority/minority groups were assigned. that includes the anticipation of the event Simon and Sulkin find that the presence of once the sample has been recruited, the expo- discussion led to more equitable outcomes sure to briefing information, small-group dis- for all participants, especially for players who cussion, listening to and asking questions ended up being in the minority. of experts and politicians, informal conver- The experiment employs many of the ben- sations among participants throughout the eficial features we highlight – control over event, and a variety of other aspects of the many aspects of the setting and of mea- experience (not the least of which is partici- surement, and random assignment to con- pants’ knowledge that they are being studied ditions. In addition, the researchers ground and will be featured on television). Perhaps their questions in specific elements of nor- it is the case that deliberation, as a con- mative theories. However, the study artifi- cept, is a “grand treatment” that loses some- cially capped discussion at only 200 seconds of thing when it is reduced to smaller facets, but online communication, which detracts from from a methodological perspective, the com- its ability to speak to the lengthier, deeper plexity of this treatment makes it difficult to exchanges that deliberative theory deals with know what, exactly, is causing the effects we or to the nature of real-world exchanges. observe. Indeed, it denies us the ability to con- Other experimental approaches have also clude that any aspect of deliberation is respon- explored the effects of online deliberation sible for the effects (rather than the briefing (see, e.g., Muhlberger and Weber [2006] materials, expert testimony, or other nonde- and developing work by Esterling, Neblo, 2 liberative aspects of the experience). Experi- and Lazer [2008a, 2008b]). The most well- mentation can and should seek to isolate the known of these so far is the Healthcare independent effects of each of these features. Dialogue project undertaken by Price and At the other end of the spectrum from the “one grand treatment” approach of the delib- 2 See also Esterling et al. (2008a) on estimating treatment effects in the presence of noncompliance with erative pollsters are experiments that involve assigned treatments and nonresponse to outcome a much more spare conception of deliber- measures. 264 Christopher F. Karpowitz and Tali Mendelberg

Capella (2005, 2007). A year-long longitu- more – an increased understanding of the dinal study with a nationally representative rationales behind various policy positions. pool of citizens and a panel of health care Regardless of whether the findings were policy elites, this study explored the effective- positive or negative from the perspective ness of online deliberations about public pol- of deliberative theory, the research design icy. The research involved repeated surveys employed by Price and Capella (2005, 2007) and an experiment in which respondents who highlights many of the virtues of thought- completed the baseline survey were randomly ful, sophisticated experiments. The research assigned to a series of four online discus- includes a large number of participants sions or to a nondeliberating control group. (nearly 2,500); a significant number of delib- In these discussions, participants were strat- erating groups (more than eighty in the first ified as either policy elites, health care issue wave and approximately fifty in the second public members (regular citizens who were wave); and random assignment from a sin- very knowledgeable about health care issues), gle sample (those who completed the baseline or members of the general public. Half of survey) to “deliberation plus information,” the groups were homogenous across strata “information only,” and “no deliberation, no for the first two conversations; the other half information” conditions, with respondents included discussants of all three types. In the in all groups completing a series of surveys second pair of conversations, half of the par- over the course of a calendar year. Price and ticipants remained in the same kind of group Capella also leverage experimental control as in the first wave, whereas the other fifty to answer questions that the “grand treat- percent were switched from homogenous to ment” approach of deliberative polling can- heterogeneous groups, or vice versa. Group not. For example, where deliberative polling tasks were, first, to identify key problems is unable to separate the independent effects related to health care and, second, to identify of information, discussion among ordinary potential policy solutions (although they did citizens, and exposure to elites, Price and not have to agree on a single solution). To Capella are able to show that information ensure compatibility across groups, trained has differing effects from discussion and that moderators followed a script to introduce exposure to elites cannot explain all aspects topics and prompt discussion and debate. of citizens’ opinion change. Given a large Price and Capella (2005, 2007) find that number of groups and the elements of the participation in online discussion led to research design that have to do with differing higher levels of opinion holding among group-level conditions for deliberation, the deliberators and a substantial shift in policy Price and Capella design has the potential for preferences, relative to those who did not even more insight into the ways group-level deliberate. This shift was not merely the factors influence deliberators and deliberative result of being exposed to policy elites, outcomes, although these have not been the because the movement was greatest among primary focus of their analyses to date. Still, those who did not converse with elites. their design may fail to satisfy some concep- In addition, participants – and especially tions of deliberation because groups simply nonelites – rated their experience with the had to identify potential solutions, not make deliberation as quite satisfying. Because a a single, binding choice. random subset of the nondeliberating con- Experiments relevant to deliberation have trol group was assigned to read online brief- also been conducted with face-to-face treating papers that deliberating groups used to ments, although we find considerable vari- prepare for the discussions, the experiments ation in the quality of the research design also allowed the researchers to distinguish the and the direct attention to deliberative the- effects of information from the effects of dis- ory. Morrell (1999), for example, contrasts cussion. Although exposure to briefing mate- familiar liberal democratic decision-making rials alone increased knowledge of relevant procedures, which include debate using facts, discussion and debate added something Robert’s Rules of Order, with what he calls An Experimental Approach to Citizen Deliberation 265

“generative” procedures for democratic talk, to lower levels of satisfaction. Finally, we note which include such deliberatively desirable that Morrell’s experiments were based on a elements as hearing the perspectives of all very small number of participants and an even group members, active listening and repeat- smaller number of deliberating groups. All ing the ideas of fellow group members, this makes comparison to other, conflicting and considerable small-group discussion. His studies difficult. research design includes random assignment Druckman’s (2004) study of the role of to the liberal democratic condition, the gen- deliberation in combating framing effects erative condition, or a no-discussion condi- is a good example of the way experiments tion. Participants answered a short survey can speak to aspects of deliberative the- about their political attitudes at the begin- ory. The primary purpose of Druckman’s ning and the end of the experimental process. research is to explore the conditions under Morrell repeats the study with multiple issues which individuals might be less vulnerable and with differing lengths of discursive inter- to well-recognized framing effects. The rele- action. In this research design, participants vance to deliberation lies in investigating how made a collective decision about an issue, an deliberation can mitigate the irrationality of element that is not present in Fishkin’s (1995) ordinary citizens and improve their civic deliberative polls but that is critical to some capacities. The study advances the literature theories of deliberation. on deliberation by assessing the impact of dif- In contrast to the comparatively positive ferent deliberative contexts. outcomes of the experiments in online delib- Druckman (2004) presented participants eration we highlight, Morrell (1999) finds with one of eight randomly assigned con- that the deliberatively superior generative ditions. These conditions varied the nature procedures do not lead to greater group- of the frame (positive or negative) and the level satisfaction or acceptance of group deci- context in which the participant received the sions. If anything, traditional parliamentary frame. Contexts included a control condi- procedures are preferred in some cases. In tion, in which participants received only a addition, in several of the iterations of the single, randomly chosen frame; a coun- experiment, Morrell finds strong mediating terframing condition, in which participants effects of the group outcome, contrary to received both a positive and a negative frame; deliberative expectations. Morrell’s findings and two group conditions, in which partici- thus call attention to the fact that the condi- pants had an opportunity to discuss the fram- tions of group discussion, including the rules ing problems with three other participants. In for group interaction, matter a great deal and the homogenous group condition, all mem- that more deliberative processes may not lead bers of the group received the same frame, to the predicted normative outcomes. and in the heterogeneous condition, half of Although we see important strengths in the group received a positive frame and half Morrell’s (1999) approach, we note that the received the negative frame. Participants in reported results do not speak directly to the the group condition were instructed to dis- value of the presence or absence of delib- cuss the framing problem for five minutes. eration. The dependent variables Morrell Druckman recruited a moderate number of reports are nearly all focused on satisfaction participants (580), with approximately 172 with group procedures and outcomes, mea- taking part in the group discussion condi- sures for which the nondeliberating control tions. This means that more than forty delib- condition are not relevant. In other words, erating groups could be studied. Morrell’s test as reported contrasts only dif- As with the Simon and Sulkin (2002) ferent types of discursive interaction. More- experiment, exposure to group discussion over, as with our preceding discussion of the in this research design is limited and may, “grand treatment,” the treatments in both therefore, understate the effect of group cases are complex, and it is not entirely clear discussion. But what is most helpful from which aspects of “generative” discussion led the perspective of deliberative theory is a 266 Christopher F. Karpowitz and Tali Mendelberg systematic manipulation of both the presence (Karpowitz and Mendelberg 2007;Men- of discussion and the context under which delberg and Karpowitz 2007; Karpowitz, discussion occurred. Druckman (2004) finds Mendelberg, and Argyle 2008). We do this that the presence of discussion matters – par- to highlight a few of the methodological ticipants in both the homogenous and het- issues that emerged as we conducted the erogeneous conditions proved less vulnerable research. Our interest in experiments began to framing effects than in the control condi- when we reanalyzed data collected previously tion. This would seem to be positive evidence by Frohlich and Oppenheimer (1992). Partic- for the relationship between deliberation and ipants in the experiment were told that they rationality, but the story is somewhat more would be doing tasks to earn money and that complicated than that. Neither discussion the money they earned would be based on a condition reduced framing effects as much as group decision about redistribution, but that simply giving the counterframe to each indi- prior to group deliberation, they would not vidual without requiring group discussion. be told the nature of the work they would Moreover, homogenous groups appeared to be doing. This was meant to simulate the be comparatively more vulnerable to framing Rawlsian veil of ignorance; that is, individ- effects compared to heterogeneous groups. uals would not know the specifics of how Results were also strongly mediated by their decision would affect them personally expertise. because they would not know how well or Druckman’s (2004) research design ref- poorly they might perform. lects several attributes worthy of emu- During deliberation, groups were instruc- lation. First, the number of groups is ted to choose one of several principles of sufficiently sizeable for meaningful statistical justice to be applied to their earnings, includ- inference. Second, Druckman uses the key ing the option not to redistribute at all. The levers of experimental control and random principle chosen would simultaneously gov- assignment appropriately. This allows him to ern the income they earned during the exper- make meaningful claims about the difference iment (which was translated into a yearly between discussion across different contexts income equivalent) and apply (hypothetically) and the difference between discussion and the to the society at large. Groups were ran- simple provision of additional information. domly assigned to one of three conditions: As we discuss in the previous section, one of imposed, unanimous, and majority rule. In the key problems of causal inference in the the imposed condition, groups were assigned “grand treatment” design has been whether a principle of justice by the experimenters. deliberation is responsible for the observed In the unanimous and majority conditions, effects or whether one particular aspect of groups had to choose a principle of jus- it – the provision of information – is respon- tice either unanimously or by majority vote, sible. Given that information is not unique to respectively. deliberation, finding that the effects of delib- The key finding of Frohlich and Oppen- eration are due primarily to information heimer’s (1992) original study was that, when would considerably lessen the appeal and given an opportunity to deliberate behind value of deliberation as a distinct mode of the veil of ignorance, most groups choose to participation. Druckman’s results do raise guarantee a minimum income below which further questions, however, especially with the worst-off member of the group would respect to what is actually happening during not be allowed to fall. In our reanalysis of the discussion period. Druckman does not their data (Mendelberg and Karpowitz 2007), look inside the “black box” of discussion to we noted that Frohlich and Oppenheimer understand how the dynamics and the content paid little attention to the ways in which of discussion vary across the homogenous and the group context shaped participants’ atti- heterogeneous conditions. tudes and group-level outcomes. Our reanal- Finally, we add a few words about our ysis showed that important features of the own experimental work on deliberation group context, such as the group’s gender An Experimental Approach to Citizen Deliberation 267 composition and its decision rule, interacted ment random assignment of individuals to to significantly affect group- and individual- group conditions. level experimental outcomes. However, the Having outlined both positive features and previous data were also limited to a signi- further questions that emerge from several ficant extent. First, participants in that highlighted experiments, we turn next to experiment were not randomly assigned to some of the challenges of effective experi- conditions. Second, the data did not include mentation about deliberation. enough groups of varying gender composition for us to be entirely confident in our statistical results. 4. Challenges For those reasons, we chose to conduct our own updated version of the experiment, We detail an argument in which we urge this time with random assignment, a suffi- more investigations using experimental cient number of groups (nearly 150), and sys- methods; more care in designing treatments tematic manipulation of the gender/decision that manipulate various aspects of delibera- rule conditions. We also carefully recorded tion; and, particularly, more frequent use of each group discussion in order to explore random assignment to conditions. However, more fully the dynamics of the group inter- the more control the investigator seeks, the actions themselves, tying the verbal behav- greater the trade-offs. Control brings arti- ior of each participant during deliberation to fice and narrow, isolated operationalizations their pre- and postdiscussion attitudes about of rich and complex concepts. The behavior the functioning of the group, the need for of interest is often embedded in the contexts redistribution, and a host of other variables. of institutions and social relationships, and Our analysis is still in its initial stages, but must therefore ultimately be moved back out we do find evidence that the group-level fac- of the lab, where every effort was made to tors, especially the interaction of group gen- isolate it, and studied again with attention der and decision rule, affect various aspects to these contexts. There are other difficul- of the group’s functioning and deliberative ties involved in the use of experiments – that dynamics. We also find significant differences is, they may be more expensive and effortful between groups that deliberated and control than other methods. Here we consider these groups that did not. trade-offs. In sum, we attempt in our work to advance One of the challenges of experimentation the study of deliberation methodologically in is the operationalization of idealized norma- several ways. We use a larger n, particularly tive theory. Experimental approaches may increasing the group n; we employ random be particularly vulnerable to the disagree- assignment; and our design both controls on ments between theorists and empiricists to deliberation itself and isolates the effects of the extent that their heightened levels of specific aspects of deliberation, some of which control bring more stylized and more artifi- derive from empirical studies of citizen dis- cial operationalizations of complex and mul- cussion (e.g., decision rule, the group’s het- tifaceted theoretical concepts. We illustrated erogeneity or homogeneity), some of which the issue with a contrast between Fishkin’s focus on normatively relevant processes of (1995) “grand treatment” versus Simon and communication (e.g., equal participation in Sulkin’s (2002) decision to trade off the com- discussion, use of linguistic terms reflecting a plexity of deliberation against the ability to concern for the common good), and some of control it. The trade-off is understandable, which supplement these theories by focusing but not necessary. It is possible to design a on sociologically important variables (e.g., controlled experiment with random assign- the group’s demographic composition). In ment of multiple conditions – one of which conducting these experiments, we also began resembles the “grand treatment” notion; oth- to directly confront some of the practical ers that isolate each major element of the challenges inherent in attempting to imple- deliberation; and still others with a control 268 Christopher F. Karpowitz and Tali Mendelberg condition identical in every way, but lacking to choose to deliberate. Karpowitz’s (2006) these elements. analysis of patterns of meeting attendance A related second challenge for experimen- in a national sample suggests that people tation is external validity. It is difficult not who attend meetings are not a random only to adequately operationalize the key con- sample of the adult population – they are cepts of normative theory – to achieve con- not only more opinionated (although not struct validity (Campbell and Stanley 1963)– more ideologically extreme) than nonatten- but also to simulate the causal relationships as ders, but also more interested in politics, they occur in the real world. We need to know more knowledgeable about it, and more likely ultimately how deliberative efforts interact with to discuss political issues frequently. In a real-world actors and institutions. For exam- controlled experiment, people also exercise ple, Karpowitz’s (2006) study of a local civic some level of choice as to whether to par- deliberation suggests that the deliberators’ ticipate, but to a much lesser extent. An knowledge that they could pursue their pref- experiment described up front as focused on erences after the deliberation was over by deliberation may better approximate a real- lobbying the city council, writing letters to world setting in which people choose to newspapers, and filing lawsuits in the courts participate in deliberation, and people may significantly affected various aspects of delib- choose to participate in the experiment for eration, including the ability of the delib- the same reasons they choose to participate eration to change minds, enlarge interests, in real-world deliberations. But some delib- resolve conflicts, and achieve other ends envi- eration experiments may not be described sioned by normative theorists. This presents that way. The question then becomes to what a challenge to the external validity of exper- extent are the processes and effects of delib- iments in that their deliberative situation is eration generalizable from the sample in the abstracted from interaction with real institu- experiment to the samples in the real world. tions. However, the cumulation of findings Attending to the relationship between such as Karpowitz’s from observational stud- deliberating groups and the wider political ies can, in turn, lead to further hypothesis context, and to the differences between those testing using experimental designs, where the who choose to deliberate and those who do impact of particular institutional contexts can not, also raises the question of how delibera- be isolated and studied rigorously. tion might affect those who do not participate Mansbridge’s (1983) study of a New Eng- directly, but who observe the deliberation of land town meeting is the classic example of others or merely read about the work of delib- how a careful observational study can lead to erating groups. Given the problem of scale, theoretically rich insights into group discus- deliberation is unlikely to be all inclusive, and sion and decision making as practiced in the those who sponsor opportunities for deliber- real world. She shows, for example, how res- ation must also communicate their processes idents of the town struggle to navigate their and results to the wider public. How those common and conflicting interests in group who were not part of the discussion under- settings, and how the presence or absence stand deliberating groups is a topic worth of conflicting interests shapes the dynamics considerable additional study, including with of the discussion and patterns of attendance experimental approaches. at the town meeting. The textured details A third and final issue with random- of real-world observation found in Mans- ized experiments is of a practical nature. It bridge’s work are often lacking in experimen- is extremely difficult to implement random tal studies, but her work can also be seen as assignment in the study of deliberation. One articulating a set of hypotheses that can be variant of this problem comes in the “grand explored much more deeply with the control treatment” design – the holistic treatment that experiments provide. and attempt to approximate the ideal con- Another aspect of external validity is that, ditions specified by normative theory require in real-world settings, citizens often have a substantial commitment of time and effort An Experimental Approach to Citizen Deliberation 269 by deliberators. A significant percentage of language – that is, we can train the analytic those assigned to a demanding deliberation microscope more directly on the process of condition may well refuse treatment, and the deliberation itself, even though this practice decision to drop out of the treatment con- is also still rare. dition may well be nonrandom, introducing Although experimental control allows bias into the estimates of causal effects. Lab- for unique causal inference, experiments based deliberation experiments may face less miss some of the richness of real-world severe problems because random assignment deliberative settings. In-depth observational takes place after subjects come to the lab, so case studies can fill the gap and uncover that participants are less likely to opt out of the meaning of key concepts (e.g., Mans- the treatment due to its demanding nature. bridge 1983; Eliasoph 1998). Indeed, if we Another variant of this problem presents were forced to choose between Mansbridge’s itself when variables of interest are at the classic work and many experiments, we might group level. This requires a large number prefer Mansbridge. The ideal research design of groups, which in turn requires a much is an iterative process in which experimenta- larger individual n than lab experiments typ- tion in the lab is supplemented and informed ically use. The practical challenges of ran- by observation of real-world settings. Our dom assignment to group conditions can be ecumenism is not, however, a call for a con- significant, especially when potential partic- tinuation of the hodgepodge of studies that ipants face a variety of different time con- currently characterizes the field. Instead, we straints. For example, in our work on group need a tighter link between the variables composition and deliberation, we found that observed in real-world discussions and those simultaneously accounting for differences in manipulated in controlled settings. In addi- participants’ availability and instituting a ran- tion, the external validity concerns typical of dom assignment procedure that ensures each experiments generally apply in the case of recruited participant a roughly equal chance deliberation, and may be addressed by sup- of being assigned to all relevant groups is a plementing controlled experiments with field complicated exercise. experiments that make use of the explosion of deliberative reform efforts in cities and towns across the United States. Field experiments 5. Conclusion – What’s Next? will be especially helpful if they allow the investigator access to accurate measures of We begin this chapter with the notion mediating and outcome variables. It is unclear that empirical research can usefully evalu- whether they do in fact allow such a degree ate the claims of deliberative theorists, and of access, but the effort is worth making. we develop an argument about the special utility of controlled experiments. The control afforded by experiments allows not only References strong causal inference, but also the ability to measure, and therefore to study, mediat- Barabas, Jason. 2004. “How Deliberation Affects ing and outcome variables with a heightened Policy Opinions.” American Political Science level of precision and accuracy. We argue that Review 98: 687–701. 1984 despite a proliferation of self-titled delibera- Barber, Benjamin. . Strong Democracy: Partic- tive “experiments,” methodologically rigor- ipation Politics for a New Age. Berkeley: Univer- ous research design with sufficient control sity of California Press. Benhabib, Seyla. 1996. “Toward a Deliberative and random assignment is still a relative rar- Model of Democratic Legitimacy.” In Democ- ity. We are anxious to see experiments with racy and Difference: Contesting the Boundaries of an increased number of participants, espe- the Political, ed. Seyla Benhabib. Princeton, NJ: cially an increased number of groups. Exper- Princeton University Press, 67–94. imental approaches can also use their high Bohman, James. 1997. “Deliberative Democ- level of control to measure the exchange of racy and Effective Social Freedom: Resources, 270 Christopher F. Karpowitz and Tali Mendelberg

Opportunities and Capabilities.” In Delibera- Democratic Deliberation? The California tive Democracy, eds. James Bohman and William Speaks Health Care Reform Experiment.” Paper Rehg. Boston: MIT Press. presented at the annual meeting of the Midwest Burkhalter, Stephanie, John Gastil, and Todd Political Science Association, Chicago. Kelshaw. 2002. “A Conceptual Deﬁnition and Esterling, Kevin, Michael Neblo, and David Theoretical Model of Public Deliberation in Lazer. 2008a. “Estimating Treatment Effects Small Face-to-Face Groups.” Communication in the Presence of Selection on Unobserv- Theory 12: 398–422. ables: The Generalized Endogenous Treat- Campbell, Donald T., and Julian Stanley. 1963. ment Model.” Paper presented at the annual Experimental and Quasi-Experimental Designs for meeting of the American Political Science Research. New York: Wadsworth. Association, Chicago. Chambers, Simone. 1996. Reasonable Democracy. Esterling, Kevin, Michael Neblo, and David Ithaca, NY: Cornell University Press. Lazer. 2008b. “Means, Motive, and Oppor- Chambers, Simone. 2003. “Deliberative Demo- tunity in Becoming Informed about Politics: cratic Theory.” Annual Review of Political Science A Deliberative Field Experiment.” Paper pre- 6: 307–26. sented at dg.o 2007, the eighth annual inter- Conover, Pamela J., David D. Searing, and Ivor national conference on digital government Crewe. 2002. “The Deliberative Potential of research, Philadelphia. Political Discussion.” British Journal of Political Fishkin, James S. 1995. The Voice of the People.New Science 32: 21–62. Haven, CT: Yale University Press. Converse, Philip E. 1964. “The Nature of Belief Frohlich, Norman, and Joe Oppenheimer. 1992. Systems in Mass Publics.” In Ideology and Dis- Choosing Justice. Cambridge: Cambridge Uni- content, ed. David E. Apter. New York: Free versity Press. Press, 206–261. Fung, Archon. 2003. “Deliberative Democracy Cook, Thomas D., and Donald T. Campbell. and International Labor Standards.” Gover- 1979. Quasi-Experimentation: Design and Anal- nance 16: 51–71. ysis Issues for Field Settings. New York: Gaertner, Samuel L., John F. Dovidio, Mary C. Wadsworth. Rust, Jason A. Nier, Brenda S. Banker, Chris- Cramer Walsh, Katherine. 2006. “Communi- tineM.Ward,GaryR.Mottola,andMissy ties, Race, and Talk: An Analysis of the Houlette. 1999. “Reducing Intergroup Bias: Occurrence of Civic Intergroup Dialogue Elements of Intergroup Cooperation.” Journal Programs.” Journal of Politics 68: 22–33. of Personality and Social Psychology 76: 388–402. Cramer Walsh, Katherine. 2007. Talking about Gamson, William A. 1992. Talking Politics.Cam- Race: Community Dialogues and the Politics of Dif- bridge: Cambridge University Press. ference. Chicago: The University of Chicago Gastil, John. 2008. Political Communication and Press. Deliberation. Thousand Oaks, CA: Sage. Devine, Dennis J., Laura Clayton, Benjamin B. Gastil, John, E. Pierre Deess, and Phil Weiser. Dunford, Rasmy Seying, and Jennifer Price. 2002. “Civic Awakening in the Jury Room: A 2001. “Jury Decision Making: 45 Years of Test of the Connection between Jury Delib- Empirical Research on Deliberating Groups.” eration and Political Participation.” Journal of Psychology, Public Policy, and Law 73: 622–727. Politics 64: 585–95. Druckman, James N. 2004. “Political Preference Gastil, John, E. Pierre Deess, Phil Weiser, and Formation: Competition, Deliberation, and the Jordan Meade. 2008. “Jury Service and Elec- (Ir)relevance of Framing Effects.” American toral Participation: A Test of the Participation Political Science Review 98: 671–86. Hypothesis.” Journal of Politics 70: 1–16. Dryzek, John S. 2007. “Theory, Evidence, and the Gastil, John, and Peter Levine. 2005. The Deliber- Tasks of Deliberation.” In Deliberation, Partic- ative Democracy Handbook: Strategies for Effective ipation and Democracy: Can the People Govern? Civic Engagement in the Twenty-First Century. ed. Shawn W. Rosenberg. Basingstoke, UK: San Francisco: Jossey Bass. Palgrave Macmillan, 237–250. Gutmann, Amy, and Dennis Thompson. 1996. Eliasoph, Nina. 1998. Avoiding Politics: How Amer- Democracy and Disagreement. Cambridge, MA: icans Produce Apathy in Everyday Life.Cam- Harvard University Press. bridge: Cambridge University Press. Gutmann, Amy, and Dennis Thompson. 2004. Esterling, Kevin, Archon Fung, and Taeku Lee. Why Deliberative Democracy? Princeton, NJ: 2010. “How Much Disagreement Is Good for Princeton University. An Experimental Approach to Citizen Deliberation 271

Habermas, Jurgen.¨ 1989. The Structural Transfor- Manin, Bernard. 1987. “On Legitimacy and Polit- mation of the Public Sphere: An Inquiry into a ical Deliberation.” Political Theory 15: 338–68. Category of Bourgeois Society, trans. by T. Burger Mansbridge, Jane. 1983. Beyond Adversary Democ- with the assistance of F. Lawrence. Cambridge, racy. Chicago: The University of Chicago MA: MIT Press. Press. Habermas, Jurgen.¨ 1996. Between Facts and Norms: Mansbridge, Jane, Janette Hartz-Karp, Matthew Contributions to a Discourse Theory of Law and Amengual, and John Gastil. 2006.“Normsof Democracy. Cambridge, MA: MIT Press. Deliberation: An Inductive Study.” Journal of Hafer, Catherine, and Dimitri Landa. 2007. Public Deliberation 2: Article 7. “Deliberation as Self-Discovery and Institu- Meirowitz, Adam. 2007. “In Defense of Exclusion- tions for Political Speech.” Journal of Theoretical ary Deliberation: Communication and Voting Politics 19: 325–55. with Private Beliefs and Values.” Journal of The- Hastie, Reid, Steven D. Penrod, and Nancy Pen- oretical Politics 19: 301–27. nington. 1983. Inside the Jury. Cambridge, MA: Mendelberg, Tali. 2002. “The Deliberative Citi- Harvard University Press. zen: Theory and Evidence.” In Political Decision Hibbing, John, and Elizabeth Theiss-Morse. 2002. Making, Deliberation and Participation: Research “The Perils of Voice: Political Involvement’s in Micropolitics. vol. 6, eds. Michael X. Delli Potential to Delegitimate.” Presented at the Carpini, Leonie Huddy, and Robert Y. Shapiro. annual meeting of the American Political Sci- Greenwich, CT: JAI Press, 151–193. ence Association, Boston. Mendelberg, Tali, and Christopher Karpowitz. Huckfeldt, Robert, and John Sprague. 1995. 2007. “How People Deliberate about Justice.” Citizens, Politics and Social Communication: In Deliberation, Participation and Democracy: Can Information and Influence in an Election Cam- the People Govern? ed. Shawn W. Rosenberg. paign. New York: Cambridge University Basingstoke, UK: Palgrave Macmillan, 101– Press. 129. Jacobs, Lawrence, Fay Lomax Cook, and Michael Merkle, Daniel M. 1996. “The National Issues X. Delli Carpini. 2009. Talking Together: Pub- Convention Deliberative Poll.” Public Opinion lic Deliberation and Politics in America. Chicago: Quarterly 60: 588–619. The University of Chicago Press. Mitofsky, Warren J. 1996. “It’s Not Delibera- Karpowitz, Christopher F. 2006. “Having a Say: tive and It’s Not a Poll.” Public Perspective 7: Public Hearings, Deliberation, and Ameri- 4–6. can Democracy.” Unpublished dissertation, Morrell, Michael. 1999. “Citizens’ Evaluations of Princeton University. Participatory Democratic Procedures: Norma- Karpowitz, Christopher F., and Tali Mendelberg. tive Theory Meets Empirical Science.” Political 2007. “Groups and Deliberation.” Swiss Political Research Quarterly 52: 293–322. Science Review 13: 645–62. Muhlberger, Peter, and L. M. Weber. 2006. Karpowitz, Christopher, Tali Mendelberg, and “Lessons from the Virtual Agora Project: The Lisa Argyle. 2008. “Group Effects and Delib- Effects of Agency, Identity, Information, and eration: The Deliberative Justice Experiment.” Deliberation on Political Knowledge.” Journal Paper presented at the International Society for of Public Deliberation 2: Article 6. Political Psychology (ISPP) 31st annual scien- Mutz, Diana C. 2006. Hearing the Other Side: tific meeting, Sciences Po, Paris. Deliberative versus Participatory Democracy.New Kinder, Donald R., and Thomas R. Palfrey. 1993. York: Cambridge University Press. Experimental Foundations of Political Science.Ann Mutz, Diana C. 2008. “Is Deliberative Democracy Arbor: University of Michigan Press. a Falsifiable Theory?” Annual Review of Political Kohut, Andrew. 1996. “The Big Poll That Science 11: 521–38. Didn’t.” Poll Watch 4: 2–3. Myers, C. Daniel. 2009. “Information Sharing in Luskin, Robert C., James S. Fishkin, and Roger Democratic Deliberation.” Paper presented at Jowell. 2002. “Considered Opinions: Deliber- the annual meeting of the Midwest Political ative Polling in Britain.” British Journal of Polit- Science Association, Chicago. ical Science 32: 455–87. Polletta, Francesca. 2008. “Just Talk: Public Macedo, Stephen. 1999. “Introduction.” In Delib- Deliberation after 9/11.” Journal of Public Delib- erative Politics: Essays on Democracy and Disagree- eration 4: 1–24. ment, ed. Stephen Macedo. New York: Oxford Price, Vincent, and Joseph N. Cappella. 2005. University Press, 3–14. “Constructing Electronic Interactions among 272 Christopher F. Karpowitz and Tali Mendelberg

Citizens, Issue Publics, and Elites: The Health- The Severity Shift.” Columbia Law Review 100: care Dialogue Project.” In Proceedingsofthe 1139–76. National Conference on Digital Government Re- Simon, Adam, and Tracy Sulkin. 2002.“Dis- search. Atlanta: Digital Government Research cussion’s Impact on Political Allocations: An Center, 139–40. Experimental Approach.” Political Analysis 10: Price, Vincent, and Joseph N. Cappella. 2007. 402–11. “Healthcare Dialogue: Project Highlights.” In Steiner, Jurg, Andre Bachtiger, Markus Sporndli, Proceedings of the National Conference on Digi- and Marco Steenbergen. 2005. Deliberative Pol- tal Government Research. Philadelphia: Digital itics in Action: Analyzing Parliamentary Discourse Government Research Center, 178. (Theories of Institutional Design). Cambridge: Rosenberg, Shawn W. 2007. Deliberation, Partic- Cambridge University Press. ipation and Democracy: Can the People Govern? Thompson, Dennis. 2008. “Deliberative Demo- Basingstoke, UK: Palgrave Macmillan. cratic Theory and Empirical Political Science.” Ryfe, David M. 2005. “Does Deliberative Democ- Annual Review of Political Science 11: 497–520. racy Work?” Annual Review of Political Science Warren, Mark E. 1992. “Democratic Theory and 8: 49–71. Self-Transformation.” American Political Science Sanders, Lynn M. 1997. “Against Deliberation.” Review 86: 8–23. Political Theory 25: 347–76. Warren, Mark E., and Hilary Pearse. 2008.“Intro- Schildkraut, Deborah. 2005. Press One for English: duction.” In Designing Deliberative Democracy: Language Policy, Public Opinion, and American The British Columbia Citizens’ Assembly, eds. Identity. Princeton, NJ: Princeton University Mark E. Warren and Hilary Pearse. Cam- Press. bridge: Cambridge University Press, 1–19. Schkade, David, Cass R. Sunstein, and Daniel Young, Iris Marion. 2000. Inclusion and Democracy. Kahneman. 2000. “Deliberating about Dollars: New York: Oxford University Press. CHAPTER 19

Social Networks and Political Context

David W. Nickerson

People are embedded in networks, neigh- of reliably estimating the importance of social borhoods, and relationships. Understanding ties becomes nearly impossible. Rather than the nature of our entanglements and how offering a comprehensive overview of the they shape who we are is fundamental to wide number of topics covered by social social sciences. Networks are likely to explain networks, this chapter focuses on the com- important parts of personal development and mon empirical challenges faced by studies of contemporary decision making. Researchers social networks by considering the challenges have found social networks to be important faced by observational studies of social net- in activities as disparate as voting (Berelson, works, discussing laboratory approaches to Lazarsfeld, and McPhee 1954), immigration networks, and describing how network exper- patterns (Sanders, Nee, and Sernau 2002), iments are conducted in the field. The chapter finding a job (Nordenmark 1999), recy- concludes by summarizing the strengths and cling (Tucker 1999), deworming (Miguel and weaknesses of the different approaches and Kremer 2004), cardiovascular disease and considers directions for future work. mortality (Kawachi et al. 1996), writing legislation (Caldeira and Patterson 1987), and even happiness (Fowler and Christakis 2008). 1. Observational Studies A wide range of political outcomes could be Cross-Sections studied using social networks; the only limitation is that the outcome be measurable. The social networks literature blossomed Ironically, the very ubiquity and importance during the 1940s and 1950s, with works such of social networks make them difficult to as The People’s Choice (Lazarsfeld, Berelson, study. Isolating causal effects is always dif- and Gaudet 1948) and Voting (Berelson et al. ficult, but when like-minded individuals clus- 1954). Using newly improved survey technol- ter together, share material incentives, are ogy, the authors administered surveys to ran- exposed to common external stimuli, and domly selected respondents densely clustered simultaneously influence each other, the job in medium-sized communities. This strategy

273 274 David W. Nickerson provided insight into what the neighbors and of social networks are exposed to many of friends of a respondent believed at the same the same external stimuli (e.g., media cover- point in time, allowing correlations in the age, economic conditions, political events). If behaviors and beliefs of friends and neigh- the external stimuli influence members of the bors to be measured. The authors found that social network similarly, then observed corre- information flowed horizontally through net- lations could be due to these outside pressures works and overturned the opinion leadership rather than the effect of the network. These model of media effects. problems can be categorized as forms of omit- 1 The advent of affordable nationally rep- ted variable bias and call into question results resentative polling largely ended this mode based on cross-sectional surveys. Within the of inquiry. Why study one community when framework of a cross-sectional study, it is dif- you could study an entire nation? Unfortu- ficult to conceive of data that could convince nately, the individuals surveyed from around a skeptic that the reported effects are not the nation had no connection to one another, spurious. so the theories developed based on these data generally assumed atomistic voters (e.g., Panels Campbell et al. 1964). The political context literature was revived only when Robert Part of the problem is that influence within Huckfeldt and John Sprague returned to a network is an inherently dynamic process. the strategy of densely clustering surveys in A person begins with an attitude or propen- communities, while adding a new method- sity for a given behavior and the network acts ological innovation. They used snowball sur- on this baseline. Observational researchers veys, where respondents were asked to name can improve their modeling of the process political discussants, who were then surveyed by collecting panel data where the same set themselves. This technique allowed Huck- of individuals is followed over time through feldt and Sprague to directly measure the multiple waves of a survey. This strategy political views in a person’s network rather allows the researcher to account for base- than infer the beliefs of discussants from line tendencies in respondents and measure neighborhood characteristics. In a series of movement away from these baselines. More- classic articles and books, they and their over, measured and unmeasured attributes of many students meticulously documented the a person can be accounted for by including degree to which political engagement is a fixed effects for each individual in the sam- social process for most people (e.g., Huck- ple. In this way, panel data can account for all feldt 1983; Huckfeldt and Sprague 1995). It time-invariant confounding factors. is fair to say that most contemporary observa- Panel studies in political science are rare tional studies of political behavior and social because of the expense involved. Studies networks either rely on survey questions to focusing on social networks are even less map social networks or ask questions about common and nearly always examine fami- politically relevant conversations (e.g., Mutz lies – one of the most fundamental networks 1974 1981 1998). in society. Jennings and Niemi’s ( , ) These empirical strategies face three pri- classic survey of families over time is the best mary inferential hurdles, making it diffi- known panel in political science examining cult to account for all plausible alternative how political attitudes are transferred from causes of correlation. First, people with simi- parents to children (and vice versa). More lar statuses, values, and habits are more likely often, political scientists are forced to rely on to form friendships (Lazarsfeld and Merton a handful of politically relevant questions in 1954), so self-sorting rather than influence panel studies conducted for other purposes could drive results. Second, members of a (e.g., Zuckerman, Dasovic, and Fitzgerald 2007 social network are likely to share utility func- ). tions and engage in similar behaviors inde- 1 Selection bias can be even more pernicious than omit- pendently of one another. Third, members ted variable bias (see Achen 1986). Social Networks and Political Context 275

Although they provide a huge advance over and the strength of the observed relationship. cross-sectional data, panel data cannot pro- First, unobserved factors that influence both vide fully satisfactory answers. Even if the alters and egos could drive the results. Sec- type of networks considered could be broad- ond, the strength of the relationship detected ened, dynamic confounding factors, such as violates a few causal models. For instance, congruent utility functions, life cycle pro- Christakis and Fowler (2007) find that “geo- cesses, and similar exposure to external stim- graphic distance did not modify the intensity uli, remain problematic. Furthermore, if the of the effect of the alter’s obesity on the ego” baseline attitudes and propensities are mea- (377). The primary mechanism for jointly sured with error and that error is corre- gaining and losing weight would presum- lated with politically relevant quantities, then ably be shared meals, calorie-burning activ- the chief advantage of panel data is removed ities such as walking, or perhaps competitive because the dynamic analysis will be biased. pressure to remain thin. However, none of these mechanisms works for geographically distant individuals, raising the concern that Network Analysis 2 selection bias is driving the results. Third, Network analysis is touted as a method to ana- Cohen-Cole and Fletcher (2008) adopt a sim- lyze network data to uncover the relationships ilar empirical strategy on a similar dataset to within a network. Sophisticated economet- Christakis and Fowler (2007, 2008) and find ric techniques have been developed to mea- evidence that acne and height are also con- sure the strength of ties within networks and tagious, which constitute failed placebo tests. their effects on various outcomes (Carring- None of these points disproves the claims by ton, Scott, and Wasserman 2005). Instead of Christakis and Fowler, but each does call into assuming the independence of observations, doubt the evidence provided and the strength network models adjust estimated coefficients of the relationships detected. to account for correlations found among The Framingham Heart Study is a nearly other observations with ties to each other. perfect observational social network dataset. Network analysis is a statistical advancement, If the answers provided remain unconvinc- but it does not surmount the core empiri- ing, then perhaps the observational strategy cal challenge facing observational studies of should be rejected in favor of techniques social networks, which is essentially a data using randomized experiments. Ordinarily, problem. Similar utility functions and expo- experiments can get around self-selection sure to external stimuli remain problems, as problems and unobserved confounding fac- do selection effects. Selection effects are not tors through randomization, but the organic only present, but are also reified in the model nature of most social networks poses a dif- and analysis being used to define the nodes ficult problem. To test the power of social and ties of the network. networks, the ideal experiment would place To illustrate the challenge facing obser- randomly selected individuals in a range of vational studies of social networks, consider varying political contexts or social networks. the recent work by Christakis and Fowler The practical and ethical concerns of mov- (2007, 2008; Fowler and Christakis 2008) ing people around and enforcing friendships using the Framingham Heart Study. To sup- are obvious. The time-dependent nature plement health and behavioral data collected of social networks also makes them inher- since 1948, the Framingham Heart Study ently difficult to manipulate. Reputation and began collecting detailed social network data friendships take a long time to develop and in 1971. Taking advantage of the panel and cannot be manufactured and manipulated in network structure of the data, Christakis and any straightforward manner. Thus, the exper- Fowler found evidence that obesity, smok- imental literature testing the effect of social ing, and happiness were contagious. Although 2 A similar problem arises when happiness is found to the claim is entirely plausible, there are three be more contagious for neighbors than coworkers or reasons to question the evidence provided spouses. 276 David W. Nickerson networks on behaviors and beliefs is still in these experiments measure conformity, how its infancy. Having said that, the next section strongly the findings apply in real-world set- discusses the laboratory tradition that began tings is unclear. First, the participants are in the 1950s. inserted into a peer group with no real connection or bond. These essentially anonymous and ahistorical relationships may accu- 2. Laboratory Experiments rately characterize commercial interactions, but differ in character from social networks Assign Context classically conceived. Second, subjects are Many tactics have been used to study social presented with an artificial task with limited networks in laboratories. The central logic or no outside information on the context (e.g., behind them is for the researcher to situate estimating the length of a line). Thus, partic- subjects in a randomly assigned social context. ipants may have little stake in the proceed- One of the most famous examples is Asch’s ings and may not take the exercise seriously (1956) series of classic experiments on con- (i.e., subjects want to avoid arguments on triv- formity. Subjects were invited to participate ial matters or believe that they are playing a in an experiment on perception where they joke on the experimenter). Asch-style experi- had to judge the length of lines. Control sub- ments measure a tendency to conform, but it jects performed the task alone, whereas sub- is unclear how and under which conditions jects in the treatment group interacted with the results translate to real-world political confederates who guessed incorrectly. Sub- settings. jects in the control group rarely made mistakes, whereas individuals in the treatment Randomly Constructed Network group parroted the errors of the confederates frequently. The initial study was criti- Creative strategies have been designed to cized for relying on a subject pool of male respond to these criticisms about external undergraduates that may not be representa- validity. A recent tactic embraces the isolation tive of the population as a whole. However, of the laboratory and uses abstract coordina- the Asch experiments have been replicated tion games with financial incentives for sub- hundreds of times in different settings (Bond jects linked to the outcome of the game. The and Smith 1996). Although the conformity advent of sophisticated computer programs to effect persists, it 1) varies across cultures, 2)is aid economic games played in the laboratory stronger for women, 3) has grown weaker in has facilitated a number of experiments that the United States over time, and 4) depends directly manipulate the social network and on parts of the experimental design (e.g., size the subject’s place in it (e.g., Kearns, Suri, and of the majority, ambiguity of stimuli) and Montfort 2006). Researchers can now iso- not others (e.g., whether the subject’s vote late the factors of theoretical interest within is public or private). Thus, the Asch (1956) social networks. For instance, researchers can experiments constitute evidence that peers – manipulate the degree of interconnectedness, even ones encountered for the first time – can information location, preference symmetry, shape behavior. and external monitoring. Much of the literature in psychology The downside of this strategy is that the employs tactics similar to those used by Asch networks are not only artificial, but also (1956). For instance, social loafing (Karau entirely abstracted, and may not approximate and Williams 1993) and social facilitation the operation of actual networks. Strategy (Bond and Titus 1983) can boast equally convergence among players may reflect the long pedigrees and replications.3 Although ability of students to learn a game rather than measure how social networks operate. Hav- 3 Social loafing dates back to at least 1913,when ing said that, such experiments serve as a use- Ringelmann found individuals pulled harder on a rope when working alone than when working in con- ful “proof of concept” for formal theories of cert with others (Kravitz and Martin 1986). social networks. If people are in networks like Social Networks and Political Context 277

X, then people will behave like Y. The chal- real-world relationships in order to approx- lenge is to link real-world phenomena to par- imate reality. However, not all details can ticular games. be incorporated, and researchers must make decisions about what features to highlight. The downside of this drive for verisimili- Role Playing tude is that the highlighted attributes (e.g., To create more realistic social networks, “parents” providing “allowance”) may shape researchers can have subjects engage in col- the behaviors of subjects, who take cues and laborative group tasks to create camaraderie, conform to expected behaviors. These fram- share information and views about a range of ing decisions therefore affect experiments and subjects to simulate familiarity, and anticipate potentially make the results less replicable. future encounters by scheduling postinter- Thus, the degree to which social network vention face-to-face discussion (e.g., Visser experiments involving role-playing approxi- and Mirabile 2004). These efforts to jump- mate organic social networks found in the real start genuine social connections or mimic world is open to question. attributes of long-standing relationships are partial fixes. If organic social networks gen- Small Groups erated over years behave differently from those constructed in the laboratory, then it Experiments where subjects deliberate in is unclear how relationship-building exercises small groups are an important subset of role- blunt the criticism. playing experiments. For example, Fishkin’s To address some concerns about exter- Deliberative Polling (Luskin, Fishkin, and nal validity, some laboratory experiments Jowell 2002) invites randomly selected mem- allow subjects great freedom of action. By bers of a community to discuss a topic for randomly assigning subjects roles to be a day. Participants are typically provided played in scenes, researchers hope to gain with briefing materials and presentations by insight into real-world relationships. The experts. The experimental component of the most famous example of this strategy is the exercise is that subjects are randomly placed Stanford Prison Experiment conducted in into small groups to discuss the topic at hand. 1971 (Haney, Banks, and Zimbardo 1973), Thus, subjects could be placed in a group where students were asked to act out the that is ideologically like-minded, hostile, or roles of prisoners and guards.4 Most role- polarized. By measuring attitudes before and playing experiments are not so extreme, but after the small-group deliberation, it is possi- the same criticisms often apply. If subjects ble to estimate the shift in opinion caused consciously view themselves as acting, then by discussion with liberal, conservative, or the degree to which the role-play reflects moderate citizens. The random assignment actual behavior is an open question. Behaviors to small-group discussion ensures that a sub- may differ substantially when subjects view ject’s exposure to the opposing or supporting participation as a lark and divorced from real- viewpoints is not correlated with any char- ity. A common critique of laboratory experi- acteristics of the individual.6 In this way, ments is that they draw on undergraduates for researchers can infer how the viewpoint of their subject pool, but the critique has added discussion partners affects an individual. bite in this setting.5 Whether more mature The evidence of attitudinal contagion individuals would behave similarly given the from these experiments is mixed (Farrar et al. roles assigned is an open question. Many role- 2009), but the model is useful to con- playing experiments incorporate features of sider. Because these experiments consist of randomly selected citizens talking to other 4 The experiment was halted after six days due to phys- randomly selected citizens, many concerns ical and psychological abuse by guards. 5 For a somewhat contradictory view on the external validity of student samples, see Druckman and Kam’s 6 Many mock jury experiments (e.g., Sunstein et al. chapter in this volume. 2003) share this characteristic. 278 David W. Nickerson about external validity are alleviated. The federates, eight confederates providing the subjects are representative of the com- wrong answer, or a group with a minority munity (conditional on cooperation), and of confederates providing the correct answer. the conversation is unscripted and natural The presence or absence of confederates and (depending on the moderator’s instructions). their role defines the social network and However, the setting itself does not occur manipulates the communication within the naturally. People discuss political matters network. The task of judging the line length is with members of their social networks, not the external shock used to measure the power randomly selected individuals – much less a of social influence. In theory, experiments set of people who have read common brief- conducted in the field could also pursue mul- ing materials on a topic. In fairness, the hypo- tiple randomization strategies because the thetical nature of the conversation is precisely categories are not mutually exclusive. In prac- Fishkin’s goal, because he wants to know the tice, a researcher will have difficulty manipu- decisions people would make were they to lating even one aspect of the social network. become informed and deliberate with one Organic social networks are difficult to map another. However, the hypothetical nature of and manipulate, so researchers have far fewer the conversation limits the degree to which analytic levers to manipulate compared to the the lessons learned from small-group activ- laboratory. ities can be applied to naturally occurring small groups. Logistical and Ethical Concerns Before discussing each of the experimental 3. Field Experiments strategies, it is worth considering a few of the practical hurdles that apply to all three Observational studies examine naturally research designs. The first difficulty is in occurring social networks, but may suffer measuring the network itself. The researcher from selection processes and omitted vari- has to know where to look for influence able biases. Laboratory experiments of net- in order to measure it, and the strategy works possess internal validity, but the social employed will inherently depend on the set- networks studied are typically artificial and ting. For instance, snowball surveys are a possibly too abstract to know how the results good technique for collecting data on social apply to real-world settings. Intuitively, con- networks in residential neighborhoods or for ducting experiments in the field could capture mapping friendships. Facebook and other the strengths of both research strategies. The social networking sites can be used on college reality is more complicated, given the diffi- campuses. Cosponsored bills in state legisla- culty of conducting experiments in the field, tures are another possibility. Many studies of the lack of researcher control, and unique interpersonal influence rely on geography as concerns about the external validity of field a proxy for social connectedness, assuming experiments themselves. that geographically proximate individuals are Three strategies can be applied to study more likely to interact with one another than social networks experimentally. Researchers with geographically distant people (Festinger, can provide an external shock and trace the Schachter, and Back 1950). Each measure- ripple through the network, control the flow ment technique defines the network along a of communication within a network, or ran- single dimension and will miss relationships domize the network itself. Although the three defined along alternative dimensions. Thus, categories cover most field experiments, the every study of social networks conducted in categorization does not apply to lab settings the field will be limited to the particular set where researchers often manipulate all three of ties explicitly measured. analytic levers simultaneously. For instance, It is important to note that the measure- intheAsch(1956) experiments, subjects are ment of the social network cannot be related randomly assigned to a network with no con- to the application of the treatment in any Social Networks and Political Context 279 way. Both treatment and control groups need be studied prior to randomization and appli- to have networks measured in identical man- cation of the treatment. The downside of ners. In most instances, this is accomplished defining the network in advance is that the by measuring social networks first and then analysis will be limited only to the networks randomly assigning nodes to treatment and the researcher measured ahead of time; less control conditions. This strategy also has obvious connections and dynamic relation- the benefit of preserving statistical power by ships will be omitted from the analysis. How- allowing for prematching networks to mini- ever, the avoidance of unnecessary assump- mize unexplained variance and assuring bal- tions and the clarity of analysis that results ance on covariates (Rosenbaum 2005). Given from clearly defining the network up front the small size of many networks studied, sta- more than compensate for this drawback. tistical power is not an unimportant consid- The second major problem facing field eration. experimental studies of networks is the inher- Although defining the network identi- ent unpredictability of people in the real cally for treatment and control variables may world, where behavior cannot be constrained. appear obvious, it imposes considerable logis- This lack of researcher control poses two pri- tical hurdles. Letting networks be revealed mary problems for experiments. First, if the through the course of the treatment imposes behavior of a volunteer network node is part a series of unverifiable assumptions and con- of the experimental treatment (e.g., initiating fuses the object of estimation. For instance, conversations), then planned protocols may in his classic Six Degrees of Separation exper- be violated. The violation is not necessar- iment, Milgram (1967) mailed letters to ran- ily because of noncompliance on the part of domly selected individuals and requested that the subjects whose outcomes are to be mea- they attempt to mail letters to a particular sured (e.g., refusing to speak with the exper- individual in a separate part of the country. If imental volunteer about the assigned topic), the subject did not know the individual (and but because the person designated to provide they would not), then they were instructed the treatment does not dutifully execute the to forward the letter to a person who would protocol in the way that laboratory assistants be more likely to know the target. Milgram typically do. Overzealous volunteers may then counted the number of times letters speak to more people than assigned, under- were passed along before reaching the target motivated volunteers may decide to exclude destination. hard to reach members of their network, or Revealed networks research designs, such the treatment may deviate substantially from as Milgram’s (1967), create data where the what researchers intend. To contain these networks measured may not be representative problematic participants and prevent biasing of the networks of interest. If network char- the overall experiment, researchers can build acteristics (e.g., social distance) correlate with safeguards into the initial experimental design the likelihood of subject treatment regime (Nickerson 2005). For instance, blocking on compliance (i.e., forward/return the letters), the network nodes that provide the treat- then inferences drawn about the nature of the ment can allow the researcher to excise prob- network will be biased (i.e., Milgram problematic participants without making arbitrary ably overestimated societal connectedness). decisions as to what parts of the network to The treatment could also be correlated with remove. the measurement of the network. Treatments A second problem that unpredictable may make certain relationships more salient behavior creates is that network experiments relative to other relationships, so the net- may be far more contingent and have less works measured in the treatment group are external validity. Suppose two people are not comparable to networks assigned to con- observed to have a strong relationship when trol or placebo conditions. The potential bias the network is initially defined. If these two introduced by these concerns suggests that people do not interact much during the researchers should measure the networks to course of the experiment itself, then they are 280 David W. Nickerson unlikely to pass the treatment along to each everyone in the community to map the spread other and the detected strength of the net- of the information. The Gold Shield Coffee work will be weak. If the waxing and waning experiment does not capture how company of interactions are random, then such differ- slogans diffuse through neighborhoods, but it ences will balance out across pairs of indi- does measure how information diffuses when viduals and the researcher will achieve an a plane drops a huge number of leaflets over unbiased estimate of the average network a very small town. characteristic to be measured. However, the A more common problem is the measure- waxing and waning could be a function of ment of baseline attitudes. Researchers often a range of systematic factors. For instance, worry about testing effects among subjects in experiments conducted on student networks pre- and post-test designs, but it is possible are likely to find dramatically different results that administering the pre-test changes the if the treatments are conducted at the begin- nature of the network. Subjects taking the ning, middle, or end of a semester. Political survey may be more likely to discuss the interest varies during and across elections, so topics covered in the survey than they would experiments on voting and social networks in the absence of the pre-test. Even if no may be highly contingent. Thus, external discussion is spurred by the pre-test, subjects validity is a large concern, and replication is may be primed to be especially attentive to an especially important aspect of advancing treatments related to the topics covered in the science of real-world networks. the pre-test. This increased sensitivity may The final practical hurdle facing re- compromise the external validity of such searchers conducting experiments on social experiments. Incorporating time lags networks is that special attention must be paid between pre-treatment measurements and to how the measurement of outcomes can the application of treatment can alleviate affect the network itself. A researcher may these concerns, as can creating pre-test want to see how inserting a piece of infor- measures that cover a wide range of topics. mation into a network alters beliefs, but the Related to the practical problems in insertion may also spur discussion in its own conducting experiments on social networks right. That is, the experiment could provide are the ethical problems. Setting aside an unbiased estimate of how the insertion of obviously unethical practices (e.g., forced the information affects the network, but can- resettlement), many practices common in not say how the existence of the information political science research are problematic in within the network alters beliefs. Early social the context of social networks. The reve- network experimenters were aware of this fact lation of attitudes about hot button issues and therefore conducted their research under (e.g., abortion, presidential approval), the the label propaganda.7 An extreme example of existence of sensitive topics (e.g., sexually this dynamic is the Gold Shield Coffee study transmitted diseases, financial distress, abor- (Dodd 1952), where randomly selected resi- tion), and holding socially undesirable views dents of a community were told the complete (e.g., racism, sexism, homophobia) could frac- Gold Shield Coffee slogan. The next day, a ture friendships and negatively affect com- plane dropped 30,000 leaflets on the town of munities and businesses. Selectively reveal- 300 households. The leaflets said that rep- ing information to subjects about neighbors resentatives from Gold Shield Coffee would can answer many interesting questions about give a free pound of coffee to anyone who social networks (e.g., Gerber, Green, and could complete the slogan and that one in Larimer 2008), but should only be practiced five households were already told the slogan. using publicly available information or after The following day, researchers interviewed achieving the explicit consent of subjects. Maintaining strict confidentially standards is 7 The fact that the military funded much of this research in order to understand the effectiveness of much more important when studying social propaganda techniques assisted this decision. networks than in atomistic survey conditions. Social Networks and Political Context 281

Even revealing the presence or absence of recycle that served as a placebo. The final network connections during a snowball sur- condition was a control group that received vey could affect relationships, so researchers no visit from researchers, but could verify that need to think carefully about the presentation the voter mobilization detected by the exper- of the study and how the assistants adminis- iment was genuine. The placebo condition tering the survey can assure absolute privacy. served to define the network. Voter turnout With these hurdles in mind, the three for the people answering the door in the vot- types of field network experiments are now ing condition would be compared to turnout discussed. among people answering the door in the recycling group. Similarly, turnout for the registered voter not answering the door could then External Shocks to the Network be compared across the voting and recycling The first experimental strategy for study- conditions. The degree to which the canvassing networks is for researchers to provide ing spilled over could then be estimated by an external shock to an existing network and comparing the indirect treatment effect (i.e., track the ripple (e.g., Miguel and Kremer cohabitants of people who opened the door) 2004; Nickerson 2008). The process involves to the direct treatment effect (i.e., for the introducing a change in a behavior or atti- people who opened the door). The design tude at one node of the network and then requires the assumption that subjects do not examining other points on the network for preferentially open the door for one of the the change as well. In principle, this strategy treatments and that only the people opening is not experimental per se and is like throwing the door are exposed to the treatment. a rock in a lake and measuring the waves. By Miguel and Kremer (2004) employed a throwing a large number of rocks into a large more common strategy in their study of a number of lakes, good inferences are possi- deworming program in Kenya by using an ble. Randomly sampling nodes in the network institution (schools) as the network node to only helps generalizability, much like random be treated. The order in which rural schools sampling does not make surveys experimen- received a deworming treatment was ran- 8 tal. To make the strategy truly experimen- domly determined. Miguel and Kremer then tal, multiple networks need to be examined compared health outcomes, school partici- simultaneously and the treatment then ran- pation, and school performance for pupils domly assigned to different networks. This at the treatment and control schools, find- random assignment allows the researcher to ing cost-effective gains in both school atten- account for outside events operating on the dance and health. The most interesting effect, networks (e.g., the news cycle) and processes however, came when the researchers looked working within the network (e.g., life cycle beyond the pupils in the experimental schools processes). However, remember that the unit to villages and schools not included in the of randomization is the network itself and not study. Untreated villages near treated schools the individuals within the networks. Thus, the also enjoyed health benefits and increased analysis should either be conducted at the net- school attendance, confirming that worms are work level or appropriately account for the a social disease. This strategy of relying on clustered nature of the treatment. institutional nodes of networks can be applied Nickerson (2008) provides an example of in a wide number of settings. The major hur- the strategy by looking for contagion in dle to employing the strategy is collecting voter turnout. Households containing two a sufficiently large number of networks or registered voters were randomly assigned to institutions to achieve precise and statistically one of three treatments. The first treat- meaningful results. ment involved face-to-face encouragement to vote in the upcoming election. The second 8 This strategy also avoids ethical concerns about deny- treatment was face-to-face encouragement to ing subjects treatment. 282 David W. Nickerson

Controlling the Flow of Communication (i.e., some volunteers treat everyone and oth- within a Network ers treat no one), but this comes at the cost of considerable statistical power. The diffi- A second strategy is to control the flow of culty in controlling communication in social communication within a neighborhood or networks makes this type of experiment diffi- network. The idea is to recruit participants cult to conduct in the field and thus probably who apply a treatment to randomly selected better suited for the laboratory. members of their social networks. Nicker- son (2007) provides an example where volunteers were recruited to encourage friends and Randomizing the Network Itself neighbors to vote in congressional elections. The final strategy randomizes the position Volunteers listed people in need of encour- of people within networks. The steps involve agement with whom volunteers would be measuring people’s opinions, attributes, or comfortable talking. The people listed were tendencies at time 0; assigning a place in a then randomly assigned to be approached network at time 1; measuring opinions at time (treatment) or not (control). The same design 2; and then modeling time 2 opinions for one principle has been applied to proprietary person as a function of opinions at time 0 of studies of campaign donations and the adop- both the subject and the others in the net- tion of consumer products. A major advan- work. Obviously, there are limited settings tage of the design is that the list provided where subjects can be randomly assigned to by the volunteers clearly defines the social places in social networks. The most common network to be examined. Because individu- use of this strategy has been to examine the als within networks are the unit of random- effect of roommates among college freshmen, ization, the design can also be much more looking at outcomes such as grades (Sacer- powerful than designs that randomize across dote 2001) or drug use and sexual behavior networks. (Boisjoly et al. 2006). Less common are ex- The biggest problem with controlling the periments where inmates are randomly as- flow of communication within a network signed security levels in prisons (Bench and is that the experimental interaction may be Allen 2003; Gaes and Camp 2009). These artificial and not approximate conversations experiments generally find that prisoners that occur organically. Neighbors, friends, assigned to more secure prisons are no more and coworkers rarely make explicitly politi- or less likely to commit crimes within prison, cal appeals to each other. Most hypothesized but that they are more likely to commit crimes mechanisms for the diffusion of norms and on release. The key to these empirical strate- peer effects are subtle and take time. It is pos- gies is establishing baseline characteristics sible that friends have a great deal of influ- prior to assignment to achieve identification. ence over each other but recoil from explicit Once the assignment is made and peers are prodding. Thus, such experiments measure residing together, outside forces could cause the effect of aggressive word-of-mouth cam- conformity independent of any peer effects, paigns within networks and not the workings thereby creating spurious relationships. of social networks in their natural state. The biggest problem with this research Inadvertent contamination is a serious strategy is that researchers rarely have the problem within social networks that needs power to randomly assign the residence of 9 to be considered. Volunteers may bring up subjects. The cases where random assign- the experiment in the course of everyday ment is practiced may not generalize to conversation, perhaps following such inno- more typical living conditions. The types and cent questions as “What’s new?” Subjects may cross-contaminate themselves by dis- 9 People randomly assigned to living quarters (e.g., cussing the unusual behavior of the volunteer prisoners, soldiers) typically have limited autonomy applying the treatment. These problems can and therefore enjoy additional human subjects’ pro- tections because of their compromised ability to offer be avoided by randomizing across networks consent. Social Networks and Political Context 283 intensities of interactions a person has in ing the estimand and designing treatment- dormitories or cell blocks may be qual- on-the-treated analysis play a very special role itatively different from interactions that in choice experiments. It is not always obvi- people typically have at work or in their ous how best to model the decision-making neighborhood. College students and prison- process, and researchers have more discre- ers are often young and may also be more tion than is typically found in the analysis of impressionable than older individuals. As a experiments. result, these types of studies can tell us a great Subject attrition is a special challenge for deal about the dynamics of these particular choice experiments in two ways. First, sub- networks, but how the results apply to other jects who take advantage of the voucher pro- settings is an open question. gram may opt to move out of the area where A step below randomizing the network researchers can easily track behavior. If out- itself is randomly providing the opportu- comes for subjects moving out of the area nity to opt in or out of a network (e.g., differ from outcomes achieved locally, then change neighborhoods or schools). The most the estimated treatment effect will be nec- famous of these experiments is the Moving to essarily biased because movement is inher- Opportunity (MTO) study (Katz, Kling, and ently correlated with the treatment. Second, Liebman 2001; Kling, Ludwig, and Katz subjects not enrolled in the experimental 2005; Kling, Liebman, and Katz 2007), where program (i.e., the control group) have little randomly selected residents of public hous- reason to comply with researcher’s requests ing were provided vouchers to move where for information and may be more likely to they see fit. The experiments then com- drop out of the study. This process could pared the outcomes of families receiving the result in a control group that is no long com- vouchers to those in the control group with parable to the treatment group. Both prob- no voucher. The MTO experiments found lems can be solved with sufficient resources that subjects electing to move felt safer and to acquire information and incentivize par- healthier, but they made few changes with ticipation, but researchers seeking to con- regard to criminal activity, employment, and duct choice experiments should take steps to educational attainment. The same type of address these two forms of attrition. experiment has been conducted related to schooling, where randomly selected families are provided vouchers to attend schools of 4. Conclusion their choosing (Howell and Peterson 2002). All experiments provide a complier aver- Social networks have been studied through- age treatment effect to some extent, but the out the history of social science, but new ana- dilemma is highlighted in these choice exper- lytic tools are providing fresh insights into iments. Many policy analysts want to know how people are tied together. Unsurpris- the effect of living in certain types of neigh- ingly, no single approach can lay claim to borhoods or attending particular schools on being preferred, and all methods have their the average person. However, choice experi- drawbacks. Although observational studies ments can only speak to how the move out of allow researchers to collect large amounts of one environment and into another affects the data and study the real-world relationships type of person who would move. Both the treat- of interest, they may be plagued by spurious ment and control group also contain people correlations that are impossible to eradicate. who would stay put given the opportunity, Laboratory experiments suffer from no omit- and the experiment is uninformative about ted variable bias and can randomly manip- these subjects. These nonmovers are revealed ulate the theoretically interesting aspects of in the treatment group, but not in the control social networks. The results in the labora- group, where randomization only assures that tory will generally be theoretically abstract the proportion of nonmovers is the same as and anonymous networks. The types of real- the treatment group. Thus, carefully defin- world networks to which the results apply is 284 David W. Nickerson an open empirical question that researchers Boisjoly, Johanne, Greg J. Duncan, Michael Kre- will need to answer. In theory, field experi- mer, Dan M. Levy, and Jacque Eccles. 2006. ments should combine the strengths of both “Empathy or Antipathy? The Impact of Diver- 96 1890 905 the observational and laboratory strategies, sity.” American Economic Review : – . Bond, Charles F., and Linda J. Titus. 1983. “Social but the reality is far messier. The cases where 241 field experiments can be applied to social net- Facilitation: A Meta-Analysis of Studies.” Psychological Bulletin 94: 265–92. works are necessarily limited, so the external Bond, Rod, and Peter B. Smith. 1996. “Culture validity of the findings is open to question. and Conformity: A Meta-Analysis of Studies The amount of control researchers have over Using Asch’s Line Judgment Task.” Psychologi- the network is also limited; thus, many theo- cal Bulletin 119: 111–37. retically and practically interesting questions Caldeira, Gregory A., and Samuel C. Patterson. will prove impossible to study. 1987. “Political Friendship in the Legislature.” As a result, a combination of the three Journal of Politics 4: 953–75. approaches is likely to prove the most fruitful. Campbell, Angus, Philip E. Converse, Warren E. 1964 As data become more ubiquitous and available Miller, and Donald E. Stokes. . The Amer- to researchers, observational studies will be ican Voter. Chicago: The University of Chicago able to address an increasing range of issues. Press. Carrington, Peter J., John Scott, and Stanley Just as lab experiments have helped guide the Wasserman. 2005. Models and Methods in Social theoretical development of game theory, lab- Network Analysis. New York: Cambridge Uni- oratory experiments on social networks will versity Press. answer increasingly complicated theoretical Christakis, Nicholas A., and James H. Fowler. questions about network density, information 2007. “The Spread of Obesity in a Large Social flow, strength of ties, and a host of other Network over 32 Years Background.” New Eng- factors. As randomized trials become more land Journal of Medicine 357: 370–79. accepted in a range of policy settings (e.g., Christakis, Nicholas A., and James H. Fowler. 2008 education, housing, legal enforcement, envi- . “The Collective Dynamics of Smoking in a Large Social Network Background.” New ronmental protection), the number of oppor- 358 2249 58 tunities to conduct field experiments on social England Journal of Medicine : – . Cohen-Cole, Ethan, and Jason M. Fletcher. 2008. networks will also increase. Little experi- “Detecting Implausible Social Network Effects mental work on networks has been done to in Acne, Height, and Headaches: Longitudinal date, but that leaves many fertile avenues for Analysis.” British Medical Journal 337:a2533. researchers. Dodd, Stuart C. 1952. “Testing Message Diffusion from Person to Person.” Public Opinion Quar- terly 16: 247–62. References Farrar, Cynthia, Donald P. Green, Jennifer E. Green, David W. Nickerson, and Stephen D. 2009 Achen, Christopher H. 1986. The Statistical Analy- Shewfelt. . “Does Discussion Group Com- sis of Quasi-Experiments. Berkeley: University of position Affect Policy Preferences? Results California Press. from Three Randomized Experiments.” Politi- 30 615 47 Asch, Solomon E. 1956. “Studies of Independence cal Psychology : – . and Conformity: A Minority of One against a Festinger, Leon, Stanley Schachter, and Kurt 1950 Unanimous Majority.” Psychological Monographs Back. . Social Pressures in Informal Groups: 70 (Whole no. 416). A Study of Human Factors in Housing. New York: Bench, Lawrence L., and Terry D. Allen. 2003. Harper & Row. “Investigating the Stigma of Prison Classifica- Fowler, James H., and Nicholas A. Christakis. 2008 tion: An Experimental Design.” Prison Journal . “Dynamic Spread of Happiness in a 83: 367–82. Large Social Network: Longitudinal Analysis 20 Berelson, Bernard R., Paul F. Lazarsfeld, and over Years in the Framingham Heart Study.” 337 2338 William N. McPhee. 1954. Voting: A Study British Medical Journal :a . 2009 of Opinion Formation in a Presidential Cam- Gaes, Gerald G., and Scott D. Camp. . “Unin- paign. Chicago: The University of Chicago tended Consequences: Experimental Evidence Press. for the Criminogenic Effect of Prison Security Social Networks and Political Context 285

Level Placement on Post-Release Recidivism.” Kling, Jeffrey R., Jens Ludwig, and Lawrence F. Journal of Experimental Criminology 5: 139– Katz. 2005. “Neighborhood Effects on Crime 62. for Female and Male Youth: Evidence from a Gerber, Alan S., Donald P. Green, and Christo- Randomized Housing Voucher Experiment.” pher W. Larimer. 2008. “Social Pressure and Quarterly Journal of Economics 120: 87–130. Voter Turnout: Evidence from a Large-Scale Kravitz, David A., and Barbara Martin. 1986. Field Experiment.” American Political Science “Ringelmann Rediscovered: The Original Arti- Review 102: 33–48. cle.” Journal of Personality and Social Psychology Haney, Craig, Curtis Banks, and Philip Zimbardo. 50: 936–41. 1973. “Study of Prisoners and Guards in a Lazarsfeld, Paul F., Bernard Berelson, and Hazel Simulated Prison.” Naval Research Reviews 9: Gaudet. 1948. The People’s Choice. New York: 1–17. Columbia University Press. Howell, William G., and Paul E. Peterson. 2002. Lazarsfeld, Paul F., and Robert K. Merton. 1954. The Education Gap: Vouchers and Urban Schools. “Friendship as a Social Process: A Substantive Washington, DC: Brookings Institution Press. and Methodological Analysis.” In Social Con- Huckfeldt, Robert. 1983. “The Social Context of trol, the Group, and the Individual, eds. Morroe Political Change: Durability, Volatility, and Berger, Theodore Abel, and Charles H. Page. Social Influence.” American Political Science New York: D. Van Nostrand, 18–66. Review 77: 929–44. Luskin, Robert C., James S. Fishkin, and Roger Huckfeldt, Robert, and John Sprague. 1995. Citi- Jowell. 2002. “Considered Opinions: Deliber- zens, Politics, and Social Communication: Informa- ative Polling in Britain.” British Journal of Polit- tion and Influence in an Election Campaign.New ical Science 32: 455–87. York: Cambridge University Press. Miguel, Edward, and Michael Kremer. 2004. Jennings, M. Kent, and Richard G. Niemi. 1974. “Worms: Identifying Impacts on Education The Political Character of Adolescence: The Influ- and Health in the Presence of Treatment ence of Families and Schools. Princeton, NJ: Externalities.” Econometrica 72: 159–217. Princeton University Press. Milgram, Stanley. 1967. “The Small-World Prob- Jennings, M. Kent, and Richard G. Niemi. 1981. lem.” Psychology Today 1: 61–67. Generations and Politics: A Panel Study of Young Mutz, Diana C. 1998. Impersonal Influence.New Adults and Their Parents. Princeton, NJ: Prince- York: Cambridge University Press. ton University Press. Nickerson, David W. 2005. “Scalable Protocols Karau, S. J., and Williams, K. D. 1993. “Social Offer Efficient Design for Field Experiments.” Loafing: A Meta-Analytic Review and Theo- Political Analysis 13: 233–52. retical Integration.” Journal of Personality and Nickerson, David W. 2007. “Don’t Talk to Social Psychology 65: 681–706. Strangers: Experimental Evidence of the Need Katz, Lawrence F., Jeffrey R. Kling, and Jeffrey for Targeting.” Paper presented at the annual B. Liebman. 2001. “Moving to Opportunity in meeting of the Midwest Political Science Asso- Boston: Early Results of a Randomized Hous- ciation, Chicago. ing Mobility Study.” Quarterly Journal of Eco- Nickerson, David W. 2008. “Is Voting Conta- nomics 116: 607–54. gious? Evidence from Two Field Experiments.” Kawachi, Ichiro, Graham A. Colditz, Alberto American Political Science Review 102: 49–57. Ascherio, Eric B. Rimm, Edward Giovannucci, Nordenmark, Mikael. 1999. “The Concentration Meir J. Stampfer, and Walter C. Willett. 1996. of Unemployment within Families and Social “A Prospective Study of Social Networks in Networks: A Question of Attitudes or Struc- Relation to Total Mortality and Cardiovascular tural Factors?” European Sociological Review 15: Disease in Men in the USA.” Journal of Epidemi- 49–59. ology and Community Health 1996: 245–51. Rosenbaum, Paul R. 2005. “Heterogeneity and Kearns, Michael, Siddharth Suri, and Nick Mont- Causality: Unit Heterogeneity and Design Sen- fort. 2006. “An Experimental Study of the Col- sitivity in Observational Studies.” American oring Problem on Human Subject Networks.” Statistician 59:147–52. Science 313: 824–27. Sacerdote, Bruce I. 2001. “Peer Effects with Ran- Kling, Jeffrey R., Jeffrey B. Liebman, and dom Assignment.” Quarterly Journal of Eco- Lawrence F. Katz. 2007. “Experimental Analy- nomics 116: 681–704. sis of Neighborhood Effects.” Econometrica 75: Sanders, Jimy, Victor Nee, and Scott Sernau. 2002. 83–119. “Asian Immigrants’ Reliance on Social Ties in 286 David W. Nickerson

a Multiethnic Labor Market.” Social Forces 81: Visser, Penny S., and Robert R. Mirabile. 2004. 281–314. “Attitudes in the Social Context: The Impact Sunstein, Cass, R. Sunstein, Reid Hastie, John W. of Social Network Composition on Individual- Payne, David A. Schkade, and W. Kip Vis- Level Attitude Strength.” Journal of Personality cusi. 2003. Punitive Damages: How Juries Decide. and Social Psychology 87: 779–95. Chicago: The Chicago University Press. Zuckerman, Alan S., Josip Dasovic, and Jen- Tucker, Peter. 1999. “Normative Inﬂuences in nifer Fitzgerald. 2007. Partisan Families: The Household Waste Recycling.” Journal of Envi- Social Logic of Bounded Partisanship in Germany ronmental Planning and Management 42: 63– and Britain. New York: Cambridge University 82. Press. Part VI

IDENTITY, ETHNICITY, AND POLITICS

CHAPTER 20

Candidate Gender and Experimental Political Science

Kathleen Dolan and Kira Sanbonmatsu

The largest literature on gender and experi- attitudes. Observational studies are also lim- mentation in political science concerns voter ited in helping us understand what we cannot reaction to candidate gender. One of the ear- observe, namely, why far fewer women than liest and most enduring questions in the study men seek office. If women fail to run because of gender and politics concerns women’s elec- they fear a gendered backlash from voters, tion to office. Because the number of women then we are unable to evaluate the experi- candidates and officeholders has increased ences of those women. in the United States over the past several The women who do run may be “a unique decades, there are more cases of women can- ‘survivor’ group” of candidates because of didates and officeholders available for empir- the recruitment processes that women have ical analysis. Today, women are a majority of had to overcome in order to become can- the electorate, and women candidates tend to didates (Sapiro 1981, 63). Attitudes toward win their races at rates similar to those of men. the women who run may not reflect the Yet, the gender gap in candidacy and office public’s response to “average” women can- holding remains large and stable. Under- didates, not unlike the sample selection prob- standing how voter beliefs about candidate lem that James Heckman (1978) identified gender shape attitudes and political behavior with regard to the study of women’s wages. remains an important area for research. Indeed, women candidates are not randomly Experimentation has helped scholars over- distributed across districts, states, or types come some of the limitations of using of elective offices because the gender-related observational studies to investigate candidate attitudes of voters and gatekeepers shape the gender. As Sapiro (1981) observed, public geographic pattern of where women emerge opinion surveys may not be able to detect as candidates and are successful. In addition, prejudice against women candidates if vot- when women run for office, they may antic- ers provide socially desirable responses. And ipate voter hostility to their gender and may if prejudice against women is subconscious, subsequently work to counteract any nega- then voters may not even be aware of their tive, gender-based effects in their campaigns.

289 290 Kathleen Dolan and Kira Sanbonmatsu

Thus, it may be difficult to observe the effects whether subjects reacted differently to the of gender in electoral politics due to selec- speech based on the gender of the speaker – tion effects and strategic decision making by was not revealed. In the first condition, stu- women candidates. For the same reasons, dents were informed that the candidate was experimentation can be particularly helpful “John Leeds,” and, in the second condition, in the study of race/ethnicity (see Chong and students were informed that the candidate Junn’s chapter in this volume). was “Joan Leeds.” Because Sapiro wanted to Isolating the effect of candidate gender in know how voters reacted to candidates in a observational studies is also difficult precisely low information context, the stimulus was an because gender contains so much informa- actual speech given by a U.S. senator that was tion. Voters may use candidate gender to ambiguous with respect to political party and infer a candidate’s personality traits, issue most policy issues. positions, party affiliation, ideology, issue With this design, Sapiro (1981) was able competence, occupation, family role, and to isolate the effect of candidate gender on qualifications. Also, the impact of candidate multiple voter inferences. Prior to the Sapiro gender may interact with other forces in the study, scholars usually relied on election political environment – candidate political results and public opinion surveys to gauge party, ideology, incumbency, prior experi- voter attitudes toward women. For example, ence – to create a situation in which candidate Darcy and Schramm’s (1977) analysis of con- gender does not influence voter attitudes and gressional election results found that female behaviors in the same way for every woman candidates were not at a disadvantage com- candidate. Indeed, it is precisely the complex- pared to male candidates, once the type of ity of the typical election environment that race was taken into account. They concluded makes it difficult for observational studies to that voters were indifferent to candidate gen- accurately capture the effect of gender. der. However, studies based on aggregate In this chapter, we describe several impor- election results do not take into account tant studies that capitalize on the benefits of the selection effects that produce successful experimentation to expand our understand- women candidates and cannot explain the low ing of whether voters hold gender stereo- numbers of women candidates. For example, types and whether voters are biased against a woman ran in only eight percent of the early women candidates. Experimentation has also 1970s general election races that Darcy and been used to understand how candidate gen- Schramm studied. Meanwhile, public opin- der interacts with factors such as party identi- ion surveys in the early 1970s continued to fication and type of elective office. At the end reveal bias against a hypothetical woman can- of this chapter, we suggest ways that schol- didate for president, although attitudes were ars could use experimental designs to answer becoming more liberal (Ferree 1974). remaining and new questions about gender Sapiro (1981) found no difference in sub- and politics in the future. jects’ willingness to support the female candidate. Nor was there a difference in how respondents evaluated the candidate’s under- 1. Gender Bias and Gender standing of policy issues, the clarity of the Stereotypes speech itself, the expected effects of the candidate’s proposals, or whether the subject Sapiro (1981) conducted one of the pioneer- agreed with the policy positions included in ing experiments in gender and politics. In the speech. Although Sapiro did not find evi- a simple design that followed the Goldberg dence of prejudice, she found a difference in (1968) experiment, Sapiro asked undergrad- perceptions of the likelihood that the candi- uate students in introductory political science date would win the race. A majority of respon- classes to read a speech by a fictional candi- dents believed that the male candidate would date for the U.S. House of Representatives. win compared to less than one half of respon- The purpose of the study – to understand dents in the female candidate condition. Such Candidate Gender and Experimental Political Science 291 doubts about women’s electability can put tions of stereotypically masculine issues, he women candidates at a disadvantage because concluded that the masculine nature of the voters, donors, interest groups, and political speech helped the female candidate over- parties may wonder if women candidates are come a traditional disadvantage. Meanwhile, worthy of investment. Leeper concluded that “voters may infer that Sapiro (1981) found that candidate gender tough, aggressive women still possess latent affected subjects’ evaluations of competence (stereotypical) warmth” (254). Voters rated on issues that were not specifically mentioned the female candidate as more competent on in the stimulus. The female candidate was female issues, such as education and main- rated as more likely to be competent on three taining honesty and integrity in government. areas typically associated with women (edu- The practical advice Leeper offered to female cation, health, and honesty/integrity in gov- candidates, therefore, was to pursue a “mas- ernment) and less competent on two areas culine” image without concern for presenting associated with men (military and farm), with a “female” side because voters will infer the no difference in other areas (environment feminine qualities. Consistent with Sapiro, he and crime). Thus, issue areas typically associ- found that subjects believed that the female ated with women in society potentially pro- candidate would be less likely to win. vide women with an advantage in the political Because of the strong evidence that vot- realm. ers hold stereotypes about issue competence, One drawback to the Sapiro (1981) study Huddy and Terkildsen (1993b) set out to is that subjects were only asked to evaluate determine the source of these political gen- one candidate and were not provided with der stereotypes. Until this point, there was the candidate’s party affiliation, making the relatively little attention given to the source experiment unlike a real-world election (see of the public’s gender stereotypes, and little McGraw’s chapter in this volume). Because nonexperimental work tried to identify these the study was conducted with a sample of sources. As the authors suggest, understand- undergraduates, the results might not hold ing the source of gender stereotypes can help in a general population. However, as with explain whether stereotypes are widely held other gender experiments, student samples and whether they can be overcome. To test make for more stringent tests of the gender their framework, they conducted an experi- bias hypothesis: it should be more difficult ment with 297 undergraduate students in the to observe gender effects among young vot- fall of 1990, in which subjects were asked ers because age is one of the most consistent to evaluate a single candidate. They manip- predictors of bias against women. ulated three between-subjects factors: the Subsequent studies – both observational sex of the candidate, whether the candidate and experimental – have confirmed the exis- was running for national or local office, and tence of gender stereotypes and the absence whether the candidate was described as pos- of explicit voter opposition to women can- sessing typically feminine or masculine per- 1 didates (e.g., Welch and Sigelman 1982; sonality characteristics. Subjects were also Rosenwasser and Seale 1988; Alexander and asked to judge whether the candidate could Andersen 1993). For example, Leeper (1991) be described as having a series of additional used a simple experimental design that traits beyond those mentioned in the descrip- varied candidate gender and asked under- tion. This allowed Huddy and Terkildsen graduate student subjects to evaluate a single to determine whether and to what degree candidate, as in the Sapiro (1981) study. people made gender-related inferences about Unlike Sapiro, however, Leeper sought to the candidates. Finally, subjects were asked investigate voter reaction to a “masculine” to indicate how well they believed that the woman candidate by creating a stimulus that candidate could handle military, economic, emphasized masculine themes and a “tough on crime” message. Because he found no 1 The office manipulation was analyzed in a separate effect of candidate gender on voter evalua- article (Huddy and Terkildsen 1993a). 292 Kathleen Dolan and Kira Sanbonmatsu compassion, and women’s issues and how sen (1993a) consider whether the impact of they evaluated the candidate’s party identi- stereotypes is conditional on the context of fication, ideology, and position on feminism. the offices women seek, specifically the level Huddy and Terkildsen’s (1993b) analysis and type of office. Women are more likely to tested two main hypotheses. The first was that hold lower-level offices, such as school board people’s stereotypes about the gender-linked and state legislative office, than higher-level personality traits of women and men (i.e., offices, such as statewide and congressional women are kind, men are aggressive) could office. At least as of yet, a woman has never lead people to assume gender-based com- been elected to the presidency or vice pres- petence in different areas (e.g., women are idency. Work by Rosenwasser et al. (1987) better at compassion issues, men at military found that college students rated a male can- issues). The second hypothesis was that the didate as more effective on “masculine” pres- political beliefs ascribed to women and men idential tasks than a female candidate, with may be the cause: the belief that women are the female candidate perceived to be more more liberal and Democratic than men could effective on “feminine” tasks. Furthermore, explain why women are perceived to be bet- Rosenwasser and Dean (1989) found that ter at handling compassion issues. In the end, students rated all political offices as more they found significant evidence that inferred masculine than feminine and that masculine traits were more important in determin- presidential tasks were deemed more impor- ing policy competence stereotypes than were tant than feminine presidential tasks. inferred beliefs about candidate partisanship Building on these studies, Huddy and and ideology. Interestingly, their manipula- Terkildsen (1993a) rightly note that most tion of gender-linked personality traits did work to that point had focused on the pres- not eliminate the effect of manipulated can- idency to the exclusion of other national didate sex on issue competency ratings. offices and state and local positions. Also, This study remains influential because it researchers had generally ignored whether demonstrates the importance of masculine the public saw women as better suited for cer- personality traits to evaluations of candi- tain types of elective office. Based on earlier date competence, including competence on findings on the public’s gender stereotypes, “women’s issues.” Their suggestion that they hypothesized that voters will take level women candidates create personas that and type of office into account when eval- emphasize their masculine traits has been uating women candidates. Specifically, they confirmed by more recent works (Walsh and expected that people would see women’s per- Sapiro 2003; Bystrom et al. 2004). Huddy sonality traits and policy competency as being and Terkildsen (1993b) also shed light on the better suited for local than national office and source of voters’ policy competence stereo- nonexecutive over executive positions. types, pointing to the role that perceived traits Huddy and Terkildsen (1993a) analyzed play in evaluations. Finally, their work moved data from the aforementioned study (Huddy the subfield forward at a time when nonex- and Terkildsen 1993b). For this article, how- perimental data on stereotypes were limited. ever, Huddy and Terkildsen (1993a) analyzed Their approach took advantage of the abil- the manipulation of candidate gender, candi- ity to manipulate the key variables of candidate gender-linked traits, and level and type date sex and gendered personality traits while of office. They began by providing subjects reducing social desirability concerns. with a list of nine masculine and seven feminine traits and asking people to evaluate the personality traits of a “good politician” run- 2. Type and Level of Elective Office ning for president, Congress, mayor, or local council member. This allowed Huddy and In another extension of our understanding Terkildsen to determine whether the pre- of the role of gender stereotypes in eval- ferred package of personality traits changed uations of candidates, Huddy and Terkild- with the level of office. As the authors Candidate Gender and Experimental Political Science 293 indicated, one of the strengths of the exper- no impact on the traits and policy strengths imental design they employed was the focus people value. on a “good” hypothetical candidate at differ- In addition, their analysis found no real ent levels of office. It would be quite difficult gender differences among subjects in the to isolate the impact of the office itself if the degree to which subjects valued male traits study consisted of voter evaluations of actual and male policy competence when evaluating candidates, with their myriad experiences and candidates for higher-level offices. But male political identities. subjects devalued feminine traits as impor- Their initial analysis provided support for tant to these offices and exhibited less willing- their hypothesis that good candidates for ness to say they would vote for the candidate national and executive office were expected with more feminine attributes than did female to hold more masculine characteristics than subjects. were candidates for legislative and local office. There were significant main effects for both level (between-subjects) and type (within- 3. Gendered Media Effects subjects) of office. The same general pattern held for people’s expectations about policy Another important area of investigation of competence; typical male policy issues such gender effects concerns media coverage. as military issues and the economy were con- Kahn (1994) used an experiment to deter- sidered more central to higher-level and exec- mine the effect of gender differences in news utive office. Compassion issues such as child coverage on candidate impression formation. care and welfare were seen as more central to Kahn’s content analysis of newspaper cov- legislative and local-level office. erage of twenty-six U.S. Senate races and Having confirmed that people hold gen- twenty-one gubernatorial races between 1984 dered expectations for different kinds of elec- and 1988 that featured women candidates tive office, Huddy and Terkildsen (1993a) revealed gendered patterns of news cover- then went on to determine whether can- age. She hypothesized that media coverage didates lose votes when they do not pos- patterns would vary across the two offices sess the “appropriate” characteristics for the because of the nature of the offices; for- office they seek. Again, their findings con- eign policy and national security issues that formed to expectation. Masculine traits were animate Senate politics are more likely to most important to candidates for national advantage male candidates, whereas statewide office, but offered little advantage to those issues such as health and education that are who sought local office. However, Huddy more likely to dominate the agendas of guber- and Terkildsen also found that typical fem- natorial candidates are expected to advantage inine traits did not offer an advantage to can- female candidates. didates for local office. Male policies, such Kahn’s (1994) content analysis of media as military and policing, were very impor- coverage revealed that women received more tant to candidates for national office, but horse race coverage than men and that the feminine, compassion issues offered no women senatorial candidates received more boost to candidates for local office. In general, negative viability assessments than men. then, they demonstrated that people have Women also received less issue coverage than a clear preference for masculine traits and men. Turning to an experiment, Kahn re- male policy competence when judging candi- created these gendered patterns of media cov- dates for national office and appear to consis- erage in order to measure their effects on tently devalue feminine traits and female pol- impression formation. In all, Kahn identi- icy competence, even when candidates seek fied fourteen dimensions of coverage that local office. The important influence Huddy she used to form her prototype articles. and Terkildsen identified is level of office, not To simulate the news coverage, she created type; their analysis found that the distinction four prototype articles (male/female incum- between executive and legislative office had bent and male/female challenger) for both 294 Kathleen Dolan and Kira Sanbonmatsu gubernatorial and senatorial races based on 4. The Intersection of the actual coverage. In two separate studies Gender and Party (one for Senate, one for governor), Kahn used a two-by-four factorial design that varied the Scholars of gender politics employing exper- four types of coverage and candidate gender. imental methods have successfully demon- Thus, Kahn was able to examine the impact strated that electoral context matters to public of “female” versus “male” press coverage on evaluations of women candidates. Yet, King both male and female candidates while also and Matland (2003), in a review of previous taking into account office type and incum- experimental work on public evaluations of bency. Her study remains a useful model for women candidates, suggest one major lim- gender scholarship because it used an obser- itation of experimental work on this topic: vational analysis in conjunction with experi- the isolation of candidate gender from other mentation. important political and social variables that Kahn’s (1994) results indicated that gen- might influence voter reactions to candidates. der differences in campaign coverage did If voter evaluations are strongly shaped by shape impression formation, although the a candidate’s (or their own) party identifica- effects were strongest for coverage of Sen- tion, then there may be no room for heuristics ate incumbents. Senate incumbent candi- such as gender to ultimately have significant dates with female coverage were less likely influence. Or, as they suggest, it may be the to be perceived as viable and less likely to be case that party cues interact with gender cues, considered strong leaders than Senate incum- which could result in women candidates of bents receiving male coverage. Female Sen- different political parties being evaluated dif- ate incumbent candidates were perceived to ferently. be more competent on health issues and King and Matland’s (2003) data came more compassionate. The analysis of guber- from an experiment embedded in a ran- natorial coverage revealed fewer gender dif- dom national telephone survey of 820 U.S. ferences than senatorial coverage, and the adults sponsored by the Republican Network gubernatorial experiment likewise produced to Elect Women (RENEW). Respondents fewer effects. Incumbent coverage differences received a description of a Republican can- in the experiment were limited to viability didate running for Congress. The candi- assessments, with the candidate in the female date’s gender was manipulated. After the brief incumbent gubernatorial condition consid- description, subjects were asked to evaluate ered to be less electable than the candidate the candidate on a number of traits and state who received male incumbent coverage. whether they would be likely to vote for this Kahn (1994) also found that, holding cov- candidate. Because of the sponsor of the sur- erage constant, women were perceived as vey, only reactions to a Republican candidate more compassionate and more honest than were evaluated. men, better able to maintain honesty and King and Matland’s (2003) goal was to test integrity in government, and more com- the power of gender cues relative to the power petent in women’s issues and the areas of of partisan cues on evaluation and willing- education and health. Women gubernatorial ness to vote for the candidate. They hypoth- candidates were perceived to be more esized that although gender cues would be knowledgeable than their male counterparts. relevant to evaluations of the traits of the can- Meanwhile, no differences were found on didate, the cue of party alone would predict stereotypical male issues (military, leadership, willingness to vote for the candidate. Instead, and the economy). Most of these effects were however, the authors found that gender and due to the differential evaluation of candi- party cues interacted. Voter party identifica- dates by female respondents. Finally, Kahn tion was indeed the strongest influence on found that the effects of gendered coverage willingness to vote for the candidates, but and candidate gender were cumulative. there was also a significant interaction with Candidate Gender and Experimental Political Science 295 candidate gender. Republican subjects in the Matland and King (2002) criticized past pool were more likely to say they would vote experiments on gender for their almost exclu- for the Republican man than the Republican sive use of college student subjects, arguing woman. The opposite was true for Demo- that college students have less well-developed cratic and independent subjects, who were political ideas and are less likely to participate 2 each more likely to vote for the Republican in politics than older people. candidate when she was a woman. Demo- cratic and independent subjects were no more likely to see the Republican as “conservative” 5. Future Directions when the candidate was presented as a man 1970 or a woman. Republican subjects, in contrast, Since the s, scholars of gender pol- perceived the Republican man as more “con- itics have grappled with the myriad ways servative” than the Republican woman. King that gender influences American political and Matland (2003) suggested that although life, using both observational and experi- Republican women pay a price with their own mental methods. However, as this chapter party members for their perceived greater lib- suggests, experimental work has been of par- eralism, they may reap a benefit from this ticular value to this endeavor, often offering stereotype among Democratic and indepen- advantages over observational methods. This dent voters. advantage can be seen in the foundational They found the same general pattern with work that we review here and in current 2008 evaluation of the traits of the candidates. research. For example, Streb et al. ( ) On each measure (the candidate “shares my employed a list experiment to tackle con- concerns,” “can be trusted,” “is a strong cerns about social desirability issues that can leader,” and “is qualified”), the Republican result from directly asking people whether woman candidate received significantly more they would support a woman for president. 2008 positive evaluations from Democratic and Winter ( ) manipulated media frames independent subjects than from Republican around issues of race and gender to deter- subjects. King and Matland (2003) conclude mine how these issues shape public opinion, that Republican women candidates may well which cannot easily be replicated through have to “make up” any votes they lose from observational work. Philpot and Walton 2007 their own party’s voters with crossover votes ( ) employed an experiment to gauge the from Democratic and independent voters. simultaneous impact of the intersection of However, this interaction of party and gen- race and sex on support for African Amer- der cues could hurt Republican women in ican women candidates. Fridkin, Kenney, 2009 primary contests. Their findings about the and Woodall ( ) manipulated media cam- primary election difficulties that the ideol- paigns to determine the impact of candidate ogy stereotype poses for Republican women gender on voter reaction to negative adver- are consistent with the conclusions of obser- tising. Experimentation can also be used to vational studies (Lawless and Pearson 2008; better understand the gender gap in public 2009 Sanbonmatsu and Dolan 2009). At the same opinion. For example, Lizotte ( ) used an experiment to analyze the gender gap in pub- time, King and Matland acknowledged that 3 the absence of a treatment for Democratic lic opinion on the use of force. We would candidates limited their ability to determine urge gender scholars to expand their use of whether the interaction of party and gender experimental methods because many ques- works the same way in each party. tions are ripe for experimentation. Future In addition to advancing knowledge about 2 Though see Druckman and Kam’s chapter in this gender stereotypes by introducing the role volume for another perspective on the use of student of party, King and Matland’s (2003) experi- samples. 3 See also Boudreau and Lupia’s chapter in this volume ment was conducted with a random national for a discussion of experimental work on the gender sample. gap in political knowledge. 296 Kathleen Dolan and Kira Sanbonmatsu research can continue to use experiments to women candidates to make gender an issue pursue the study of intersectionality along the in their campaigns (Bystrom et al. 2004; lines of the work by King and Matland (2003) Dittmar 2010). Experimental research could on party identification and by Philpot and help determine whether – and when – women Walton (2007) on candidate race. Pinpoint- candidates can benefit by making their gender 4 ing the interaction of gender with features identity an explicit campaign issue. of the electoral context remains an important Sarah Palin’s 2008 vice presidential cam- area for investigation. paign provides a different, but equally Several new lines of inquiry emerge from important, window into stereotypes. Her the 2008 presidential election. First, the sex- appearance on the national stage as a socially ism and misogyny evident in some of the conservative Republican may alter the dom- reaction that Hillary Clinton encountered inant stereotypes about political women, during the 2008 campaign was unexpected which are largely derived from Democratic given the thrust of the existing literature women. Future studies can probe the extent about the absence of bias against women to which Palin has reshaped voter assump- candidates, as well as the expectation that tions about the behavior and traits of women gender stereotypes will be attenuated in high- in electoral politics. information contexts (Carroll 2009; Lawless Finally, future experimental work on gen- 2009; Carroll and Dittmar 2010; Lawrence der could move beyond a reliance, as seen in and Rose 2010). Clinton’s experience led some early works, on college student popula- Freeze, Aldrich, and Wood (2009) to con- tions. Several recent works have successfully duct an experiment in order to understand the employed experiments embedded in surveys persuasiveness of messages about a candidate of nationally representative samples (King from a sexist source. The role that gender and Matland 2003; Streb et al. 2008;Frid- stereotypes played in voter and media reac- kin et al. 2009). For some analyses about the tion to Hillary Clinton suggests that stereo- effects of candidate gender, experiments con- types can play a role even in the presence ducted with a representative sample could sig- of substantial information about a candidate. nificantly strengthen existing findings. Much work remains to be done about how voters form impressions about candidates in high-information contexts. Clinton’s bid also 6. Conclusion calls into question the long-standing finding that voters will infer feminine traits in An impressive and growing body of gen- the absence of their explicit presentation in der politics research employing experimen- campaigns. To battle stereotypes about the tal methods points the way to future areas ability of a woman to serve as commander in of exploration. The realities of the political chief, Clinton may have portrayed an image world suggest that questions of the impact that was too masculine; voters seem to have of gender on candidates, voters, issues, cam- penalized her for her failure to appear more paigns, and media coverage will continue feminine. unabated into the future. The increasing Clinton’s experience also raises questions number of women candidates running for a about whether she would have fared bet- range of political offices and the candidacies ter had she emphasized the historic aspect of Hillary Clinton and Sarah Palin signal that of her entry into the race and her poten- there is still much to learn about how, when, tial to become the country’s first female and why gender is an important political president. Studies that seek to understand consideration. Increased reliance on the internal campaign decision making about experimental method can expand our under- candidate gender are few (Fox 1997;Dittmar standing of this critical influence. 2010). Yet, content analysis of the relationship between gender and political adver- 4 See Schneider (2007) for an experimental investiga- tisements reveals that it is uncommon for tion of women’s campaign strategies. Candidate Gender and Experimental Political Science 297

References Types of Office.” Political Research Quarterly 46: 503–25. Alexander, Deborah, and Kristi Andersen. 1993. Huddy, Leonie, and Nayda Terkildsen. 1993b. “Gender as a Factor in the Attribution of Lead- “Gender Stereotypes and the Perception of ership Traits.” Political Research Quarterly 46: Male and Female Candidates.” American Jour- 527–45. nal of Political Science 37: 119–47. Bystrom, Dianne, Mary Christine Banwart, Lynda Kahn, Kim Fridkin. 1994. “Does Gender Make Lee Kaid, and Terry Robertson. 2004. Gender a Difference? An Experimental Examination of and Candidate Communication: VideoStyle, Web- Sex Stereotypes and Press Patterns in Statewide Style, NewsStyle. New York: Routledge. Campaigns.” American Journal of Political Science Carroll, Susan. 2009. “Reflections on Gender and 38: 162–95. Hillary Clinton’s Presidential Campaign: The King, David, and Richard Matland. 2003. “Sex and Good, the Bad, and the Misogynic.” Politics & the Grand Old Party.” American Politics Research Gender 5: 1–20. 31: 595–612. Carroll, Susan, and Kelly Dittmar. 2010.“The Lawless, Jennifer. 2009. “Sexism and Gender Bias 2008 Candidacies of Hillary Clinton and in Election 2008: A More Complex Path for Sarah Palin: Cracking the ‘Highest, Hardest Women in Politics.” Politics & Gender 5: 70– Glass Ceiling.’” In Gender and Elections: Shap- 80. ing the Future of American Politics. 2nd ed., Lawless, Jennifer, and Kathryn Pearson. 2008. eds. Susan J. Carroll and Richard L. Fox. “The Primary Reason for Women’s Under- New York: Cambridge University Press, 44– representation? Reevaluating the Conventional 77. Wisdom.” Journal of Politics 70: 67–82. Darcy, Robert, and Susan Schramm. 1977. “When Lawrence, Regina G., and Melody Rose. 2010. Women Run against Men.” Public Opinion Hillary Clinton’s Race for the White House: Gen- Quarterly 41: 1–13. der Politics & the Media on the Campaign Trail. Dittmar, Kelly. 2010. “Inside the Campaign Boulder, CO: Lynne Rienner. Mind: Gender, Strategy, and Decision-Making Leeper, Mark. 1991. “The Impact of Prejudice in Statewide Races.” Paper presented at the on Female Candidates: An Experimental Look annual meeting of the Midwest Political Sci- at Voter Inference.” American Politics Quarterly ence Association, Chicago. 19: 248–61. Ferree, Myra M. 1974. “A Woman for President? Lizotte, Mary-Kate. 2009. The Dynamics and Ori- Changing Responses, 1958–1972.” Public Opin- gins of the Gender Gap in Support for Mili- ion Quarterly 38: 390–99. tary Interventions. Doctoral dissertation, Stony Fox, Richard Logan. 1997. Gender Dynamics in Brook University. Congressional Elections. Thousand Oaks, CA: Matland, Richard, and David King. 2002. Sage. “Women as Candidates in Congressional Elec- Freeze, Melanie S., John Aldrich, and Wendy tions.” In Women Transforming Congress,ed. Wood. 2009. “Candidate Evaluations, Nega- Cindy Simon Rosenthal. Norman: University tive Messages, and Source Bias.” Paper pre- of Oklahoma Press, 119–45. sented at the 32nd annual scientific meeting of Philpot, Tasha, and Haynes Walton. 2007. “One the International Society of Political Psychol- of Our Own: Black Female Candidates and the ogy, Dublin. Voters Who Support Them.” American Journal Fridkin, Kim, Patrick Kenney, and Gina of Political Science 51: 49–62. Serignese Woodall. 2009.“BadforMen,Better Rosenwasser, Shirley, and Norma Dean. 1989. for Women: The Impact of Stereotypes during “Gender Role and Political Office: Effects of Negative Campaigns.” Political Behavior 31: 53– Perceived Masculinity/Femininity of Candi- 78. date and Political Office.” Psychology of Women Goldberg, Philip. 1968. “Are Women Prejudiced Quarterly 13: 77–85. against Women?” Transaction 5: 28–30. Rosenwasser, Shirley, Robyn R. Rogers, Sheila Heckman, James. 1978. “Sample Selection Bias as Fling, Kayla Silver-Pickens, and John Bute- a Specification Error.” Econometrica 47: 153– meyer. 1987. “Attitudes towards Women and 161. Men in Politics: Perceived Male and Female Huddy, Leonie, and Nayda Terkildsen. 1993a. Candidate Competencies and Participant Per- “The Consequences of Gender Stereotypes for sonality Characteristics.” Political Psychology 8: Women Candidates at Different Levels and 191–200. 298 Kathleen Dolan and Kira Sanbonmatsu

Rosenwasser, Shirley, and Jana Seale. 1988. “Atti- ability Effects and Support for a Female Amer- tudes toward a Hypothetical Male or Female ican President.” Public Opinion Quarterly 2008: Presidential Candidate – A Research Note.” 76–89. Political Psychology 9: 591–98. Walsh, Katherine Cramer, and Virginia Sapiro. Sanbonmatsu, Kira, and Kathleen Dolan. 2009. 2003. “Marketing Congressional Candidates to “Do Gender Stereotypes Transcend Party?” Male and Female Audiences: The Performance Political Research Quarterly 62: 485–94. of Gender in Campaign Television Advertise- Sapiro, Virginia. 1981. “If U.S. Senator Baker ments.” Paper presented at the annual meeting Were a Woman: An Experimental Study of of the Midwest Political Science Association, Candidate Images.” Political Psychology 3: 61– Chicago. 83. Welch, Susan, and Lee Sigelman. 1982. “Changes Schneider, Monica Cecile. 2007. Gender Bending: in Public Attitudes toward Women in Politics.” Candidate Strategy and Voter Response in a Mar- Social Science Quarterly 63: 312–22. keting Age. Doctoral dissertation, University of Winter, Nicholas. 2008. Dangerous Frames: How Minnesota. Ideas about Race and Gender Shape Public Streb, Matthew, Barbara Burrell, Brian Frederick, Opinion. Chicago: The University of Chicago and Michael Genovese. 2008. “Social Desir- Press. CHAPTER 21

Racial Identity and Experimental Methodology

Darren Davis

Interest in how social group attachments sciousness, racial consciousness, linked fate, translate into political attitudes and behav- race identification).1 ior motivated much of the early attention to At the same time, however, research on social identity. According to Tajfel (1981), racial identity and political behavior could the underlying foundation in the develop- benefit from adherence to the conceptual ment of political and social beliefs was “the foundations of social identity theory, as well shared perceptions of social reality by large as from greater reliance on experimental numbers of people and of the conditions research. Several problems exist in the con- leading to these shared perceptions” (15), ceptual development and measurement of as opposed to personality and environmen- racial identity research that beg out for a reex- tal characteristics. Although Tajfel was not amination of racial identity. These problems referencing the forces that make one’s racial are closely tied to a heavy reliance on survey- identity relevant to an individual, but instead based research that essentially ignores the focused on the development of social iden- two-stage development of identity and thus tity around the prejudice toward Jews, racial overstates the importance of racial identity identity (at least in regard to political and in influencing political behavior. Controlling social behavior research) has been among the accessibility of racial identity among a the most powerful explanations of behavior. multitude of identities a person might pos- Despite the multitude of identities a per- sess, as well as the level of identity salience, son may possess and the events that make is critical to the study of racial identity. such identities more or less salient, social iden- Otherwise, racial identity may be viewed as tity theory has had special insight and significance into the connection between racial 1 Research has indeed focused on other identities, such identity and political behavior. In the field as patriotism, nationalism, gender, and social class. of political behavior, no form of identity has But research on these identities appears to lag behind research on racial identity. Equally important, social received nearly the amount of attention and identity theory has not been as readily applied to scrutiny as racial identity (e.g., group con- those identities.

299 300 Darren Davis somewhat artificial because researchers rior. Motivating individuals are the need for impose an identity and assert a certain level self-esteem and the desire for a positive self- of psychological importance. Such artificial- evaluation. ity of identity is compounded by the fact This process linking categorization and that racial identity is almost always measured self-esteem to group attachments is simple contemporaneously with other political and enough, but it should be clear that not all social attitudes, which makes causal state- identities are equally accessible and impor- ments tenuous. tant at the same time. Identities contribute In the sections that follow, I review the to our self-concept, but, for the most part, essence of social identity theory, explore the they need to be activated and made salient survey-based approaches to studying racial in order to be useful in political and social identity and political behavior, and then pro- decision making. For the positive distinctive- pose an experimentally based research agenda ness of group identities to become politically for overcoming such limitations. and socially important to the individual, a mechanism must exist for activating or making salient the psychological attachment to 1. Social Identity Theory social categories. Information and political and social events that increase the salience of Social identity theory was originally devel- different identities at different times abound. oped to explain the psychological basis of However, a different set of assumptions intergroup discrimination. The core idea is and processes seems to characterize the role that people tend to simplify the world around of racial identity among African Americans. them and that a particularly important sim- Without separating the components of social plifying device is the categorization of indi- identity theory within racial identity, the pri- viduals, including themselves, into groups macy of a racial identity among African Amer- according to their similarities and differences. icans is assumed to be a dominant identity Describing one’s self and others as African and, as a result, an African American racial American, conservative, a woman, and so identity is considered more easily activated forth is a way in which categories are cre- and sustained than other identities. This ated and maintained. People recognize at an might or might not be true, but it is a testable early age their differences and similarities to proposition, just the same. The safest assump- others.2 Such a simple and automatic classi- tion, and one that should guide methodolog- fication process is often assumed to be suf- ical approach in this area, is that racial iden- ficient to produce distinctive group behavior tity is highly variable and highly contextual and prejudice. Early on, scholars recognized among African Americans. A person might that the mere categorization or designation of think of him- or herself as African American group boundaries could provoke discrimina- and receive positive self-esteem from such a tion; however, Turner (1978, 138–39) would racial identity, but it is important to recognize later show that social categorization per se that the African American identity competes (the “minimal group” paradigm) was, by with other identities. It might come as a sur- itself, not sufficient for in-group favoritism. prise to some, but African Americans might Once people have categorized themselves also think of themselves as Americans, par- and others into distinct groups, self-esteem is ents, teachers, middle class, and so on. enhanced by creating favorable comparisons of their own groups vis-a-vis` other groups, thus making their own groups appear supe- 2. Racial Identity in Political Behavior Research 2 Categorization leads to the formation of stereotypes to aid in the processing of information, but the positive and negative attributions underlying discrimina- Most treatments of racial identity in polit- tion occur when individuals interact with others. ical behavior research, such as with racial Racial Identity and Experimental Methodology 301 consciousness, group consciousness, linked ical mistrust. A problem with this approach to fate, and racial identification, seem to focus racial identity is that, although African Amer- on a contrived or artificial identity and fail ican identity may be related to attributing to sufficiently capture the esteem that comes racial explanations, identity is not required from preferring one group over another or to to make such assessments. account for how one goes from identification Miller et al. (1981) define group conscious- to an embodiment of the group.3 The var- ness as a “politicized awareness, or ideol- ious elements of social identity theory exist ogy, regarding the group’s relative positions independently of each other in the literature. in society, and a commitment to collective Although racial or group consciousness could action aimed at realizing the group’s inter- be considered to capture the salience of racial ests” (495). This measure supposedly differs identity, linked fate, common fate, or group from group identification, which “connotes identity could also capture the categorization a perceived self-location within a particular of racial identity.4 social stratum, along with a psychological Consider the public opinion literature that feeling of belonging to that particular stra- seeks to connect racial and group conscious- tum.” Group consciousness is considered a ness to political behavior. Verba and Nie multidimensional concept integrating group (1972) recognize that racial consciousness identification, polar affect (i.e., a preference or the “self-conscious awareness of one’s for members of one’s in-group and a dislike group membership” among African Amer- for the out-group), polar power (i.e., dissatis- icans could be a potent force in political faction with the status of the in-group), and participation. The authors offer few details system blame (i.e., a belief that inequities in about the origins and activation of group the system are responsible for the status of the consciousness among African Americans, and in-group). Miller et al.’s conceptualization of their measurement of racial consciousness racial consciousness encompasses many of the is far removed from the essence of social consequences of identity, but it leaves unre- identity theory. Using responses to public solved how an individual decides for him- or opinion survey questions, they measure black herself which identities are relevant and how consciousness by whether blacks voluntarily group identity evolves from simple attach- raised the issue of race in response to a series ment to consciousness (or salience). Miller of open-ended questions asking about the et al. suggest that, through behavior and presence of any conflict within their commu- interactions, individuals learn of the discon- nities or any problems they perceived in their tent of one’s group position, which makes the personal lives, the community, or the nation. group salient or personally meaningful. Shingles (1981) subsequently repeated this Other prominent attempts to assess racial measure to conclude that black consciousness identity among African Americans have been is grounded in low political efficacy and polit- equally assertive in giving individuals an identity. African Americans are assumed to 3 By contrived, I mean that it is almost impossible possess a racial identity, and it is assumed to determine a priori the multitude of identities one may possess. But, in the construction of survey to take precedence over all other possible research, the researcher has to decide which iden- identities. No other identities or compet- tities to measure. Thus, these two processes seem ing attachments are considered. For instance, somewhat incompatible. 1989 4 Although one can argue that racial identity is dif- Gurin, Hatchett, and Jackson ( ) initially ferent from racial consciousness, group identity, or conceive of identity as a multidimensional linked fate, I see those concepts as tapping different construct with different behavioral conse- aspects of the same multicomponent of racial identity. They simply tap different dimensions of racial quences. Based on a common fate and an identity. Whereas group identity and racial identity exclusivist identity, racial identity reflected may be viewed as assessing the identity component of an implicit affiliation with the in-group. racial identity, group consciousness and racial consciousness may be viewed as assessing the salience Building off this measurement approach, component. Dawson (1994) intended his “linked fate” 302 Darren Davis to be a simpler construct of racial identity: interests, and feelings about things.” Once the as African Americans observe an attachment respondents finish rating how close they feel to other African Americans, they also come to all groups, they are asked to pick the one to believe that their interests, mostly eco- group to which they feel closest. nomic, are linked to the economic interests More recent research by Transue (2007) of their racial group (77). Unfortunately, this examining identity salience and superordi- measure seems to be driven more by avail- nate identity is also instructive. Using an able survey-based items than by an under- experiment embedded in a public opinion standing of social identity, as the mecha- survey, Transue examines the salience of mul- nism through which group affiliation or even tiple identities, both subgroup and superor- shared fate becomes salient is not explicit. dinate, on policy preferences. Identities are As a result, it may be premature to suggest primed through the random assignment of that affiliation automatically leads to linked respondents to two different question treat- fate (especially along an economic dimen- ments: one group is asked about their close- sion) and that it is always a salient evaluative ness to their ethnic or racial group (subgroup consideration. Within this same tradition, salience), and the other group is asked about Tate (1994) equates common fate to racial their closeness to other Americans (superor- identification. dinate group salience). Respondents are also Relating objective group membership to assigned to two different dependent variables, psychological attachment, Conover’s (1984) willingness to improve education (superordi- concept of group identification closely mir- nate treatment) and willingness to improve rors Tajfel’s (1981) treatment of identity: a educational opportunities for minorities (sub- self-awareness of one’s objective membership group treatment). in the group and a sense of attachment to It is clear from these studies that ra- the group. Beginning with an awareness of cial identity research reflects more of an their group affiliations, individuals’ salience afterthought than an intentionally designed or attachment to a group (perhaps from past research agenda. Racial identity has not been experiences or in response to political events) the focus of specialized attention, but rather becomes a component of their self-concept. it has been an idea superimposed on exist- Group identity, then, becomes a point of ref- ing data. As is often the case in these cir- erence in organizing and interpreting infor- cumstances, such an approach creates many mation and guides how individuals process problems. Because attitudinal measures occur information concerning others (Conover roughly at the same time in survey research, 1984, 763). In following Tajfel’s initial con- it is problematic to make causal statements ceptualization, Conover is able to show that about racial identity. Measures assumed to objective group membership acting in con- be influenced by racial identity could actually cert with a sense of psychological attachment prime racial identity. And, because survey- produces distinctive perceptual viewpoints. based approaches require questions ahead of This survey-based measure first determines time, racial identity or a set of identities respondents’ objective group membership are usually imposed on respondents, which from available survey-based measures, such as might or might not be how they view them- class, gender, age, and race. Although some- selves. In short, this imposed racial iden- what artificial, this objective measure does tity might well be viewed as contrived or consider a range of identities. Then, Conover artificial. determines whether respondents feel espe- Survey-based approaches are often not cially psychologically attached to the objec- conducive to studying racial identity. Exper- tive groups to which they presumably belong. imentally based methodology, in contrast, She accomplishes this by asking respondents can provide the control necessary to measure which groups they feel particularly close to – racial identity properly and to make convinc- people who are “most like you in their ideas, ing causal statements. Racial Identity and Experimental Methodology 303

3. Value of Experiments in the Study als would not be forced to respond to a priori of Racial Identity social groups with whom they might not have a strong attachment. Such an experimental My argument, so far, has been that the reli- feature would make it possible for subjects ance on, or dominance of, the survey re- themselves to identify their most salient social search enterprise in political behavior group. research has had a profound impact on the study of racial identity. Survey research is invaluable, but the approach to studying racial 4. Experimental Opportunity identity requires more attention. I now turn to how an experimental approach can pro- The argument that individuals should be duce more valid measures of racial identity, allowed to choose the identities that they which would permit stronger assessment of consider relevant and salient is grounded in the direction of causality. the political tolerance literature. Beginning Individuals belong to multiple groups and with the work by Stouffer (1955), politi- they possess multiple identities. In addi- cal tolerance was conceived as the willing- tion to racial groups, individuals may also ness to extend democratic rights (i.e., being identify with their gender, country, schools allowed to speak publicly, teach in pub- and universities, organizations and clubs, and lic schools, or publish books) to groups on occupations. The possible identities are too the political left (i.e., suspected communists, numerous to list, and doing so would be futile atheists, and socialists). As it turned out, because the most important groups are those Stouffer’s measure of tolerance assumed that that individuals select for themselves. It is certain groups in American society, particu- almost impossible to determine which of the larly those on the political right, would not identities are important for an individual’s be extended democratic rights. By not realiz- self-concept, but this has not prevented those ing that many individuals may not find such who study the connection between identity groups as threatening, the measure of tol- and political behavior from doing so. For erance would be contaminated by ideology. African Americans, a racial identity and racial Individuals on the political right would be consciousness are assumed to be the most mistaken as political. To correct this concep- prominent identity and the identity from tual and measurement issue, Sullivan, Piere- which they receive the most esteem. African son, and Marcus (1979) suggested that polit- Americans are seen as fixating on racial ical tolerance implies willingness to permit identity as a consequence of their history, the expression of those ideas or interests that culture, and perceptions of racism and dis- one opposes or finds objectionable. Following crimination. I am not suggesting that a this line of reasoning, Sullivan et al. proposed racial identity is irrelevant to a person’s self- a content-controlled measure of tolerance, concept, but I am questioning the common whereby individuals were allowed to iden- assumption that racial group identity is always tify functionally equivalent unpopular groups the most important. The reality is that racial they opposed. Operationally, individuals in a group identity is one of many identities. public opinion interview were provided with Experimental methodology seems more a list of groups on both sides of the political flexible than survey research in allowing a spectrum (i.e., atheists, proabortionists, Ku multiple identity approach. Similar to the Klux Klan members) from which they were salience approach, subjects can be presented to select the groups that they liked the least. with multiple identities that might conflict After the selection of their functionally equiv- or be incompatible. Subjects would then be alent groups, individuals were then presented expected to identify with their most salient with a series of statements about a range of and relevant identities. Because there would democratic activities in which members of be a choice among social groups, individu- that group might participate. 304 Darren Davis

Selected Identities Race Identity-1 Identity-2 Identity-3

Randomized level High High High High of salience Low Low Low Low

Example of testable hypotheses: > H1: RaceHigh RaceLow; = > H2: RaceHigh RaceLow 0; = = H0: RaceHigh RaceLow 0; > H3: RaceHigh Identity-3High; = > H4: RaceHigh Identity-3High 0; = H0: RaceHigh Identity-3High. Figure 21.1. Example of Experimental Design for Racial Identity

The take-away from Sullivan et al.’s (1979) same follow-up questions across the board) content-controlled measure is that function- would be problematic because each respon- ally equivalent groups are important con- dent would have each identity primed or siderations in comparing how individuals made salient in the same survey context and perceive groups. Instead of assigning or over a matter of seconds. Because such an assuming an identity based on some predeter- approach is taxing on the individual and each mined characteristic, such as race or gender, identity would be primed temporarily, this is individuals must be allowed to choose their not an ideal approach to assessing the role own identities. of identity. Actually, this approach would be An interesting question is how such an worse than imposing a single identity.6 approach would work for racial identity. For An interesting approach would entail ran- starters, it would be important for individuals domly assigning high salience and low to select groups with whom they closely iden- salience primes for each identity that an indi- tify (of course, without using the ambiguous vidual selects. In this way, each individual term “identify”). Borrowing from the racial receives only one primed identity (either high identity literature (Conover 1984), individu- or low), which can then be compared to als could be asked about the groups “they feel similar identities or compared to a similarly particularly close to and people who are most like primed alternate identity. Such an approach them in their ideas, interests, and feelings about would be a direct test of the salience of racial things.” Similar to the tolerance measure, identity over an alternate identity. Equally which was asked about four groups that indi- important, such an approach would be a direct viduals like least, this identity measure could test of racial identity against itself and at dif- also ask about the top four identity groups.5 ferent levels of salience. Next, for each group it would be necessary Consider the example depicted in Figure to determine the identity salience or psycho- 21.1, in which individuals are allowed to logical attachment. Assuming that racial iden- select a number of identities that they con- tity is among the selected identities, it would sider important (Race, Identity-1, Identity-2, be important to distinguish the salience of Identity-3) and a randomized assignment racial identity from the salience of other iden- of salience for each identity group (High, tities. Thus, priming identities by assigning 6 the same treatment to everyone (asking the The likelihood of individuals selecting the same identities is very low, but this approach requires only that individuals select a racial identity. Because the 5 Another way of measuring this ﬁrst part of iden- alternate identities would only be used for compar- tity could also involve linked fate or common fate ison, the actual content of those identities is not measures. important. Racial Identity and Experimental Methodology 305

Low).7 Individuals would be randomly of an experimental methodology is its capac- selected, and only one identity per person ity to make stronger claims about the causal randomly primed (although it would facilitate relationships of racial identity. matters if, across a certain number of individuals, a racial identity could be selected). It is often the case that we are interested in examining racial identity at different levels References of salience. The expectation is for high racial 1984 identity salience to be more powerful than Conover, Pamela Johnston. . “The Influence of Group Identifications on Political Percep- low racial identity salience in predicting 46 760 some form of political behavior. If there tion and Evaluation.” Journal of Politics : – 85. were no differences between them, then we Dawson, Michael C. 1994. Behind the Mule: Race could conclude that racial identity was unim- and Class in African-American Politics. Princeton, portant. Another important test involves the NJ: Princeton University Press. extent to which racial identity is more influ- Gurin, Patricia, Shirley Hatchett, and James S. ential than other identities. Thus, instead of Jackson. 1989. Hope and Independence: Blacks’ assuming that racial identity is more salient Response to Electoral and Party Politics. New York: than other identities, it could be tested Russell Sage Foundation. empirically. Miller, Arthur H., Patricia Gurin, Gerald Gurin, and Oksana Malanchuk. 1981. “Group Con- sciousness and Political Participation.” Amer- ican Journal of Political Science 25: 494– 5. Conclusion 511. Shingles, Richard D. 1981. “Black Conscious- ness and Political Participation: The Missing This chapter is about how survey-based 75 76 approaches can contribute to a flawed con- Link.” American Political Science Review : – 91. ceptualization of racial identity in politi- Stouffer, Samuel. 1955. Communism, Conformity, cal behavior research and how experimen- and Civil Liberties. New York: Doubleday. tal methodology might involve a better Sullivan, John L., James Piereson, and George E. approach. Perhaps the most serious problem Marcus. 1979. “An Alternative Conceptualiza- takes the form of imposing an artificial or tion of Political Tolerance: Illusory Increases contrived identity. Individuals possess a mul- 1950s–1970s.” American Political Science Review titude of identities that become more or 73: 781–84. less salient with information and in inter- Tajfel, Henri. 1981. Human Groups and Social actions with others. Instead of seeking to Categories. Cambridge: Cambridge University Press. capture a range of these identities among 1994 African Americans, there has been a tendency Tate, Katherine. . From Protest to Politics: The New Black Voters in American Elections.New for researchers to impose a racial identity, York: Russell Sage Foundation. regardless of whether such an identity is rel- Transue, John E. 2007. “Identity Salience, Identity evant to the individual. Acceptance, and Racial Policy Attitudes: Amer- Causal statements are made concerning ican National Identity as a Uniting Force.” racial identity when all attitudinal measures American Journal of Political Science 51: 78– are measured contemporaneously. The polit- 91. ical and social attitudes that racial identity has Turner, John C. 1978. “Social Categorization been expected to influence are just as likely to and Social Discrimination in the Minimal determine racial identity. The greatest value Group Paradigm.” In Differentiation between Social Groups: Studies in the Social Psychology of Intergroup Relations, ed. Henri Tajfel. London: 7 These identities can be any identities, as long as the Academic Press, 235–50. individual selects them. With the exception of a racial Verba, Sidney, and Norman H. Nie. 1972. Partic- identity, the alternative identities do not have to be identical across individuals. The interest is only in a ipation in America: Political Democracy and Social racial identity. Equality. New York: Harper & Row. CHAPTER 22

The Determinants and Political Consequences of Prejudice

Vincent L. Hutchings and Spencer Piston

Researchers have been interested in the dis- we define this term. What is prejudice? Social tribution of prejudice in the population, as psychologists were among the first to answer well as its effects on policy preferences, vote this question. For example, Allport (1979) choice, and economic and social outcomes defined (ethnic) prejudice as “an antipathy for racial minorities, since the dawn of the based upon a faulty and inflexible general- social sciences. One of the earliest scholars to ization. It may be felt or expressed. It may address these questions was W. E. B. DuBois be directed toward a group as a whole, or (1899) in his classic work, The Philadelphia toward an individual because he is a member Negro. Referring to prejudice in chapter 16 of of that group” (9). As we show, subsequent his book, DuBois wrote, “Everybody speaks scholars would expand and modify this defini- of the matter, everyone knows that it exists, tion in significant ways. However, the impor- but in just what form it shows itself or how tance of faulty generalizations or stereotypes influential it is few agree” (322). Although has remained a central component of virtually these words were written more than 100 years all definitions that would follow Allport. We ago, this observation still does a good job therefore rely on this broad definition unless of summarizing our understanding of prej- otherwise indicated. udice. There have been, to be sure, significant advances in this literature since the time of DuBois, but social scientists continue to 1. The Study of Prejudice in the disagree about the influence of racial preju- Political Science Literature dice in modern American politics. The aim of this chapter is to explore and adjudicate some The political science literature on prejudice1 of these differences, paying particular atten- has focused primarily on the role that tion to the strengths, limitations, and contri- prejudice plays in structuring policy pref- butions provided by the use of experimental erences and whether it influences candidate methods. Before examining the ways that scholars 1 We do not discuss implicit prejudice here (see Taber have studied prejudice, it is important that and Lodge’s chapter in this volume).

306 The Determinants and Political Consequences of Prejudice 307 preferences. The latter has centered on con- (1971) relied on an attitudinal scale composed tests involving two white candidates, as well of several different survey items. Respondents as elections between a white candidate and are asked to what extent they agree with an African American candidate. Each area the following statements, among others: 1) of study has focused almost exclusively on “Irish, Italian, Jewish and many other minori- white attitudes and has sought to determine ties overcame prejudice and worked their way whether prejudice remains a dominant, or up. Blacks should do the same”; and 2) “It’s at least significant, predictor of preferences really a matter of some people not trying hard in the post–Civil Rights era.2 As we discuss enough; if blacks would only try harder they in the next section, although observational could be just as well off as whites.” In general, work on each area has produced a number the symbolic racism scale has been shown to of contributions, it has often failed to isolate powerfully and consistently predict candidate the precise role that prejudice plays in public support and racial policy preferences.4 opinion, as well as the circumstances in which Although the symbolic racism scale has its influence is more or less powerful. In this frequently emerged as the most powerful cor- chapter, we highlight some of the ways in relate of racial policy preferences, a num- which experiments have helped address these ber of critics have challenged this construct questions. (Sniderman and Tetlock 1986; Sniderman and Piazza 1993; Sidanius and Pratto 1999). Their critiques take a variety of forms; how- Racial Policy Preferences ever, for our purposes, the most relevant Much of the early work on prejudice in the questions involve conceptual and measure- political science literature relied on observa- ment issues. Sniderman and his coauthors tional studies. Sears and Kinder (1971), for offer an alternative, although perhaps not example, used cross-sectional survey data to wholly incompatible, view of the role of race explore the impact of prejudicial attitudes on in modern American politics. This perspec- policy preferences and candidate support in tive focuses on the institutional or political Los Angeles. They would go on to develop forces that structure these views. In other the theory of symbolic racism, which main- words, they argue that the ways in which tains that a new and subtler form of racism politicians frame racial issues determine how emerged in the aftermath of the Civil Rights most Americans express their racial policy movement, spurred in part by the urban riots preferences.5 Thus, although it is true that of the late 1960s. Sears and Kinder argue Americans disagree on racial policy questions, that symbolic racism, unlike previous mani- this disagreement owes more to partisan or festations of antiblack prejudice, did not posit ideological differences than it does to racial the biological inferiority of African Ameri- attitudes per se (Sniderman and Carmines cans. Rather, the latent antipathy that many 1997). whites still felt toward blacks was now com- To support this alternative view, Sni- bined with the belief that blacks did not try derman and his colleagues rely heavily on hard enough to get ahead and violated tradi- parities in social and economic outcomes are due tional American values such as hard work and to blacks not trying hard enough, that blacks are individualism (Kinder and Sears 1981; Sears demanding too much too fast, and that blacks have 1988).3 To test this theory, Sears and Kinder gotten more than they deserve. 4 Some later indexes, such as the modern racism scale and the racial resentment scale, are designed to cap- 2 This focus on attitudes in the post–Civil Rights era ture similar or overlapping concepts (McConahay is in part a practical one because political scientists 1982; Kinder and Sanders 1996). To limit confu- showed little interest in questions of race or racial sion, we focus on the symbolic racism scale, but the bias prior to the late 1960s (Walton, Miller, and strengths and weaknesses associated with this theory McCormick 1995). can also be applied to its intellectual progeny. 3 More recently, Sears and Henry (2005)identified 5 This emphasis on the importance of framing effects four specific themes associated with the theory of can also be found in the work of Kinder and Sanders symbolic racism: the belief that racial discrimination (1996) and Nelson and Kinder (1996), as discussed against blacks has mostly disappeared, that racial dis- later in this chapter. 308 Vincent L. Hutchings and Spencer Piston question wording experiments embedded in sal terms (condition three) relative to race- national surveys. This approach has the specific, and racially justified, terms (condi- advantage of strong internal and exter- tion one). nal validity. In one study, Sniderman and Sniderman and Carmines (1997) acknowl- Carmines (1997) develop what they refer to edge that their finding, by itself, does not rule as the “Regardless of Race Experiment.” In out the possibility that it could be the result of this experiment, a random half of their sam- antiblack attitudes. However, they argue that ple is asked about their support for job train- if prejudice is driving the lower support for ing programs for blacks after being told that policies targeted at blacks, then whites who “some people believe that the government in are less committed to racial equality should Washington should be responsible for probe the most likely to embrace programs that viding job training to [blacks] because of the do not mention race. Although plausible, historic injustices blacks have suffered” (italics this is not what they find. White respon- added). The second half of the sample is also dents who are committed to racial equality asked about job training programs, but with are much more likely to support government the following rationale: “Some people believe assistance to the poor regardless of how the that the government in Washington should program is framed. More important, moving be responsible for providing job training to from racially targeted and racially justified [blacks] not because they are black, but because the characterizations to a more universal frame government ought to help people who are out of increases support for these programs at simi- work and want to find a job, whether they’re white lar rates for those scoring high or low on their or black.” Sniderman and Carmines expect the racial equality scale. second frame, owing to its more universal Kinder and Sanders (1996) employ a sim- character, to be much more popular among ilar experimental manipulation of question whites. As expected, they find that support wording in their examination of support for is higher for race-targeted job training pro- government efforts to assist blacks. In 1988, grams when they are justified in universal half of the American National Election Stud- terms rather than strictly racial ones (thirty- ies (ANES) sample was asked about efforts to four percent vs. twenty-one percent), consis- improve the social and economic position of tent with their broader argument. blacks (condition 1), whereas the other half In another experiment, dubbed the was asked about blacks and other minori- “Color-Blind Experiment,” Sniderman and ties (condition 2). As with Sniderman and Carmines (1997) test their argument about Carmines (1997), Kinder and Sanders (1996) the appeal of universal programs more find that support for this policy increases directly. With this experiment, white respon- by about seven percentage points in the lat- dents were divided into three groups and ter condition. They get larger effects in the asked whether the federal government should same direction among blacks, although these seek to improve conditions for “blacks who results fall short of statistical significance. are born into poverty ...because of the con- They conclude that “race neutral programs tinuing legacy of slavery and discrimination” do appear to be more popular among the or to “make sure that everyone has an equal American public, black and white” (184). opportunity to succeed.” The last group is Experiments such as these do provide some distinguished from the other two in that, conceptual clarity that is often missing when instead of asking about blacks, respondents researchers rely primarily on observational were asked about efforts to alleviate poverty studies. Instead of asking the reader to accept for “people” because, as in the second condi- a particular interpretation of an attitudinal tion, it is the government’s role to “make sure construct, Sniderman and Carmines (1997) that everyone has an equal opportunity to suc- simply manipulate the rationale behind the ceed.” They find that support for antipoverty policy (i.e., group specific or universal) or, programs increases by eighteen percentage in the cases of Sniderman and Carmines as points when the policy is described in univer- well as Kinder and Sanders (1996), whether The Determinants and Political Consequences of Prejudice 309 a policy refers specifically to African Amer- could only be attributed to the race-specific icans or is applied to some broader group. nature of the justification. They typically find diminished support for The Kinder and Sanders (1996) experi- the racially targeted program, which might ment is open to a somewhat different crit- be the result of prejudice. However, even icism. Although the authors conclude that when Sniderman and Carmines focus only on race-neutral programs are more popular, the racially targeted programs, as in the “Regard- inclusion of other minority groups in the less of Race Experiment,” they find that question (condition 2) arguably does not more universal justifications lead to greater diminish the role of race and is therefore support among whites. Similarly, with the not race neutral. It is clear that referencing “Color-Blind Experiment,” they and Kinder other minority groups increases the popular- and Sanders find that the role of prejudice ity of the program, but it does not neces- in prompting this greater support is minor sarily follow that the absence of any refer- because universal programs are more popu- ence to race would also increase popularity. lar with liberals, conservatives, and African The discussion of the results also implies that Americans. Still, even results derived from blacks and whites are more supportive of pro- an experiment can be open to interpretation. grams to assist blacks and other minorities for To draw an inference that differences across the same reasons (i.e., universal programs conditions are due to race and only race, the are more popular). It is possible, for exam- experimental conditions must differ on this ple, as various studies discussed later indicate, dimension and nothing else. When this is not that those whites who find broader programs the case, it is impossible to isolate the source more appealing adopt this view because of of the difference (or lack of difference) across their negative attitudes about blacks. African conditions. Americans, however, may be motivated by For instance, the “Regardless of Race a sense of solidarity with other nonwhite Experiment” purports to show that univer- groups and may or may not view truly race- sal justifications are more compelling than neutral programs more favorably. race-specific ones, but, in this case, the appro- Some studies have addressed these con- priate inference is uncertain. The two afore- cerns about precision in experimental de- mentioned conditions do not just differ in signs. Iyengar and Kinder (1987) provide terms of whether the rationale for the policy one example. They were interested in the is group specific. The justifications also differ impact of television news reports on the in terms of their emphasis on the past and in perceived importance of particular issues. their characterization of the unemployed as Instead of modifying survey questions, Iyen- eager “to find a job.” They may also differ gar and Kinder unobtrusively altered the con- in terms of their intrinsic persuasiveness in tent of network news accounts on various that a reference to “historic injustices” might national issues. In one experiment, subjects be too vague and, consequently, less convinc- viewed one of three stories about an increase ing than references to unemployed workers in the unemployment rate. The content was seeking to find a job. In short, we cannot be constant across conditions except that in the sure whether justifications framed in univer- two treatment conditions, the discussion of sal terms are inherently more persuasive or employment information was followed by an whether this particular race-specific rationale is interview with a specific unemployed indi- simply inferior to the specific universal rationale vidual. In one case, this person is white, and they employed. A more convincing race-specific in the other, the person is an African Amer- rationale might have asked about support for ican. Given that the only salient difference job training programs “because government across the two treatment conditions is the ought to help blacks who are out of work and race of the unemployed worker, lower lev- want to find a job.” In this example, the only els of concern with the unemployment prob- differences across conditions would be the use lem when the black worker is shown would of the term “black,” and so any difference represent evidence of racial prejudice. This 310 Vincent L. Hutchings and Spencer Piston is exactly what the authors find. Moreover, of individuals engaged in routine, exemplary, consistent with the prejudice hypothesis, they or scandalous activities. Specifically, eighty- find that whites who have negative views of four University of Michigan undergraduates blacks are most likely to diminish the impor- were randomly assigned to one of three con- tance of unemployment when the worker is ditions, where they either viewed photos of an African American. We should note, how- whites engaged in activities such as garden- ever, that these results are not entirely unas- ing (the control group) or pictures of blacks sailable because they achieve only borderline engaged in stereotypical activities such as ille- statistical significance, raising concerns about gal drug use, or counterstereotypical imagery the reliability of these findings. Further dis- of blacks interacting with their family or in cussion of this experiment can be found in school settings. If, as some have argued, neg- Iyengar’s chapter in this volume. ative attitudes about African Americans are Reyna et al. (2006) address some of the often dormant among whites, then exposure limitations identified in earlier experimental to frames that serve to remind them of these work on prejudice. These scholars rely on stereotypes might enhance the influence of a question wording experiment in the 1996 prejudice. Consistent with this view, Nelson General Social Survey. In this within-subjects and Kinder find that the relationship between experiment, respondents were presented with attitudes about blacks and support for racial a question about racial preferences with either preferences is significantly stronger for sub- blacks or women identified as the target jects exposed to stereotypical depictions of group. Reyna and her colleagues find that, African Americans. overall, respondents were significantly more Although the symbolic racism researchers supportive of this policy when women were have primarily relied on observational stud- the target group rather than African Amer- ies, they have recently called on experimen- icans, suggesting that not all group-specific tal methods to defend various elements of policies are created equal. Moreover, follow- the theory. One particularly persistent crit- ing up on the source of these effects, Reyna icism was that symbolic racism theorists had and her colleagues report that political con- never empirically demonstrated that this new servatism among college-educated respon- form of racism derives from a blend of dents was significantly related to greater antiblack affect and traditional American val- opposition to preferences for blacks. They ues. Sears and Henry (2003) sought to address report that these effects are mediated by this criticism using a split ballot design in “responsibility stereotypes,” which turn out the 1983 ANES Pilot Study. They adapted to be measured by one of the standard items in the six-item individualism scale so that each the symbolic racism index.6 Providing addi- item referred specifically to either blacks or tional confidence in their results, Reyna and women. Respondents received only one of her colleagues replicate their basic findings these scales, depending on which ballot they with a convenience sample of white adults were provided, but all participants received from the Chicago area (n = 184). the general individualism scale that made no Increasingly, researchers have begun to reference to race or gender. If individual- focus on the circumstances under which racial ism in the abstract is primarily responsible prejudice influences the policy positions of for white opposition to group-specific prefer- whites, rather than whether these attitudes ences, then the individualism scale adapted to play any role in structuring public opinion. In apply to women should be as strongly linked one of the earlier examples of this approach, to opposition to racial preferences as the indi- Nelson and Kinder (1996) had their sub- vidualism scale that was modified to apply to jects view and evaluate several photographs blacks. Similarly, the general individualism scale should also share the same predictive 6 This question reads as follows: “Irish, Italians, Jew- properties as the black individualism scale. If, ish, and many other minorities overcame prejudice and worked their way up. Blacks should do the same on the other hand, opposition to racial pref- without any special favors.” erences is primarily driven by individualistic The Determinants and Political Consequences of Prejudice 311 principles applied only to blacks, then one of Nonracial Policy Preferences the core assumptions of the symbolic racism theory would be sustained. Consistent with In addition to experiments designed to their theory, they find that only the black explore the role of prejudice in shaping atti- individualism scale is significantly correlated tudes on racial policies, scholars used this with opposition to racial preferences. tool to examine the influence of prejudice Although Sears and Henry (2003) resolve on ostensibly nonracial policies. Most of this one of the outstanding criticisms of the theory work focused on crime or welfare policy. of symbolic racism, it is still unclear whether For example, Gilliam and Iyengar (2000) the construct is confounded with political explore whether the race of criminal sus- ideology. Feldman and Huddy (2005)pro- pects influenced levels of support for puni- vide some support for this contention. These tive crime policies among viewers of local researchers relied on a question wording television news. Unlike most of the experi- experiment embedded in a representative mental work in political science, they include sample of white New York State residents whites and African Americans in their con- (n = 760). Respondents were asked whether venience sample (n = 2,331). Gilliam and they supported college scholarships for high- Iyengar present their subjects with one of achieving students. The treatment consisted three different (modified) newscasts: one fea- of manipulating whether the students were turing a black criminal suspect, one featuring described as white, black, poor white, poor a white suspect, and one in which there is no black, middle-class white, middle-class black, photograph or verbal description of the sus- middle class, or simply poor. The racial group pect. With the exception of the race of the categories alone do not produce any evidence alleged perpetrator, the newscasts are identi- of a double standard, but, once class was intro- cal in every way. As anticipated, they find that duced, Feldman and Huddy find evidence support for punitive crime policies increases of racial prejudice. Specifically, respondents by about six percentage points when sub- were much more supportive of college schol- jects view the black suspect, but the effects arships for white middle-class students (sixty- are much weaker and statistically insignifi- four percent) than they were for black middle- cant when the suspect is white or ambiguous. class students (forty-five percent). Also, they Interestingly, Gilliam and Iyengar find that find that the symbolic racism scale is asso- this only applies to the whites in their study ciated with this racial double standard, but because the effects are either insignificant or only for self-identified liberals. Conservatives run in the “wrong” direction for blacks. This scoring higher on the symbolic racism scale result provides additional support for their are more likely to oppose the scholarship proclaim that some form of racial prejudice con- gram for all groups, not just blacks.7 This tributes to white support for punitive crime suggests that the symbolic racism scale acts policies. In light of these findings, experimen- more like a measure of political ideology talists assessing the role of prejudice in shap- among conservatives. These results are not ing policy preferences should include minor- consistent with the theory of symbolic racism, ity subjects whenever possible (see Chong and but it is possible that Feldman and Huddy’s Junn’s chapter in this volume). results might differ if run on a national Peffley and Hurwitz (2007) engage in sample. a similar analysis, although they focus on the issue of support or opposition to the death penalty. Specifically, they are inter- 7 Feldman and Huddy (2005) are quick to point out ested in whether exposure to various argu- that, although the symbolic racism scale does not behave like a measure of prejudice for this group, ments against the death penalty would reduce conservatives are less supportive of the scholarship support among blacks and whites. In the program for blacks. Thus, they find evidence of a first of their two treatment groups, drawn racial double standard for this group, with opposition to college scholarships particularly low when the from a national telephone sample, Peffley and target group is middle-class blacks. Hurwitz present their respondents with the 312 Vincent L. Hutchings and Spencer Piston following preamble before asking their views icy. Gilens (1996) examined this issue using a on the death penalty: “some people say that question wording experiment embedded in the death penalty is unfair because too many a 1991 national survey. In addition, these innocent people are being executed.” Their results were supplemented with a mail-in second treatment group is presented with questionnaire delivered to the respondents a different introduction: “some people say who completed the telephone survey. The that the death penalty is unfair because most respondents in this study were assigned to people who are executed are African Amer- one of two conditions: in the first condition, icans.” Finally, respondents in the baseline they were asked their impressions of a hypo- condition are simply asked their views on thetical welfare recipient characterized as a this policy without any accompanying frame. thirty-something black woman with a ten- They report that both frames are persua- year-old child who began receiving welfare in sive among African Americans. In the inno- the past year. In the second condition, these cence condition, support for the death penalty attributes are identical except that the woman drops by about sixteen percentage points and in question is described as white. Gilens finds in the racial condition it drops by twelve that the white and African American wel- percentage points. Whites, in contrast, are fare recipients are evaluated similarly; how- entirely unaffected by the innocence treat- ever, in the black condition, negative attitudes ment, but, surprisingly, support for the death about welfare mothers are much more likely penalty increases by twelve percentage points to influence views of welfare policy. Gilens in the racial condition. Peffley and Hurwitz concludes that whites often have such nega- attribute this to a priming effect, wherein tive attitudes about welfare because their pro- individual attributions of black criminality totypical recipient is a black woman rather become much more predictive of support for than a white woman. the death penalty in the racial condition than in either of the other conditions. Experiments and Prejudice beyond the The finding that white support for the Black–White Divide in the United States death penalty increases when respondents are informed that blacks are more likely to Although most of the experimental work in be on death row is striking, but we should political science on the subject of prejudice interpret this result with some caution. For has focused on white attitudes about blacks starters, unlike many others, Peffley and Hur- and related policies, some recent work has witz (2007) do not find that antiblack stereo- also begun to focus on attitudes about Lati- types contribute to white support for the nos. The key policy domain in this liter- death penalty. In addition, their experiment ature is typically immigration. For exam- was designed to examine reactions to partic- ple, Brader, Valentino, and Suhay (2008) ular antideath penalty appeals, rather than to employed a two-by-two design manipulat- isolate the specific role of antiblack prejudice ing the ethnic focus of an immigration story in shaping death penalty views. If this latter (Mexican or Russian) as well as the tone (neg- aim were their goal, then perhaps they would ative or positive). Their subjects participated have designed conditions noting that “most over the Internet and were drawn from the people who are executed are men” or “most random digit dial–selected panel maintained of the people executed are poor.” Both state- by Knowledge Networks. The authors find ments are true, and if support for the death that opposition to immigration rises substan- penalty increased among whites in the race tially among whites when the negative story condition but not the others, then this would highlights immigration from Mexico and that represent strong evidence of a racial double respondent anxiety is the principal mediator standard. of this result. The other prominent, and ostensibly non- There is also an emerging literature in racial, policy domain that may be influ- political science that uses experiments to enced by antiblack prejudice is welfare pol- explore the influence of prejudice outside The Determinants and Political Consequences of Prejudice 313 the United States. This research must often to conditions in which two out-groups, immi- confront challenges that are not present in grants from Africa and immigrants from East- the American context, such as depressed eco- ern Europe, are evaluated along two dimen- nomic and social conditions, as well as mul- sions: the extent to which they have negative tiple national languages. Gibson (2009) man- personal characteristics, and the extent to ages to overcome these hurdles in his study which they are responsible for social prob- about the politics of land reconciliation in lems in Italy. Subjects are randomly assigned South Africa. In one experiment embedded to evaluate Eastern European immigrants on in a nationally representative sample, Gib- both dimensions, African immigrants on both son exposes black, white, colored, and Asian dimensions, or Eastern European immigrants respondents to one of several vignettes about on one dimension and African immigrants a conflict over land ownership. In each ver- on the other. The authors find that preju- sion of the vignette, one farmer claims that dice toward Eastern Europeans is as strong land currently occupied by another farmer a predictor of prejudice toward Africans as is was stolen from him and his family during prejudice toward Africans on another dimen- the apartheid era. There were multiple version. These results lend strong support to the sions of these vignettes, but the key manipula- authors’ argument that prejudice rests more tions involved the race of the farmer claiming in the eye of the beholder than in the charac- current ownership of the land and the judi- teristics of the out-group being evaluated. cial judgment as to who rightfully owned the property. In some cases, the dispute involves two black South Africans, and in other cases, 2. Prejudice and Candidate Choice the contemporary occupant is white, and the farmer making the historical claim is black. In The question of whether white voters dis- examining respondents’ views as to whether criminate against black candidates is still an the outcome was fair, Gibson finds that the open one. Observational work has been sug- race of the respondent as well as the race gestive, but it suffers from the inability to iso- of the claimants and the nature of the judg- late candidate race, leaving open the possibil- ment affected perceptions of fairness. White ity that confounding variables are driving the and black South Africans differed most sig- (lack of) results. We briefly review two of the nificantly. Whites were much more likely to best examples of observational work on this judge the outcome as fair if the contemporary question. As we show, their limitations yield owner of the land was awarded ownership an important opportunity for experimental and also described as white. Blacks, in con- work to make a contribution. We argue, how- trast, were considerably more likely to view ever, that experiments on racial discrimina- the outcome as fair if the farmer with the his- tion in the voting booth have not yet taken torical claim was awarded the land and if the full advantage of this opportunity. In partic- losing claimant was white. ular, imprecision in the experimental design The work of Sniderman et al. (2000) rep- has made it difficult to rule out the possibil- resents another intriguing experiment con- ity of alternative explanations. We therefore ducted outside the United States. The goal recommend that future work pay increased of the experiment was to determine whether attention to this issue, and we also argue that, expressed prejudice was more a function of given the mixed results, scholars should turn the attitude holder than the attitude object. from the question of whether white racism That is, rather than prejudice being “bound hurts black candidates to begin identifying up with the specific characteristics of the conditions under which prejudice hurts the out-group” (53), the authors hypothesize that chances of black candidates. prejudice is a function of an intolerant per- Highton (2004) examines U.S. House sonality; that is, intolerant people will express elections in 1996 and 1998. Using exit polls prejudice against any out-group. To test this conducted by the Voter News Service, he hypothesis, the authors assign Italian citizens measures discrimination as the difference 314 Vincent L. Hutchings and Spencer Piston between white support for white candidates failure to show up to the voting booth. Lack- and white support for black candidates, con- ing a measure of racial attitudes, the ability to trolling for such factors as incumbency, fund- assign candidate race, and control over such ing, experience, and demographic character- candidate characteristics as age, name recog- istics. Highton finds no difference between nition, ideological orientation, and personal- white support for white and black candidates ity, Highton cannot rule out the possibility of and on that basis determines that white voters confounding variables. showed no racial bias in these elections. Citrin, Green, and Sears (1990) exam- However, because Highton (2004) uses ine a case study of Democratic candidate exit poll data, he lacks a measure of racial atti- Tom Bradley’s loss to Republican candidate tudes. As a result, he cannot assess whether George Deukmejian in the 1982 guberna- prejudice is tied to vote choice. To be sure, torial contest in California. Unlike High- by itself this may not be much of a problem, as ton (2004), they have access to measures of Highton directly examines whether there is a racial attitudes, making use of data from polls racial double standard. But the process that conducted by the Los Angeles Times and the determines candidate race may be endoge- Field Institute that were conducted among nous to the vote choice decision. For example, a statewide sample of white Californians. consider the hypothetical case of black politi- Importantly, however, Citrin et al. do not cal figure A who decides to run for office. His simply measure the effect of racial attitudes personality is no more appealing than is the on vote choice because they recognize that norm for politicians, so he loses the primary racial attitudes are deeply implicated in pol- due to racial discrimination – that is, he is not icy attitudes. Racial attitudes might have an sufficiently exceptional to overcome the racial effect on voting, therefore, not due to the bias of some white voters. He therefore is candidate’s race, but because voters might not considered in Highton’s analysis because bring their racial attitudes to bear on their Highton counts that contest as one without a evaluations of the candidate’s policy plat- black candidate. Now consider the hypothet- form. Citrin et al. therefore pursue the clever ical case of black political figure B who has an strategy of comparing the influence of racial exceptionally appealing personality. He wins attitudes on vote choice for Bradley to the the primary because his outstanding person- influence of racial attitudes on vote choice for ality overwhelms the effect of prejudice. He other Democrats pursuing such state offices now counts as a black candidate in Highton’s as lieutenant governor. They find no addi- analysis. If this scenario is common, so that tional effect of prejudice on vote choice for only black candidates who are exceptional Bradley, and conclude that Bradley’s race did among politicians make it through the pri- not hurt him among white voters. mary and to the general House election, then As Citrin et al. (1990) recognize, their it is possible that discrimination does hurt attempt to control for all other relevant fac- black candidates in House elections, but that tors besides race is by necessity incomplete. such discrimination is not evident due to the Although other candidates for state office may effects of other candidate characteristics. If have shared membership in the Democratic Highton had a measure of racial prejudice and Party with Bradley, they surely did not share found it to be uncorrelated with vote choice, the exact same policy platform. Furthermore, then this might mitigate the aforementioned Bradley’s personality, experience, and name concern, but he does not. recognition were different. He was even run- Furthermore, Highton (2004) does not ning for a different office. We would have control for competitiveness of the contest; it increased confidence in the work of Citrin could be that white voters are voting for black and his colleagues if their controls for other candidates simply because they lack alterna- candidate factors were less crude than sim- tive viable options. Finally, he does not mea- ply having other white Democrats represent sure turnout, leaving open the possibility that the counterfactual white Bradley. Such is the white discrimination operates through the potential for experiments – to control for The Determinants and Political Consequences of Prejudice 315 factors that observational studies cannot in racial considerations may have been primed, order to be sure that any relationships (or causing subjects to bring their racial attitudes lack thereof) between candidate race and vote to bear on candidate evaluations when they choice are not an artifact of some other rela- might not otherwise have done so. Because tionship. Indeed, given that very few of the this characteristic of their experiment is arti- African American members of the House of ficial, it could be that most of the time Representatives hail from majority-white dis- the race of the candidate does not influence tricts, it seems plausible that greater con- their vote choice. Moskowitz and Stroh may trol over candidate characteristics might yield have identified that candidate race can matter a finding of antiblack discrimination among under certain conditions, but it is not cer- whites. tain that those conditions occur in the real Moskowitz and Stroh (1994), for exam- world. ple, presented their undergraduate experi- Terkildsen (1993) avoids some of the prob- mental subjects with a realistic editorial, cam- lems of artificiality in the design of her study. paign brochure, and photo, all describing First, she measures racial attitudes in the a hypothetical candidate (although subjects post-test and thus does not run the risk of were not told that the candidate was hypo- priming racial attitudes just before ask- thetical). Subjects read these materials and ing subjects to evaluate candidates. Second, then evaluated the candidate. Subjects were Terkildsen uses an adult convenience sam- randomly assigned to one of two groups ple selected for jury duty, decreasing gen- wherein the descriptions were equivalent in eralizability concerns somewhat. However, all respects except one – the race of the can- Terkildsen’s analysis also shares a limitation didate. Because the race of the candidate with Moskowitz and Stroh’s (1994). Unlike was the only thing that varied, Moskowitz in an actual election, in which voters typ- and Stroh can be more confident than can ically choose between candidates, subjects Highton (2004) or Citrin and his colleagues were asked to evaluate the candidate in isola- (1990) that any difference between groups tion. This is a particularly important limita- was a result of candidate race and thus has the tion, given that previous work suggests that potential to demonstrate stronger evidence of stereotypes may work differently when can- a racial double standard. Experimental con- didates are evaluated alone instead of in com- trol also gives Moskowitz and Stroh the abil- parison to each other (Riggle et al. 1998). ity to assess the mechanism through which Sigelman et al. (1995), in contrast, do ask prejudice affects the vote choice because they subjects to choose between two candidates, measure subjects’ perceptions of the candi- one of whom is black in one condition but date’s policy positions and personality char- white in the other. Using a convenience sam- acteristics. Indeed, Moskowitz and Stroh find ple composed of adults selected for jury duty that prejudiced white subjects discriminate in the Tucson, Arizona area, the authors against black candidates, and that they do so find no evidence of race-based discrimina- by attributing to black candidates unfavor- tion. The authors use a nine-cell design, and able character traits and policy positions with candidate A is identical across cells: a conser- which they disagree. vative candidate whose race is unidentified. Although Moskowitz and Stroh’s (1994) Candidate B varies by ideology (conservative, work overcomes some of the problems moderate, or liberal) and by race (described inherent in observational studies, they also by the experimenters as “Anglo,” “black,” or encounter a unique set of limitations. For “Hispanic”). example, the experimental context differed Sigelman and her colleagues (1995)also from an actual campaign environment in one claim to find evidence of an interaction effect potentially devastating way. Subjects were between race and ideology, in which racial asked questions about their racial attitudes minorities are perceived as more liberal than just prior to being presented with material are white candidates. The authors manipu- about the hypothetical candidates. As a result, late ideology by changing the content of each 316 Vincent L. Hutchings and Spencer Piston candidate’s speech; subjects are expected to Reeves (1997) claims to find evidence of infer the ideology of the candidates by read- racial discrimination when the campaign issue ing their speeches. Unfortunately, the con- is affirmative action, but what is actually evi- tent of the speech varies not only in the dent in the data is an increase in the number ideological principles espoused, but also in of white subjects who claim to be undecided. the highlighting of racial issues. Whereas the To be sure, Reeves finds that the distribution conservative candidate argues that minori- of racial attitudes among these respondents ties “have become too dependent on govern- suggests that they would probably not sup- ment” and the liberal candidate claims that port a black candidate, but this argument is minorities “have been victims of terrible dis- only suggestive. crimination in this country,” the moderate A somewhat more recent vintage of stud- candidate does not mention race at all. As ies on the influence of racial prejudice in a result, it is difficult to tell whether sub- candidate selection focused more on indi- jects evaluating the ideology of a given candi- rect effects. With these studies, the empha- date were reacting to the candidate’s race, the sis is on contests featuring two white candi- ideological principles espoused in the candidates and the prospect that subtle racial cues date’s speech, the extent to which racial issues are employed to the disadvantage of one of were highlighted in the speech, or, quite plau- the candidates. According to this literature, sibly, interactions among the three. political candidates in the post–Civil Rights The experimental design employed by era no longer make direct racial appeals to Reeves (1997) overcomes many of these prob- whites because such efforts would be repudi- lems. For example, to avoid potential unin- ated by voters across the political spectrum. tended priming effects, racial attitudes are Instead, covert references are made to race, measured six months prior to the imple- leading whites to bring their latent antiblack mentation of the experiment. Furthermore, attitudes to bear on candidate preferences. respondents, identified in a representative This process has been dubbed “racial prim- mail survey of Detroit area residents, eval- ing” (Mendelberg 2001). uate the candidates in a comparative context. Whether political campaigns devise subtle Evaluations are based on realistic newspaper racial cues in order to surreptitiously activate articles describing a debate between two ficti- the racial views of the electorate is a difficult tious candidates. Finally, in a four-cell design, issue to study. When voters bring their racial Reeves manipulates the race of one of the attitudes to bear on some voting decisions candidates (black or white) and the issue area rather than others, we cannot be sure using being debated (the environment or affirma- observational studies whether this occurred tive action). Thus, his design allows him to because of specific campaign tactics or some directly examine the impact of highlighting other unrelated event. Experimental manipu- racial issues. lation provides perhaps the only way to con- Unfortunately, Reeves (1997) only ana- fidently evaluate this possibility, but, as we lyzes those subjects who choose to respond illustrate, even here many questions remain to his questionnaire, neglecting to consider unanswered. the possibility that the decision to respond is Mendelberg (2001) was among the first to endogenous to the effect of the treatments. examine this question by developing a series In his experiment, unlike with most survey of experiments manipulating whether a fic- experiments, subjects receive a questionnaire titious gubernatorial candidate’s antiwelfare by mail that they can read before determin- appeal was racially implicit (i.e., visual ref- ing whether they want to mail it back to erences to race but not verbal), explicit (i.e., the experimenter. Exposure to the treatment, visual and verbal references to African Amer- therefore, may have some impact on the deci- icans), or counterstereotypical (i.e., antiwhite sion to participate in the study, but Reeves rather than antiblack). Her subjects are drawn does not analyze whether response rates vary from a random sample of New Jersey house- across experimental conditions. holds. In addition to manipulating racial cues, The Determinants and Political Consequences of Prejudice 317

Mendelberg also manipulated whether par- considerable time prior to the treatment in ticipants were told that their views con- order to avoid this potential problem. formed with or violated societal norms on race. She finds that concern with violating norms prevents explicit messages from acti- 3. Conclusion vating antiblack attitudes with one caveat: racially liberal subjects who are unconcerned In sum, although experiments have con- when told that their views violate the norm tributed to our knowledge of the role that are much more likely to support the racially prejudice may play in shaping policy pref- conservative candidate. This result suggests erences and candidate support, a number of that further research needs to be done to questions remain unanswered. Experiments explore exactly how concern for norms mod- clearly do provide advantages, as they address erates the effects of racial priming. some of the weaknesses of observational stud- Despite this support for the racial prim- ies. The main weakness of observational ing hypothesis, confirmed and replicated by research is its inability to control for all pos- 2002 Valentino, Hutchings, and White ( ), a sible confounds, and the main strength of debate has emerged in the literature regard- experiments is their ability to do just that. ing the influence of implicit and explicit racial Experiments, however, are not a panacea and appeals. Employing an experimental design consequently bring their own set of limita- 2001 similar to Mendelberg’s ( ), but with a tions. The limitations of the work reviewed nationally representative sample treated over here include (at least occasionally) an over- = 6 300 the Internet (n , ), Huber and Lap- reliance on convenience samples that conse- 2006 inski ( ) find that implicit appeals are not quently undermine external validity, lack of more effective than explicit messages in prim- realism in the treatments, and imprecision 2008 ing racial attitudes. In Mendelberg’s ( ) in the experimental design, thereby clouding response, she questions whether the treat- the inferences that can be drawn. Impreci- ment was delivered successfully and argues sion in the experimental design is especially that, because subjects’ racial predispositions troubling, given that the main advantage of were measured just prior to exposure to the experiments is the ability of the experimenter experimental treatments, any differences in to eliminate alternative potential confounds. priming effects may have been neutralized. Some of these concerns will always be difficult 2008 Huber and Lapinski ( ) reject these crit- to address, as they represent inherent prob- icisms. More important for our purposes, lems with the use of experiments in the social however, is that the racial priming literature sciences. However, by devoting our attention is vulnerable to the charge that measuring to improvement in design, we can minimize racial attitudes immediately prior to or fol- some of the most glaring weaknesses of this lowing the treatment may affect the results. valuable tool. When the measurement occurs prior to treatment, researchers run the risk of dampen- ing any priming effect due to “contamination” of the control group. However, when References measured after the treatment, the distribu- 1979 tion of racial attitudes may be influenced by Allport, Gordon W. . The Nature of Preju- the manipulation (Linz 2009). Thus, it is not dice: 25th Anniversary Edition. New York: Basic so much that liberals and conservatives are Books. Brader, Ted, Nicholas A. Valentino, and Elizabeth sorting themselves out more appropriately Suhay. 2008. “What Triggers Public Opposi- as a consequence of exposure to an implicit tion to Immigration? Anxiety, Group Cues, and racial appeal. Rather, the candidate prefer- Immigration Threat.” American Journal of Polit- ences may remain firm and the racial attitudes ical Science 52: 959–78. may be changing. Future work in this area Citrin, Jack, Donald Philip Green, and David should try to measure racial attitudes some O. Sears. 1990. “White Reactions to Black 318 Vincent L. Hutchings and Spencer Piston

Candidates: When Does Race Matter?” Pub- Mendelberg, Tali. 2008. “Racial Priming Re- lic Opinion Quarterly 54: 74–96. vised.” Perspectives on Politics 6: 109–23. DuBois, William Edward Burhardt. 1899. The Moskowitz, David, and Patrick Stroh. 1994. “Psy- Philadelphia Negro. Philadelphia: University of chological Sources of Electoral Racism.” Polit- Pennsylvania Press. ical Psychology 15: 307–29. Feldman, Stanley, and Leonie Huddy. 2005. Nelson, Thomas E., and Donald R. Kinder. 1996. “Racial Resentment and White Opposition to “Issue Frames and Group-Centrism in Amer- Race Conscious Programs: Principles of Prej- ican Public Opinion.” Journal of Politics 58: udice?” American Journal of Political Science 49: 1055–78. 168–83. Peffley, Mark, and Jon Hurwitz. 2007. “Persuasion Gibson, James L. 2009. Overcoming Historical Injus- and Resistance: Race and the Death Penalty in tices: Land Reconciliation in South Africa.New America.” American Journal of Political Science York: Cambridge University Press. 51: 996–1012. Gilens, Martin. 1996. “‘Race Coding’ and Oppo- Reeves, Keith. 1997. Voting Hopes or Fears? sition to Welfare.” American Political Science White Voters, Black Candidates, and Racial Pol- Review 90: 593–604. itics in America. New York: Oxford University Gilliam, Franklin D., and Shanto Iyengar. 2000. Press. “Prime Suspects: The Influence of Local Tele- Reyna, Christine, P. J. Henry, William Korf- vision News on the Viewing Public.” American macher, and Amanda Tucker. 2006. “Examin- Political Science Review 44: 560–73. ing the Principles in Principled Conservatism: Highton, Benjamin. 2004. “White Voters and The Role of Responsibility Stereotypes as Cues African American Candidates for Congress.” for Deservingness in Racial Policy Decisions.” Political Behavior 26: 1–25. Journal of Personality and Social Psychology 90: Huber, Greg A., and John Lapinski. 2006.“The 109–28. ‘Race Card’ Revisited: Assessing Racial Prim- Riggle, Ellen D., Victor C. Ottati, Robert S. Wyer, ing in Policy Contests.” American Journal of James Kuklinski, and Norbert Schwarz. 1998. Political Science 48: 375–401. “Bases of Political Judgments: The Role of Huber, Greg A., and John Lapinski. 2008.“Test- Stereotypic and Nonstereotypic Information.” ing the Implicit–Explicit Model of Racialized Political Behavior 14: 67–87. Political Communication.” Perspectives on Poli- Sears, David O. 1988. “Symbolic Racism.” In Elim- tics 6: 125–34. inating Racism: Profiles in Controversy,eds.Phyl- Iyengar, Shanto, and Donald R. Kinder. 1987. lis A. Katz and Dalmas A. Taylor. New York: News That Matters. Chicago: The University Plenum Press, 53–84. of Chicago Press. Sears, David O., and P. J. Henry. 2003.“TheOri- Kinder, Donald R., and Lynn M. Sanders. 1996. gins of Symbolic Racism.” Journal of Personality Divided by Color: Racial Politics and Democratic and Social Psychology 85: 259–75. Ideals. Chicago: The University of Chicago Sears, David O., and P. J. Henry. 2005. “Over Press. Thirty Years Later: A Contemporary Look at Kinder, Donald R., and David O. Sears. 1981. Symbolic Racism and Its Critics.” In Advances in “Prejudice and Politics: Symbolic Racism ver- Experimental Social Psychology, ed. Mark Zanna. sus Racial Threats to the Good Life.” Journal New York: Academic Press, 95–150. of Personality and Social Psychology 40: 414–31. Sears, David O., and Donald R. Kinder. 1971. Linz, Gabriel. 2009. “Learning and Opinion “Racial Tensions and Voting in Los Ange- Change, Not Priming: Reconsidering the Evi- les.” In Los Angeles: Viability and Prospects for dence for the Priming Hypothesis.” American Metropolitan Leadership, ed. Werner Z. Hirsch. Journal of Political Science 53: 821–37. New York: Praeger, 51–88. McConahay, J.B. 1982. “Self-Interest versus Sidanius, Jim, and Felicia Pratto. 1999. Social Dom- Racial Attitudes as Correlates of Anti-Busing inance: An Intergroup Theory of Social Hierarchy Attitudes in Louisville: Is It the Buses or the and Oppression. New York: Cambridge Univer- Blacks? Journal of Politics 44: 692–720. sity Press. Mendelberg, Tali. 2001. The Race Card: Cam- Sigelman, Carol K., Lee Sigelman, Barbara J. paign Strategy, Implicit Messages, and the Norm Walkosz, and Michael Nitz. 1995. “Black Can- of Equality. Princeton, NJ: Princeton Univer- didates, White Voters.” American Journal of sity Press. Political Science 39: 243–65. The Determinants and Political Consequences of Prejudice 319

Sniderman, Paul M., and Edward G. Carmines. Implications of Candidate Skin Color, Preju- 1997. Reaching beyond Race. Cambridge, MA: dice, and Self-Monitoring.” American Journal Harvard University Press. of Political Science 37: 1032–53. Sniderman, Paul M., Pierangelo Peri, Rui J. P. de Valentino, Nicholas A., Vincent L. Hutchings, Figueiredo, Jr., and Thomas Piazza. 2000. The and Ismail K. White. 2002. “Cues That Mat- Outsider: Prejudice and Politics in Italy. Princeton, ter: How Political Ads Prime Racial Attitudes NJ: Princeton University Press. during Campaigns.” American Political Science Sniderman, Paul M., and Thomas Piazza. 1993. Review 96: 75–90. The Scar of Race. Cambridge, MA: Harvard Uni- Walton, Hanes, Jr., Cheryl Miller, and Joseph versity Press. P. McCormick. 1995. “Race and Political Sniderman, Paul M., and Philip E. Tetlock. 1986. Science: The Dual Traditions of Race Rela- “Symbolic Racism: Problems of Motive Attri- tions and African American Politics.” In Polit- bution in Political Analysis.” Journal of Social ical Science in History, eds. John Dryzek, Issues 42: 129–50. James Farr, and Stephen Leonard. Cam- Terkildsen, Nayda. 1993. “When White Vot- bridge: Cambridge University Press, 145– ers Evaluate Black Candidates: The Processing 74. CHAPTER 23

Politics from the Perspective of Minority Populations

Dennis Chong and Jane Junn

Experimental studies of racial and ethnic of minority groups. We review the method- minorities in the United States have focused ology and findings of these experimental on the influence of racial considerations in studies to highlight their contributions and political reasoning, information processing, limitations and to make several general obser- and political participation. Studies have ana- vations and suggestions about future direc- lyzed the types of messages and frames that tions in this field. As we show, randomization prime racial evaluations of issues, the effect of and control strengthen the internal validity of racial arguments on opinions, and the impact causal inferences drawn in experiments; how- of racial cues on political choices. Underly- ever, of equal importance, the interpretation ing this research is the premise, developed in and significance of results depends on addi- observational studies (e.g., Bobo and Gilliam tional considerations, including the measure- 1990;Dawson1994;Tate1994;Lien2001; ment of variables, the external validity of the Chong and Kim 2006; Barreto 2007), that experiment, and the theoretical coherence of there are racial and ethnic differences in how the research design. individuals respond to cues and information. For this reason, almost all studies give special attention to the mediating and moder- 1. Racial Priming ating influences of racial group identification, a core concept in the study of minority Virtually all racial priming research has exam- politics. ined the public opinion of whites (e.g., Gilens There are too few studies yet to consti- 1999; Gilliam and Iyengar 2000; Mendelberg tute a research program, but the initial forays 2001) and been modeled on prior studies of have successfully featured the advantages of media priming of voter evaluations (Iyen- experimental design and distinct perspectives gar and Kinder 1987). The theory of racial priming is that attitudes toward candidates We are grateful to Xin Sun, Caitlin O’Malley, and and policies can be manipulated by fram- Thomas Leeper for research assistance on this project. ing messages to increase the weight of racial

320 Politics from the Perspective of Minority Populations 321 considerations, especially prejudice. Among War experiment referred to how the war whites, “implicit” racial messages that indi- drained money from social spending. rectly address race – using images or code As in prior studies of white opinion, the words – are hypothesized to prime racial experiments confirmed that resentment of considerations more effectively than racially blacks among whites was strongly related to explicit messages that violate norms of equal- support for the war and opposition to wel- ity. Explicit statements are potentially less fare spending only in the implicit condi- effective because they can raise egalitarian tion. Among blacks, racial identification was concerns that suppress open expression of strongly related to support in the explicit con- prejudice. In contrast, implicit appeals smug- dition on both issues, but, surprisingly, was gle in racial primes that activate prejudice and unrelated in the implicit condition. Thus, turn opinion against a policy or candidate explicitness of the cue has differential effects without triggering concerns about equality. among blacks and whites, roughly as pre- This is the racial priming theory as applied dicted by the theory. to whites (Mendelberg 2001; cf. Huber and There are some oddities, however. Racial Lapinski 2008). resentment among whites significantly reduces support for the war in the racially explicit condition, when the unequal burden Priming Racial Considerations of the war on blacks is emphasized. The among Blacks racial priming theory predicts that explicit The dynamics of racial appeals are likely to statements should weaken the relationship be different among blacks because racial mes- between resentment and support for the war, sages aimed at blacks often promote group but not reverse its direction. In the wel- interests without raising conflicting consid- fare experiment, egalitarian values are also erations between race and equality. There- strongly primed among whites in the implicit fore, in contrast to white respondents, blacks racial condition, in addition to out-group should be more likely to evaluate an issue using resentments. This means that egalitarian racial considerations when primed with either considerations potentially counteract racial explicit or implicit racial cues. resentment, even in the implicit case, con- To test the idea that explicit and implicit trary to expectations. Finally, in contrast messages affect blacks and whites differently, to past research (Kinder and Sanders 1996; White (2007) designed an experiment using Kinder and Winter 2001), blacks do not news articles to manipulate the verbal framing respond racially to the implicit message that of two issues: the Iraq War and social welfare war spending reduces social spending or to policy. The sample included black and white the implicit cues used to describe the welfare college students and adults not attending col- issue. lege. Participants were randomly assigned to The anomalies of an experimental study one of the treatments or to a control group can sometimes yield as much theoretical and that read an unrelated story. For each issue, methodological insight as the confirmatory one of the frames explicitly invoked black findings. In this case, anomalies force us to group interests to justify the position taken in reconsider the appropriate test of the prim- the article, a second frame included cues that ing hypothesis. A possible explanation for the implicitly referred to blacks, and a third frame weak effect of implicit cues among blacks is included nonracial reasons. In the welfare that the treatment affects the overall level experiment, there were two implicit frame of support for policies, in addition to the conditions in which the issue was associated strength of the relationship between racial pre- with either “inner-city Americans” or “poor dispositions and policy preferences. A flat Americans” on the assumption that these ref- slope coefficient between racial identification erences would stimulate racial resentment and policy positions does not eliminate the among whites and group interests for blacks. possibility that levels of support or opposi- Similarly, the implicit racial cue in the Iraq tion – reflected in the intercept term – change 322 Dennis Chong and Jane Junn among both strong and weak identifiers in They designed an experiment in which par- response to the treatment. The priming ticipants recruited from the Los Angeles area hypothesis therefore requires an examination were randomly assigned to one of four con- of both intercepts and slopes. ditions. In the control condition, participants Furthermore, the imprecise definition and watched a news video that did not include operationalization of explicit and implicit a crime story. In the three other conditions, cues raises measurement issues. In the wel- participants viewed a crime story in which the fare experiment, two implicit cues referring race of the alleged perpetrator was manipu- to “inner-city Americans” and “poor Ameri- lated. In one of the crime stories, there was cans” were incorporated in arguments made no description of the murder suspect. In the in support of welfare programs. Likewise, other two conditions, digital technology was the implicit racial condition in the Iraq War used to change the race (black or white) of the experiment refers to the war taking atten- suspect shown in a photograph. tion away from “domestic issues,” includ- Gilliam and Iyengar (2000) found that ing “poverty,” “layoffs,” “inadequate health whites who were exposed to a crime story care,” and “lack of affordable housing.” With- (regardless of the race of the suspect) tended out explicitly mentioning blacks, both treat- to be more likely to give dispositional ments refer to issues that are associated with explanations of crime, prefer harsh penal- blacks in the minds of many Americans. How- ties, and express racially prejudiced attitudes. ever, the “nonracial” condition in the welfare Black respondents, in contrast, either were experiment also refers to “poor” Americans unmoved by the treatments or were moved or “working” Americans losing food stamps, in the opposite direction as whites, toward Medicaid, and health care and falling into less punitive and prejudiced attitudes. The poverty, which are the same kinds of domestic limited variance (within racial groups) across policy references used in the implicit condi- treatments is partially explained by one of tions in the two experiments. As we elaborate the more fascinating and disconcerting find- shortly, the imprecise definition of explicit ings of the study. A large percentage (sixty- and implicit cues raises general issues of mea- three percent) of the participants who viewed surement and pre-testing of treatments that the video that did not mention a suspect are central to experimentation. nonetheless recalled seeing a suspect, and most of them (seventy percent) remembered seeing a black suspect. This suggests that The Media’s Crime Beat the strong associations between crime and A second priming study worth exploring in race in people’s minds led participants to fill detail for the substantive contributions and in missing information using their stereo- methodological issues it raises is Gilliam types. In effect, the “no suspect” condition and Iyengar’s (2000) study of the influence served as an implicit racial cue for many of local crime reporting in the Los Ange- participants, reducing the contrast between les media. Whereas White’s study investi- the “black suspect” and “no suspect” con- gated how priming affects the dimensions ditions. This finding reinforces the need to or considerations people use to evaluate an pre-test stimuli to determine whether treat- issue, Gilliam and Iyengar focus on attitude ments (and nontreatments) are working as change in response to news stories that stim- desired. ulate racial considerations underlying those Gilliam and Iyengar (2000) also analyzed attitudes. Los Angeles County survey data to show that Gilliam and Iyengar (2000) hypothesize frequent viewers of local news were more that the typical crime script used in local likely to express punitive views toward crim- television reporting (especially its racial bias inals and to subscribe to both overt and against blacks) has had a corrosive effect on subtle forms of racism. This corroboration viewers’ attitudes toward the causes of crime, between the observational and experimen- law enforcement policies, and racial attitudes. tal data bolsters the external validity of the Politics from the Perspective of Minority Populations 323 experimental effects. However, one wonders ies highlights the variable effects of simi- what impact can be expected from an exper- lar experimental treatments. Bobo and John- imental treatment that is a miniscule frac- son hypothesize that because blacks are more tion of the total exposure to crime stories likely to believe that the criminal justice sys- that participants received prior to joining the tem is racially biased, they are more likely experiment. Before reporting their experi- to be influenced by frames accentuating bias mental results, Gilliam and Iyengar them- in the system when they are asked their selves caution readers that “our manipulation opinion of the death penalty and other sen- is extremely subtle. The racial cue, for exam- tencing practices. For each survey experi- ple, is operationalized as a five-second expo- ment, respondents were randomly assigned sure to a mug shot in a ten-minute local news to receive one of several framed versions of presentation. Consequently we have modest a question about the criminal justice system expectations about the impact of any given (the treatment groups) or an unframed ques- coefficient” (567). tion (the control group). Yet, their treatment produces large effects. Most of the tests revealed surprisingly lit- For example, exposure to the treatment fea- tle attitude change among either blacks or turing a black suspect increases scores on whites in response to frames emphasizing the new racism scale by twelve percentage racial biases on death row, racial disparities in points. Compare that amount to the differ- the commission of crimes, and wrongful con- ence between survey respondents who hardly victions. The only frame that made a slight ever watched the local news and those who difference emphasized the greater likelihood watched the news on a daily basis: the most that a killer of a white person would receive frequent viewers scored twenty-eight per- the death penalty than a killer of a black percentage points higher on the new racism scale. son. This manipulation significantly lowered It is puzzling how a single exposure to a sub- support for the death penalty among blacks, tle manipulation can produce an effect that but not among whites (although the percent- is almost fifty percent of the effect of regular age shifts are modest). news watching. Perhaps the decay of effects Attitudes toward drug offenses proved to is rapid and the magnitude of the exper- be more malleable and responsive, specifically imental treatment effect varies across par- to frames emphasizing racial bias in sentenc- ticipants depending on their pre-treatment ing. Attempts to change views of capital pun- viewing habits. Both the experiment and the ishment may yield meager results, but efforts survey indicate that the style of local televi- to reframe certain policies associated with the sion coverage of crime in Los Angeles has war on drugs may have substantial effects on had a detrimental effect on viewers’ attitudes opinion. toward race and crime. However, to reconcile Peffley and Hurwitz (2007) also test the results of the two studies, we need more whether capital punishment attitudes are evidence of how viewers’ attitudes are shaped malleable among blacks and whites in over time when they are chronically primed response to arguments about racial biases (with variable frequency) by media exposure. in sentencing and the danger of executing innocent people. In contrast to Bobo and Johnson (2004), they find that both 2. Attitude Change arguments reduce support for the death penalty among blacks. However, the most Important studies by Bobo and Johnson shocking result is that the racial bias argu- (2004), Hurwitz and Peffley (2005), and Pef- ment causes support to increase significantly fley and Hurwitz (2007) employ survey exper- among whites. Peffley and Hurwitz explain imental methods on national samples to study that the racial bias argument increases sup- the malleability of black and white atti- port for the death penalty among preju- tudes under different framing conditions. A diced individuals by priming their racial atti- comparison of the results from these stud- tudes. This priming effect is made more 324 Dennis Chong and Jane Junn surprising if we consider the racial bias a two-by-two design in which participants argument to be an explicit racial argument received one of four combinations of frames that might alert white respondents to guard embedded in a media story about a recent against expressing prejudice. Supreme Court decision limiting affirmative Peffley and Hurwitz (2007) do not recon- action. The decision was described either cile their findings with the contrary results in as a decision barring preferential treatment Bobo and Johnson’s (2004) survey experiment for any group or as a major blow to affirmative beyond speculating that the racial bias frames action and social justice; in each media story, in the other survey may have been harder to there was either a critical comment about comprehend. Among other possible explana- Justice Clarence Thomas or no comment tions is that Bobo and Johnson’s use of an about Justice Thomas’ conservative vote on Internet sample overrepresented individuals the issue. with strong prior opinions about the death The participants were 146 white and black penalty who were inoculated against framing students from a large Midwestern university. manipulations. Bobo and Johnson, however, Comparisons of the sample to the American conclude that the frames are resisted irrespec- National Election Studies and National Black tive of the strength of prior opinions because Election Study samples revealed, as expected, they find no differences in the magnitude of that both black and white participants were framing effects across educational levels.1 younger, better educated, and wealthier than Two other anomalies in the Peffley and blacks and whites in the national sample. Hurwitz (2007) study are worth mentioning Black participants were also much more inter- briefly, and we return to them in the gen- ested in politics than was the national black eral discussion of this body of research. First, sample. “consistent with our expectations, blacks The dominant finding for black partici- apparently need no explicit prompting to pants is that they (in contrast to white par- view questions about the death penalty as ticipants) have firm positions on affirmative a racial issue. Their support for the death action regardless of how a recent conservative penalty, regardless of how the issue is framed, court ruling is framed. Among blacks, only is affected substantially by their belief about their attitude toward blacks (measured by the causes of black crime and punishment” racial resentment items) and gender predicted (1005). Although Peffley and Hurwitz antic- their attitude toward affirmative action; the ipated this result, it might be viewed as frames were irrelevant. being somewhat surprising in light of White’s The insignificance of framing in this (2007) demonstration that racial attitudes are experiment illustrates the difficulty of gen- related to public policies only when they eralizing beyond the experimental laboratory are explicitly framed in racial terms. Second, participants to the general population. Affir- among both black and white respondents, mative action is likely to be a more salient racial arguments do not increase the accessi- issue to African Americans, and attitudes on bility of other racial attitudes, such as stereo- salient issues are likely to be stronger and typical beliefs about blacks. more resistant to persuasion. Whether this is true for only a small subset of the black population or for most blacks can only be settled Framing Affirmative Action Decisions with a more representative sample. Clawson, Kegler, and Waltenburg’s (2003) study of the framing of affirmative action illustrates the sensitivity of results to the sam- 3. Racial Cues and Heuristics ple of experimental participants. They used The next set of studies we review involves experimental tests of the persuasiveness of 1 Their explanation assumes that the strength of attitudes toward the death penalty is positively correlated different sources and messages. These stud- with education. ies focus on minority responses to consumer Politics from the Perspective of Minority Populations 325 and health messages, but they are relevant for tions, strength of racial identity did not mod- our purposes because their findings on how erate the effect of the race of the source. racial minorities use racial cues in processing The favoritism that blacks show toward information can be extrapolated to political a black source in a public health message choices. is also demonstrated in an experiment by In the basic experimental design, partic- Herek, Gillis, and Glunt (1998) on the fac- ipants (who vary by race and ethnicity) are tors influencing evaluation of AIDS mes- randomly assigned to receive a message from sages presented in a video. Blacks evaluated a one of several sources that vary by race or black announcer as more attractive and cred- ethnicity and expertise. The primary hypoth- ible than a white announcer, but these in- esis is that sources that share the minority group biases were not manifest among whites. participant’s race or ethnicity will be evalu- Blacks also favored videos that were built ated more highly along with their message. around culturally specific messages, in con- A second hypothesis is that the impact of trast to multicultural messages. The manipu- shared race or ethnicity will be moderated by lations in this experiment affected proximate the strength of the participant’s racial iden- evaluations of the announcer and the mes- tity. Finally, these studies test whether white sage, but did not affect attitudes, beliefs, and participants favor white sources and respond behavioral intentions regarding AIDS. negatively to minority sources. Whittler and Spira’s (2002) study of con- Appiah (2002) found that black audi- sumer evaluations hypothesizes that source ences recalled more information delivered characteristics will serve as peripheral cues, by a black source than a white source in a but can also motivate cognitive elaboration videotaped message. This study also found of messages. Studies have shown that whites that white participants’ recall of informa- sometimes focus more heavily on the con- tion about individuals on a videotape was tent of the message when the source is black unaffected by the race of those individuals. (White and Harkins 1994; Petty, Fleming, White subjects’ evaluation of sources was and White 1999). The sample consisted of based on social (occupation, physical appear- 160 black adults from a southeastern city ance, social status) rather than racial features, assigned to a 2 × 2 experimental design. Par- perhaps because race is less salient to individ- ticipants received a strong or weak argument uals in the majority. from either a black or white speaker advertis- Wang and Arpan (2008) designed an ing a garment bag. experiment to study how race, expertise, and The evidence in the Whittler and Spira group identification affected black and white (2002) study is mixed. Participants over- audiences’ evaluations of a public service looked the quality of arguments and rated announcement (PSA). The participants for the product and advertising more favorably the experiments were black and white under- if it was promoted by a black source, but this graduate students recruited from a university bias was evident only among participants who in the southeastern United States and from a identified strongly with black culture. Iden- historically black college in the same city. tification with the black source appeared to Black respondents not only rated a black generate more thought about the speaker and source more highly than a white source, but the advertisement; however, because addi- also reacted more positively toward the PSA tional thinking was also biased by identifica- when it was delivered by a black source. How- tion, greater thought did not lead to discrim- ever, the effect of the source on blacks and ination between strong and weak arguments. whites was again asymmetric. Race did not Forehand and Deshpande (2001) argue bias white respondents’ evaluations in the that group-targeted ads will be most effec- same way; instead, whites were more affected tive on audiences that have been ethnically in their evaluation of the message by the primed. Same-ethnicity sources or group- expertise (physician or nonphysician) of the targeted messages may not have a signif- source than were blacks. Contrary to expecta- icant impact on audiences unless ethnic 326 Dennis Chong and Jane Junn self-awareness is initially primed to make the mentioned Whittler and Spira study, some audience more receptive to the source. respondents relied entirely on the (periph- The subjects in the Forehand and Desh- eral) racial cue to form their judgment, but pande (2001) study were Asian American even those respondents who gave more atten- and white students from a West Coast tion to the substance of the message con- university. Advertisements were sandwiched strued it in light of the source. between news segments on video, with ethnic Surprisingly, we did not discover any primes preceding advertisements aimed at the experimental research using this basic design ethnic group. Similar results were obtained to analyze the effect of race and ethnicity on in both experiments. Exposure to the ethnic minority voter choice. An innovative experi- prime caused members of the target audi- mental study by Terkildsen (1993) examined ence to respond more favorably to the eth- the effects of varying the race (black or white) nic ad. However, the magnitude of the effect and features (light or dark skin tone) of can- of the ethnic prime was not magnified by didates, but only on the voting preferences of strong ethnic identification, so the expecta- white respondents (who evaluated the white tion of an interaction with enduring identifi- candidate significantly more positively than cations was not met. This is a surprising result either of the two black candidates.). because we would expect strong identifiers to Abrajano, Nagler, and Alvarez (2005) took be more likely both to recognize the ethnic advantage of an unusual opportunity in Los prime and to base their judgment on it. Expo- Angeles County to disentangle ethnicity and sure to the ethnic prime among members of issue distance as factors in voting. In this nat- the nontarget market (whites in the exper- ural experiment using survey data, Abrajano iment) resulted in less favorable responses, et al. analyzed the electoral choices of vot- but the magnitudes were statistically insignif- ers in two open city races involving Latino icant. Once again, it does not appear that an candidates running against white candidates. ethnic prime has a negative effect on individ- In the mayoral race, the white candidate was uals who do not share the same ethnicity. more conservative than the Latino candidate, but the white candidate was the more liberal candidate in the city attorney election. They Extensions to Vote Choice found that Latino voters were more affected An obvious extrapolation from these stud- by the candidates’ ethnicity and much less ies is to examine how variation in the race affected by their issue positions than were or ethnicity of a politician influences polit- white voters. ical evaluations and choices. Kuklinski and Hurley (1996) conducted one of the few experimental studies in political science along 4. Political Mobilization: Get Out these lines. African Americans recruited from the Vote the Chicago metropolitan region were randomly assigned to one of four treatments Aside from laboratory and survey research or to a control group. Each treatment pre- on persuasion and information processing, sented a common statement about the need the study of political mobilization is the for self-reliance among African Americans, other area in which there has been sustained but the statement was attributed to a differ- experimental research on minority groups.2 ent political figure in each of the four condi- Field research on the political mobilization tions: George Bush, Clarence Thomas, Ted of minorities comes with special challenges Kennedy, or Jesse Jackson. If the statement because it requires investigators to go beyond was attributed to Bush or Kennedy, then par- standard methodologies for data collection in ticipants were more likely to disagree with it, the midst of electoral campaigns. Researchers but if the observation originated from Jackson or Thomas, then they were significantly more 2 A full review of GOTV experiments is provided by likely to agree. As in the case of the afore- Michelson and Nickerson’s chapter in this volume. Politics from the Perspective of Minority Populations 327 must take care to locate the target populations the robo-calling nor the direct mail had a for study, provide multilingual questionnaires discernible and reliable influence on voter and interviewers in some cases, and design turnout among Latinos, but the live tele- valid and reliable treatments appropriate to phone calls did have a positive effect on mobi- minority subjects. lizing voters. Garcia Bedolla and Michelson (2009) Trivedi (2005) attempted to discern report on a field experimental study of a mas- whether distinctive appeals to ethnic group sive effort to mobilize voters through direct solidarity among Indian Americans would mail and telephone calls in California during increase voter turnout. Despite three alter- primary and general election phases of the native framings with racial and ethnic cues, 2006 election. The content of the direct mail there were no significant effects on voter included a get out the vote (GOTV) message, turnout of any of the three groups that but varied in terms of procedural information received the mailing. Wong’s (2005)field such as the voter’s polling place and a photo experiment during the 2002 election included included in the mailer that was adjusted “to a postcard mailing or a phone call for ran- be appropriate to each national-origin group” domly assigned Asian American registered (9). The authors found no significant impact voters in Los Angeles County who resided of direct mail and a positive effect of a phone in high-density Asian American areas. In con- call on voting turnout among the Asian Amer- trast to other studies that show no effect from ican subjects contacted (with considerable direct mail, Wong found positive effects for variance across groups classified by national both mobilization stimuli on Asian American origin). Considering the extremely low base voter turnout. rate of voting in the target population, the Finally, Michelson (2005) reports on a treatment had a large proportional impact. series of field experiments with Latino sub- The authors’ conclusions from this set of jects in central California. Her results show experiments and other GOTV studies in Cal- that Latino Democrats voted at a higher ifornia and elsewhere point to the signifi- rate when contacted by a coethnic canvasser. cance of a personal invitation to participate. Michelson suggests possible social and cogni- At the same time, however, they admitted, tive mechanisms that explain why “coethnic” “We do not have a well-defined theoretical contacts may stimulate higher levels of partic- understanding of why an in-person invita- ipation: “Latino voters are more likely to be tion would be so effective, or why it could receptive to appeals to participate when those counteract the negative effect of low voter appeals are made by coethnics and copar- resources” (271). tisans. . . . In other words, if the messenger The results from the Garcia Bedolla and somehow is able to establish a common bond Michelson (2009) field experiments are con- with the voter – either through shared ethnic- sistent with earlier studies of Latinos, Asian ity or through shared partisanship – then the Americans, and African Americans that have voter is more likely to hear and be affected by shown direct mail to be ineffective and per- the mobilization effort” (98–99). sonal contact to be effective interventions Taken together, none of the field exper- with minority voters. In a large-scale national iments shows strong or consistent positive field experiment with African American vot- effects from direct mailings, regardless of ers during the 2000 election, Green (2004) content, format, or presentation of the infor- found no significant effects on turnout with mation in a language assumed to be most a mailing and small but statistically signifi- familiar to the subject. Second, personal con- cant effects from a telephone call. In another tact with a live person in a telephone call has large field experiment during the 2002 elec- a modest absolute effect, and potentially a tion, Ramirez (2005) analyzed results from greater effect if the contact occurs in per- attempted contact with nearly a half-million son, especially if that individual is a “coeth- Latinos by the National Association of Latino nic” canvasser. Third, effective methods for Elected and Appointed Officials. Neither mobilizing specifically minority voters are 328 Dennis Chong and Jane Junn essentially the same as those found to work hension may be impaired because of language on majority group populations. Personal difficulties or inability. Furthermore, the ad contacts, unscripted communications, and hoc design of the ethnic cues makes them face-to-face meetings provide more reliable equivalent to an untested drug whose effects boosts to turnout than do more automated, have not heretofore been demonstrated in remote techniques (Gerber and Green 2000). pre-testing. Unlike laboratory experiments To the extent that field experimental in which effects are measured promptly, the research on the mobilization of minority vot- GOTV treatments are expected to influence ers is distinct from studies of white voters, it behavior (not simply attitudes) days or weeks is due to the nascent hypothesis that identifi- later, so it is perhaps not surprising that they cations and social relationships based on race have proved to be anemic stimulants. and ethnicity ought to moderate the impact of mobilization treatments. This intuition or hunch lies behind experimental manipula- 5. General Observations and tions that try to stimulate group identifica- Future Directions tion using racial or ethnic themes or imagery in communications. A Latino voter, for exam- In our review, we considered not only ple, is told his or her vote will help empower whether a treatment had a significant effect the ethnic community, or an Asian American in a particular study, but also the interpre- is shown a photograph of Asian Americans tation of those effects (does the explanation voting out of duty as U.S. citizens. accurately reflect what occurred in the exper- The prediction that treatments will ele- iment?) and the consistency of experimental vate racial and ethnic awareness and thereby effects across related studies (are the results increase one’s motivation to vote is based on of different studies consistent with a com- several assumptions about both the chronic mon theory?). We elaborate in this section accessibility of social identifications and the by discussing how to strengthen the internal durability of treatment effects. Some obser- and external validity of experiments through vations drawn previously from our review of improvements in measurement, design, and political psychology research should lower theory building. our expectations about the impact of such efforts to manipulate racial identities. In com- Generalizing from a Single Study munications experiments on racial priming, participants are often college students who The variability of results across studies rec- are unusually attentive to experimental treat- ommends careful extrapolation from a single ments; volunteers who agree to participate are study. For example, the failure of a framed eager to cooperate with the experimenter’s argument to move opinion may be explained instructions and pay close attention to any by the imperviousness of participants on the materials they are asked to read or view. This issue or by the weakness of the frames. Bobo combination of higher motivation and capac- and Johnson (2004) chose the former inter- ity means the laboratory subject receives what pretation in arguing that “with respect to the amounts to an especially large dosage of the death penalty, our results point in the direc- treatment. Finally, if we assume the treatment tion of the relative fixity of opinion” (170). to be not only fast acting but also fast fading – However, they also concluded from their sur- like a sugar rush – then its effects will only be vey experiment that reframing the war on detected if we measure them soon after it is drugs as “racially biased” might significantly administered. reduce support for harsh sentencing practices The typical GOTV field experiment devi- among both blacks and whites. ates from these conditions in each instance. Peffley and Hurwitz’s (2007) contrary Attention to the treatment and interest in finding that whites increase their support for politics are low (indeed, sometimes the partic- the death penalty when told it is racially ipants are selected on this basis), and compre- biased led them to conclude that “direct Politics from the Perspective of Minority Populations 329 claims that the policy discriminates against reactions to these frames using the implicit– African Americans are likely to create a back- explicit theoretical framework. lash among whites who see no real discrimi- This conceptual task is made more diffi- nation in the criminal justice system” (1009). cult because the dividing line between explicit Whether the frames used in the two stud- and implicit varies across audiences. Different ies varied or the participants varied in the audiences, owing to differences in past learn- strength of their existing attitudes cannot ing experiences, will draw different connota- be resolved without systematically compar- tions from the same message. Certain mes- ing the frames and prior attitudes of the par- sages are so blunt that they obviously draw ticipants; the same ambiguity between the attention to racial considerations. Other mes- strength of arguments versus prior opin- sages allude indirectly to race and can be ions hovers over the null findings in Claw- interpreted in racial terms only by those who son et al.’s (2003) study of affirmative action are able to infer racial elements from ostensi- attitudes among blacks. At a minimum, repli- bly nonracial words or symbols because of cation of results using both identical and common knowledge that such symbols or varied treatments and different samples of words connote racial ideas (Chong 2000). respondents would increase our confidence in Of course, if the common knowledge is either the stability or malleability of opinion so widespread as to be unambiguous, then on these issues. even the implicit message becomes explicit to everyone in the know. Thus, there is supposedly a sweet spot of ambiguity wherein lie Importance of Pre-Testing Measures messages that cause people to think in racial The internal validity of a study depends on terms either without their knowing it or with- reliable and valid measures. A general les- out their having to admit it because there son drawn from the experiments on persua- is a plausible nonracial interpretation of the sion and information processing, as well as message. the GOTV studies, is the need to pre-test We do not have a ready solution for distin- stimuli to establish that treatments have the guishing between explicit and implicit mes- characteristics attributed to them. These pre- sages. One possibility is that the location of a tests will be sample dependent and should be message on a continuum ranging from more administered to individuals who do not par- to less explicit will correspond to the bal- ticipate in the main experiment. ance of racial and nonracial interpretations Our review provided several instances and thoughts that are spontaneously men- where progress on the effects of racial tioned when interpreting the message (Feld- priming and framing would be aided by man and Zaller 1992). However, individu- more clearly defined measures of explicit als who are careful to monitor their public and implicit messages. Mendelberg (2001) behavior may not candidly report their spon- classifies visual racial cues as implicit mes- taneous thoughts, especially if those thoughts sages and direct verbal references to race as are racial in nature. explicit messages. White (2007) distinguishes A covert method of eliciting the same between explicit cues that mention race and information uses subliminal exposure to a implicit verbal cues that allude to issues and given message followed by tests of reaction terms that are commonly associated with race. times to racial and nonracial stimuli. More However, a domestic issue cue that is assumed explicit messages may be expected to pro- to be an implicit racial cue in one experiment duce quicker reactions to racial stimuli than is defined as a nonracial cue in another exper- implicit messages. Lodge and Taber (in press) iment. Bobo and Johnson (2004) and Peffley provide a convincing demonstration of how and Hurwitz (2007) introduce frames that implicit testing of competing theoretical posi- refer directly to the disproportionate treat- tions can shed light in the debate over the ment of blacks under the criminal justice sys- rationales underlying support for symbolic tem, but do not interpret their respondents’ racial political values (for a review, see Sears, 330 Dennis Chong and Jane Junn

Sidanius, and Bobo 2000). A racial issue (e.g., parries these attacks undoubtedly has much affirmative action) is used as a prime (pre- to do with the success of the original strat- sented so briefly on the screen that it regis- egy. Despite the theory, all testing has been ters only subconsciously), and the words auto- essentially static and limited to a one-time matically activated by this issue (determined administration of one-sided information. by the speed of recognizing them) are inter- Another way to increase the realism of preted to be the considerations raised by an designs is to examine communications pro- issue. Coactivation of concepts is said to rep- cesses over time (Chong and Druckman resent habits of thought, reflecting how indi- 2008). The persuasion and information pro- viduals routinely think about the issue. Using cessing studies we reviewed were one-shot this method, Lodge and Taber show that, studies in which the magnitudes of com- among supporters of affirmative action, ideol- munication effects were measured immedi- ogy and racial considerations were activated ately following exposure to the treatment. or facilitated by the affirmative action issue The design of these laboratory experiments prime. However, among opponents, only contrasts with conditions in the real world, racial words were activated (e.g., gang, afro). where individuals typically receive streams of Therefore, it appears that the liberal position messages and act on them at the end of a on the issue drew on more principled con- campaign. The interpretation of a one-shot siderations, whereas the conservative position experiment should therefore take account of rested on racial considerations. the previous experiences of participants and the subsequent durability of any observed effects. A treatment may have a larger impact Adding Realism through Competition if it is received early rather than late in and Over-Time Designs a sequence of communications because the In the framing studies we examined, partici- effect of a late treatment may be dulled by past pants were exposed to arguments on only one messages. In addition, we want to measure side of the issue under investigation. In con- the durability of effects in the post-treatment trast, competition between frames and argu- period. A significant treatment effect may ments reflects the reality of political debate in decay rapidly either on its own accord or democratic systems. Multiple frames increase under the pressure of competing messages. the accessibility of available considerations, Ultimately, the effect of a treatment will be and competition between frames can moti- time dependent. vate more careful deliberation among alter- The importance of taking account of par- natives (Chong and Druckman 2007). Fram- ticipants’ pre-treatment experiences is evi- ing effects produced by a one-sided frame dent in the study of death penalty frames. are often not sustained in competitive envi- The effectiveness of arguments against the ronments (Sniderman and Theriault 2004; death penalty likely depends on whether Chong and Druckman 2007; cf. Chong and participants have previously heard and fac- Druckman 2008). tored these arguments into their attitudes The theory of implicit and explicit mes- on the issue (Gaines, Kuklinski, and Quirk sages, in particular, would benefit from 2007). A paradox in this regard is the sizable a competitive experimental design because effects generated by Gilliam and Iyengar’s racial priming describes an inherently (2000) “subtle” media crime news interven- dynamic process in which strategic political tion, which led us to wonder why experimen- messages are transmitted and countered and tal participants who have been “pre-treated” subject to claims and counterclaims about the with everyday exposure to crime news cov- meaning of the message and the intent of the erage would nevertheless remain highly sen- messenger. The essence of an intrinsic racial sitive to the experimental treatment. If one- message is that it can be defended against shot experimental exposure to crime stories attacks that it is a racial (and perhaps racist) produces an effect that is fifty percent of the message. How the originator of the message effect of chronic real-world exposure, how Politics from the Perspective of Minority Populations 331 much of this short-term effect decays and awareness by itself will have little effect on how much endures in the long-term effect? political participation without the constel- To disentangle these processes of learning lation of intervening cognitive factors that and decay of opinion, we need to move motivate individuals to participate. from one-shot experimental designs to panel A fruitful theory of racial identification experiments, in which we measure attitude provides testable hypotheses of the conditions change in response to a series of exposures to in which voters can be more easily mobi- treatments over time (Chong and Druckman lized on the basis of their race or ethnic- 2008). A panel design would allow us to deter- ity. Minority voters should be more respon- mine how the size and durability of effects are sive to racial cues when electoral candidates moderated by past experiences, the passage of and issues place group interests at stake, time, and subsequent exposure to competing and collective action is an effective means to messages. obtain group goals (Chong 2000). The selection of future sites of GOTV field experiments therefore might exploit political con- Integrating Theory and Design texts in which minority voters are likely to be Although experiments are well suited to test- especially susceptible to messages and con- 3 ing whether an arbitrary treatment has an tacts that prime their racial identification. impact on an outcome variable (in the absence Contact by coethnic organizers, which has of a theoretically derived hypothesis), such a proved effective in past studies, may have even theory is ultimately required to explain and greater impact in these circumstances, espe- bring coherence to disparate results. Other- cially if campaign workers are drawn from wise, experiments are at risk of being a series the voter’s social network. When political of one-off exercises. opportunities for gain present themselves, The theoretical challenge of specifying the monitoring within the group – verified to be meaning and measurement of racial group one of the most powerful influences on vot- identification and its relationship to politi- ing in GOTV research (Gerber, Green, and cal behavior and attitudes is one of the most Larimer 2008) – would apply added social significant hurdles in research on the polit- pressure on individuals to contribute to pub- ical psychology and behavior of minorities. lic goods. GOTV studies have tested whether stimulat- The importance of theoretical developing racial identification can increase turnout ment to experimental design applies to the in elections, but this research has not been areas of research that we cover. Although we guided explicitly by a theory of the mech- group the persuasion, priming, and framing anisms that activate racial identification or research as a “set” of studies that addresses of the factors that convert identification to common issues, they are not unified theoreti- action. As noted, the failure of efforts to moti- cally. This inhibits development of a research vate voter turnout by using racial or ethnic program in which there is consensus around appeals can be explained in large part by the certain theoretical concepts and processes superficial nature of the treatments. GOTV that serve as a framework for designing new experiments might draw on the results of experimental studies. past survey research on racial identification, The communications research we discuss which showed that the connection between here can be interpreted in terms of exist- group identification and political participa- ing dual process theories of information pro- tion is mediated by perceptions of group sta- cessing (Petty and Cacioppo 1986; Petty tus, discontent with the status quo, and beliefs 3 There is a risk that the individuals in these more about the origins of group problems and effi- racialized political contexts may have already been cacy of group action (Miller et al. 1981; Shin- activated by the ongoing campaign prior to the exper- gles 1981; Marschall 2001; Chong and Rogers imental manipulation. This pre-treatment of respon- 2005 dents may dampen the impact of any further exper- ). This model of racial identification sug- imental treatment that duplicates what has already gests that experimental manipulation of racial occurred in the real campaign. 332 Dennis Chong and Jane Junn and Wegener 1999; Fazio and Olson 2003). for advancing our understanding of U.S. pol- According to this theory, individuals who itics. Indeed, the insights generated by exper- have little motivation or time to process infor- imental studies of framing, persuasion, racial mation will tend to rely on economical short- priming, and political mobilization in both cuts or heuristic rules to evaluate messages. majority and minority populations have made Attitude change along this “peripheral” route, it difficult to think of these topics outside the however, can be transitory, leading to short- experimental context. Observational studies term reversion to past beliefs. of these subjects are hampered by selec- Conversely, individuals who are motivated tion biases in the distribution and receipt and able to think more deeply about a sub- of treatments and lack of control over the ject, due to incentives or predispositions to design of treatments. One of the most impor- do so, will process information along a “cen- tant advantages of experimental over tradi- tral” route by giving closer attention to the tional observational or behavior methods is quality of arguments in the message. Individ- the promise of greater internal validity of the uals who hold strong priors on the subject are causal inferences drawn in the experiment. more likely to ignore or resist contrary infor- We can test the impact of alternative treat- mation and to adhere to their existing atti- ments without strong priors about the mech- tudes. In some cases, the cognitive effort they anism that explains why one treatment will be expend will end up bolstering or strengthen- more effective than another. ing their attitudes. However, if the arguments The range of possible studies is exciting. are judged to be strong and persuasive, they For example, we can randomly manipulate can lead to attitude change that is enduring. the background characteristics of hypothet- A variety of additional studies can be built ical candidates for office in terms of their around the dynamics of dual process theo- partisanship, race, or ideology and estimate ries as they pertain to racial identities and the impact of these differences on candidate attitudes. The motivation and opportunities preference among voters. Similarly, experi- of participants can be manipulated to deter- ments could be designed to highlight or frame mine the conditions that increase or reduce specific features of candidates or issues to the salience of race. We can experiment with observe the effect of such manipulations on manipulating information processing modes voter preferences. The salience of the voter’s by varying the speed of decision making, the racial and ethnic identity can be heightened stakes of the decision, and increasing per- or reduced to see how group identity and sonal accountability to see whether central campaign messages interact to change voter or peripheral routes are followed. Explicit preferences or increase turnout. Efforts to racial messages, for example, should exert less embed research designs and studies in a the- influence on people who have had sufficient ory of information processing may yield the opportunity and motivation to engage in self- most fruitful set of results. In experimental monitoring of their responses to the racial designs, the dynamics of dual process theo- cue (Terkildsen 1993). Cognitive elaboration, ries can be exploited to manipulate partici- of course, should not be expected to ensure pants’ motivation and opportunity to evaluate attitude change. As the Whittler and Spira information in order to reveal the conditions (2002) study discovered, racial identities can that systematically influence the salience of anchor viewpoints (akin to party identifica- race. tion) through motivated reasoning and biased At the same time, experimentation can- information processing. not be regarded as a substitute for theory building. Results across studies are often conflicting, illustrating the sensitivity of results 6. Conclusion to variations in measurement and the sample of experimental participants. In the con- Experimental research on the political per- text of research on racial and ethnic minori- spectives of minorities holds much promise ties, we discuss how theory is essential for Politics from the Perspective of Minority Populations 333 conceptual development, designing treat- Clawson, Rosalee A., Elizabeth R. Kegler, and Eric ments, interpreting results, and generalizing N. Waltenburg. 2003. “Supreme Court Legit- beyond particular studies. We also identify imacy and Group-Centric Forces: Black Sup- what we believe to be promising directions port for Capital Punishment and Affirmative Action.” Political Behavior 25: 289–311. to address the external validity of experi- 1994 mental designs, including the incorporation Dawson, Michael. . Behind the Mule Race and Class in African American Politics.Princeton,NJ: of debate and competition, use of over-time Princeton University Press. panel designs, and greater attention to the Fazio, Russell H., and Olson, Michael A. 2003. interaction between treatments and political “Attitudes: Foundations, Functions, and Con- contexts. sequences.” In The Handbook of Social Psychol- ogy, eds. Michael A. Hogg and Joel M. Cooper. London: Sage, 139–60. References Feldman, Stanley, and John Zaller. 1992.“The Political Culture of Ambivalence: Ideological Abrajano, Marisa A., Jonathan Nagler, and R. Responses to the Welfare State.” American 36 268 307 Michael Alvarez. 2005. “A Natural Experiment Journal of Political Science : – . 2001 of Race-Based and Issue Voting: The 2001 City Forehand, Mark R., and Rohit Deshpande. . of Los Angeles Elections.” Political Research “What We See Makes Us Who We Are: Quarterly 58: 203–18. Priming Ethnic Self-Awareness and Advertis- 38 Appiah, O. 2002. “Black and White Viewers’ Per- ing Response.” Journal of Marketing Research : 336 48 ception and Recall of Occupational Characters – . on Television.” Journal of Communications 52: Gaines, Brian J., James H. Kuklinski, and Paul J. 2007 776–93. Quirk. . “Rethinking the Survey Experi- 15 1 21 Barreto, Matt A. 2007. “Si Se Puede! Latino Can- ment.” Political Analysis : – . didates and the Mobilization of Latino Vot- Garcia Bedolla, Lisa, and Melissa R. Michelson. 2009 ers.” American Political Science Review 101: 425– . “What Do Voters Need to Know? Test- 41. ing the Role of Cognitive Information in Asian Bobo, Lawrence, and Frank Gilliam. 1990. “Race, American Voter Mobilization.” American Poli- 37 254 74 Sociopolitical Participation and Black Empow- tics Research : – . 2000 erment.” American Political Science Review 84: Gerber, Alan S., and Donald P. Green. . 379–93. “The Effects of Canvassing, Telephone Calls, Bobo, Lawrence D., and Devon Johnson. 2004.“A and Direct Mail on Voter Turnout: A Field Taste for Punishment: Black and White Amer- Experiment.” American Political Science Review 94 653 63 icans’ Views on the Death Penalty and the War : – . on Drugs.” DuBois Review 1: 151–80. Gerber, Alan S., Donald P. Green, and Christo- 2008 Chong, Dennis. 2000. Rational Lives: Norms and pher W. Larimer. . “Social Pressure and Values in Politics and Society. Chicago: The Uni- Voter Turnout: Evidence from a Large-Scale versity of Chicago Press. Field Experiment.” American Political Science 102 33 48 Chong, Dennis, and James N. Druckman. 2007. Review : – . 1999 “Framing Public Opinion in Competitive Gilens, Martin. . Why Americans Hate Welfare: Democracies.” American Political Science Review Race, Media, and the Politics of Anti-Poverty Policy. 101: 637–55. Chicago: The University of Chicago Press. 2000 Chong, Dennis, and James N. Druckman. 2008. Gilliam, Frank D., Jr., and Shanto Iyengar. . “Dynamic Public Opinion: Framing Effects “Prime Suspects: The Impact of Local Televi- over Time.” Paper presented at the annual sion News on the Viewing Public.” American 44 560 73 meeting of the American Political Science Journal of Political Science : – . 2004 Association, Boston. Green, Donald P. . “Mobilizing African- Chong, Dennis, and Dukhong Kim. 2006.“The American Voters Using Direct Mail and Com- Experiences and Effects of Economic Status mercial Phone Banks: A Field Experiment.” 57 245 55 among Racial and Ethnic Minorities.” Amer- Political Research Quarterly : – . ican Political Science Review 100: 335–51. Herek, Gregory M., J. Roy Gillis, and Erik K. 1998 Chong, Dennis, and Reuel Rogers. 2005. “Racial Glunt. . “Culturally Sensitive AIDS Edu- Solidarity and Political Participation.” Political cational Videos for African American Audi- Behavior 27: 347–74. ences: Effects of Source, Message, Receiver, 334 Dennis Chong and Jane Junn

and Context.” American Journal of Community Petty, Richard E., and John T. Cacioppo. 1986. Psychology 26: 705–43. Communication and Persuasion: Central and Huber, Gregory A., and John S. Lapinski. 2008. Peripheral Routes to Attitude Change. New York: “Testing the Implicit–Explicit Model of Racial- Springer-Verlag. ized Political Communication.” Perspectives on Petty, Richard E., Monique A. Fleming, and P. H. Politics 6: 125–34. White. 1999. “Stigmatized Sources and Persua- Hurwitz, Jon, and Mark Pefﬂey. 2005. “Explaining sion: Prejudice as a Determinant of Argument the Great Racial Divide: Perceptions of Fair- Scrutiny.” Journal of Personality and Social Psy- ness in the U.S. Criminal Justice System.” Jour- chology 76: 19–34. nal of Politics 67: 762–83. Petty, Richard E., and D. T. Wegener. 1999.“The Iyengar, Shanto, and Donald R. Kinder. 1987. Elaboration Likelihood Model: Current Status News That Matters: Television and American Opi- and Controversies.” In Dual Process Theories in nion. Chicago: The University of Chicago Social Psychology, eds. Shelly Chaiken and Yaa- Press. cov Trope. New York: Guilford Press, 41–72. Kinder, Donald R., and Lynn M. Sanders. 1996. Ramirez, Ricardo. 2005. “Giving Voice to Latino Divided by Color: Racial Politics and Democratic Voters: A Field Experiment on the Effective- Ideals. Chicago: The University of Chicago ness of a National Nonpartisan Mobilization Press. Effort.” Annals of the American Academy of Polit- Kinder, Donald R., and Nicholas Winter. 2001. ical and Social Science 601: 66–84. “Exploring the Racial Divide: Blacks, Whites, Sears, David O., Jim Sidanius, and Lawrence Bobo, and Opinions on National Policy.” American eds. 2000. Racialized Politics: The Debate about Journal of Political Science 45: 439–56. Racism in America. Chicago: The University of Kuklinski, James H., and Norman L. Hurley. Chicago Press. 1996. “It’s a Matter of Interpretation.” In Polit- Shingles, Richard D. 1981. “Black Consciousness ical Persuasion and Attitude Change,eds.Diana and Political Participation: The Missing Link.” C. Mutz, Paul M. Sniderman, and Richard American Political Science Review 75: 76–91. A. Brody. Ann Arbor: University of Michigan Sniderman, Paul M., and Sean M. Theriault. 2004. Press, 125–44. “The Structure of Political Argument and the Lien, Pei-te. 2001. The Making of Asian Amer- Logic of Issue Framing.” In Studies in Pub- ica through Political Participation. Philadelphia: lic Opinion: Attitudes, Nonattitudes, Measurement Temple University Press. Error, and Change, eds. Willem E. Saris and Lodge, Milton, and Charles Taber. in press. Paul M. Sniderman. Princeton, NJ: Princeton The Rationalizing Voter. New York: Cambridge University Press, 133–65. University Press. Tate, Katherine. 1994. From Protest to Politics: The Marschall, Melissa. 2001. “Does the Shoe Fit? New Black Voters in American Elections. Cam- Testing Models of Participation for African- bridge, MA: Harvard University Press. American and Latino Involvement in Local Terkildsen, Nayda. 1993. “When White Vot- Politics.” Urban Affairs Review 37: 227–48. ers Evaluate Black Candidates: The Processing Mendelberg, Tali M. 2001. The Race Card: Cam- Implication of Candidate Skin Color, Preju- paign Strategy, Implicit Messages and the Norm of dice, and Self-Monitoring.” American Journal Equality. Princeton, NJ: Princeton University of Political Science 37: 1032–53. Press. Trivedi, Neema. 2005. “The Effect of Identity- Michelson, Melissa R. 2005. “Meeting the Chal- Based GOTV Direct Mail Appeals on the lenge of Latino Voter Mobilization.” Annals of Turnout of Indian Americans.” Annals of the the American Academy of Political and Social Sci- American Academy of Political and Social Science ence 601: 85–101. 601: 115–22. Miller, Arthur, Patricia Gurin, Gerald Gurin, Wang, Xiao, and Laura M. Arpan. 2008. “Effects and Oksana Malanchuk. 1981. “Group Con- of Race and Ethnic Identity on Audience Evalu- sciousness and Political Participation.” Ameri- ation of HIV Public Service Announcements.” can Journal of Political Science 25: 494–511. Howard Journal of Communications 19: 44–63. Pefﬂey, Mark, and Jon Hurwitz. 2007. “Persuasion White, Ismail K. 2007. “When Race Matters and and Resistance: Race and the Death Penalty in When It Doesn’t: Racial Group Differences America.” American Journal of Political Science in Response to Racial Cues.” American Politi- 51: 996–1012. cal Science Review 101: 339–54. Politics from the Perspective of Minority Populations 335

White, Paul H., and Stephen G. Harkins. 1994. Advertising Messages?” Journal of Consumer “Race of Source Effects in the Elaboration Psychology 12: 291–301. Likelihood Model.” Journal of Personality and Wong, Janelle S. 2005. “Mobilizing Asian Ameri- Social Psychology 67: 790–807. can Voters: A Field Experiment.” Annals of the Whittler, Tommy E., and Joan Scattone Spira. American Academy of Political and Social Science 2002. “Model’s Race: A Peripheral Cue in 601: 102–14.

Part VII

INSTITUTIONS AND BEHAVIOR

CHAPTER 24

Experimental Contributions to Collective Action Theory

Eric Coleman and Elinor Ostrom

Collective action problems are difficult prob- thiness and fair contributions to the provi- lems that pervade all forms of social organi- sion of collective benefits, on the other. To zation from within the family to the orga- provide PGs, it is believed that governments nization of production activities within a must devise policies that change incentives firm and to the provision of public goods to coerce citizens to contribute to collective (PGs) and the management of common pool action. resources (CPRs) at local, regional, national, Formal analysis of collective action prob- and global scales. Collective action problems has been strongly affected by the path- lems occur when a group of individuals could breaking work of Olson (1965)onThe Logic achieve a common benefit if most contribute of Collective Action and the use of game the- needed resources. Those who would benefit ory (e.g., Hardin 1982; Taylor 1987), which the most, however, are individuals who do not improved the analytic approach to these contribute to the provision of the joint bene- problems. By replacing the naive assumptions fit and free ride on the efforts of others. If all of earlier group theorists (Bentley 1949;Tru- free ride, then no benefits are provided. man 1958) that individuals will always pursue Political scientists trying to analyze collec- common ends, these modes of analysis force tive action problems have been influenced by analysts to recognize the essential tensions a narrow, short-term view of human rational- involved in many potential social interactions. ity combining an all-powerful computational Using the same model of individual behavior capacity, on the one hand, with no capabil- used to analyze production and consumption ity to adapt or acquire norms of trustwor- processes of private goods to examine collective action problems was an essential first The authors want to acknowledge financial support from step toward providing a firmer foundation for the National Science Foundation and the Workshop in all types of public policy. The empirical sup- Political Theory and Policy Analysis at Indiana Univer- port for these predictions within a competi- sity. Suggestions and comments from Roy Duch, Arthur Lupia, and Adam Seth Levine are appreciated, as is the tive market setting gave the enterprise an ini- editing assistance of David Price and Patty Lezotte. tial strong impetus.

339 340 Eric Coleman and Elinor Ostrom

Homo economicus has turned out to be a Player 2 special analytic tool rather than the general Cooperate Defect theory of human behavior. Models of short- Player 1 Cooperate (1,1) (–1,2) term material self-interest have been highly Defect (2, –1) (0,0) successful in predicting marginal behavior Figure 24.1. A Prisoner’s Dilemma Game in competitive situations in which selection pressures screen out those who do not maximize external values, such as profits in a com- Camerer (2003) stresses that games are not petitive market (Alchian 1950;Smith1991) equivalent to game theory. Games denote the or the probability of electoral success in party players, strategies, and rules for making deci- competition (Satz and Ferejohn 1994). Thin sions in particular interactions. Game theory, models of rational choice have been less suc- in contrast, is a “mathematical derivation of cessful in explaining or predicting behavior what players with different cognitive capabil- in one-shot or finitely repeated social dilem- ities are likely to do in games” (3). Collec- mas in which the theoretical prediction is that tive action game theory was dominated until no one will cooperate (Poteete, Janssen, and recently by the view of short-term rational Ostrom 2010, ch. 6; Ostrom 2010). Research self-interest exposited by Olson (1965). using observational data also shows that some In the PD game, if there are no possibil- groups of individuals do engage in collective ities of enforceable binding contracts, then action to provide local PGs or manage CPRs this view predicts defection because it is a without external authorities offering induce- strictly dominant strategy (Fudenberg and ments or imposing sanctions (see National Tirole 1991, 10). This prediction can change Research Council [NRC] 2002). if repeated play is allowed. A long stream of political science research, led by Axelrod (1984), has examined repeated PD games and 1. Cooperation in the Prisoner’s found that if play is repeated indefinitely, then Dilemma cooperative strategies are theoretically possible. However, if there is a predetermined One way to conceptualize a collective action end to repetition, then the strictly dominant dilemma is the prisoner’s dilemma (PD) game theoretical strategy remains to defect in each shown in Figure 24.1. Imagine that two states round (Axelrod and Hamilton 1981, 1392). engage in a nuclear arms race. There are two This view of human behavior led many to players (the two states) who choose simul- conclude that without external enforcement taneously to either cooperate (reduce arma- of binding contracts, or some other change ments) or defect (build armaments). If only in institutional arrangements, humans will one state cooperates, then it will be at a not cooperate in collective action dilemmas strategic disadvantage. In this case, the out- (Schelling 1960, 213–14; Hardin 1968). Pre- come (cooperate, defect) has a payout of −1 dictions from the PD game generalize to the to the cooperating state. The defecting state CPR and PG dilemmas discussed in this chap- receives a payout of 2 because of strategic ter – little or no cooperation if they are one gains. If both states cooperate, then each shot or finitely repeated and a possibility of would receive a payout of 1 because they do cooperation only in dilemmas with indefinite not bear the costs of building armaments. repetition. If neither cooperates, then both receive a The behavioral revolution in economics payout of zero because neither has gained a and political science led some to question strategic advantage. In short, states are best these predictions (see Ostrom 1998). Experi- off if they can dupe others into cooperating mental game theory has been indispensible in while they defect, moderately well off if they challenging the conventional view of human both cooperate, worse off if they both defect, behavior and improving game-theoretic pre- and in very bad trouble if they are the sole dictions in collective action dilemmas. As cooperator. behaviors inconsistent with predictions from Experimental Contributions to Collective Action Theory 341 the conventional view are uncovered, ana- We view field and experimental work as lysts change their theories to incorporate the complementary; that is, we have more con- anomalies. For example, some authors have fidence in experimental results because they examined preferences for inequality aver- are confirmed by fieldwork and vice versa. sion (Fehr and Schmidt 1999), preferences Although observational studies make impor- for fairness (Rabin 1993), and emotional tant contributions to our understanding of states (Cox, Friedman, and Gjerstad 2007)to what determines whether a group acts collec- explain behavior in collective action dilem- tively, they are limited in that there are a host mas (see also Camerer 2003, 101–13). of confounding factors that might account for In the next section, we briefly discuss how the effects. Experimental work is uniquely field evidence suggested that the conventional suited to isolate and identify these effects, model did not adequately explain behavior and then to calibrate new models of human in collective action dilemmas. The careful behavior. control possible in experiments is uniquely Let us now turn to experimental contri- suited both to uncovering the precise degree butions to collective action theory. Many of anomalous behavior and for calibrating types of experiments involve collective action new models. As these new models of behavior dilemmas for the subjects involved, includ- are developed, they should then be subjected ing Trust Game experiments (see Wilson and to the same rigorous experimental and nonex- Eckel’s chapter in this volume), Ultimatum perimental testing as the conventional model. Game experiments, and a host of others (see Camerer 2003 for a review). In this chapter, we focus on two games that have received Evidence from Observational Studies much attention in the experimental literature. Much has been written about collective action In Section 2, we review research and contri- in observational studies (NRC 1986, 2002). butions from PG experiments, and in Section These studies are particularly useful for 3, we review CPR experiments. In Section 4, experimentalists because they provide exam- we discuss the emerging role of laboratory ples of behaviors and strategies that peo- experiments in the field, and in Section 5,we ple employ in real settings to achieve coop- conclude. eration. Experiments have then made these behaviors possible in laboratory settings in order to assess exactly how effective they can 2. Public Goods Experiments be. For example, much early work indicated that people are willing to sanction nonco- The PD game discussed in Section 1 isaspe- operators at a personal cost to themselves cial case of a PG game. Suppose that instead (Ostrom 1990). This literature influenced of cooperate and defect, we labeled these experimentalists to create the possibility of columns “contribute” and “withhold.” The allowing sanctioning in laboratory environ- game is structured such that a public good is ments (Ostrom, Gardner, and Walker 1994; provided to all players in proportion to the Fehr and Gachter¨ 2000), which led to the number of players who contribute. Suppose development of the inequality aversion model that the public good can be monetized as $4 (Fehr and Schmidt 1999). Other variables per player contribution. If both players confound in observational research that have tribute, each receives $4. If only one player influenced experimental work are group size contributes, each receives $2. Suppose further and heterogeneity (Olson 1965), rewarding that it costs each player $3 to contribute. This cooperators (Dawes et al. 1986), communi- is the same game structure as depicted in the cation and conditionally cooperative strate- PD game in Figure 24.1.1 We can also write gies (Ostrom 1990), and interpersonal trust (Rothstein 2005). By and large, these same 1 If both players contribute, then they each receive $1 ($4 from the public good minus $3 of contribu- factors appear to be important in experimen- tion costs). If only one player contributes, that player tal studies. receives –$1 ($2 from the public good minus $3 of 342 Eric Coleman and Elinor Ostrom the payouts from this PG game in equation action dilemma. The closer the MPCR is to ∈{ , } 2 form. Let Ci 0 1 represent the decision 1, the higher the benefits from cooperation. to contribute, so that Ci takes the value of 1 The experimental protocols for PG games if player i decides to contribute and Ci takes are typically abstract, instructing subjects to the value of 0 if player i decides to withhold. allocate their endowment to a group or an The payment to player i in this one-shot PD individual fund. The individual fund has a game is rate of return equal to B. The rate of return on the group fund is equal to the MPCR. The 4 + π = (Ci C j ) − 3 . 1 PG game is also often referred to as a volun- i 2 Ci ( ) tary contribution mechanism (VCM) because subjects make voluntary contributions to this If player i maximizes his or her own income, group fund. then he or she will select the level of Ci that maximizes Equation (1). In this case, the pre- Baseline Public Goods Experiments diction is that Ci = 0 because the net effect of one person’s contribution is −1.Letus In the baseline PG experiment, subjects are relax some of the assumptions from Equa- each endowed with the same number of tion (1) to develop a general PG game. First, tokens and receive the same MPCR and the we add n players to the game and relax the same rate of return to their private accounts. assumption that the decision to contribute is If one assumes that subjects behave accord- binary. That is, instead of contributing or not ing to narrow self-interest, then one would contributing, suppose that the subjects can expect no contributions in any round of the determine a specific amount to contribute, game. Ci. In general, Ci can be allowed to vary up In one of the first PG experiments, Isaac to some initial endowment and can take any and Walker (1988a) endowed each subject, value between 0 and the endowment, Ei.Let in groups of four, with sixty-two tokens and the marginal benefits to unilateral contribu- repeated play for twenty rounds. The MPCR 0 003 tion be any value Ai. Let us call the costs of was $ . per token, whereas the return to 0 01 contributing Bi. The general PG game, then, the private account was $ . . The dotted line is (NC–NC) at the bottom of Figure 24.2 shows that in the first round, subjects contributed n about fifty percent of their endowment to π = Ai − , i C j Bi Ci the public good. Over time, these contribu- n =1 j tions steadily fall toward zero. This result is fairly robust, having been replicated in a where number of studies (Isaac and Walker 1988b, 184). This result appears to confirm the tra- ∈ 0, . 2 Ci [ Ei ] ( ) ditional model of narrow self-interest, especially if the anomalies at the beginning rounds The parameter Bi is often set to 1, and Ai is set can be attributed to learning (see Muller et al. to be the same for every player. In this case, 2008). the primary characteristic used to describe the game is the ratio of marginal benefits to the A Communication number of subjects, n . This ratio is known as the marginal per capita return (MPCR). If When participants in the CPR experi- B = 1,thenA must be less than n for this to be ment cannot communicate, their behavior a PG game and for this to remain a collective approaches zero contributions over time. Participants in most field settings, however, contribution costs), but the noncontributing player receives $2 ($2 from the public good and no expense incurred from contributing). If neither player con- 2 See Isaac and Walker (1988b) for results related to tributes, both receive zero dollars. changing the relative size of the MPCR. Experimental Contributions to Collective Action Theory 343

Mean Contributions by Treatments and Rounds 1 .2 .4 .6 .8 1 0 1 .2.4.6.8 1 2 3 4 5 6 7 8 9 11 10 12 13 14 15 16 17 18 19 20 C–NC NC–C NC–NC C–NC First 10 Rounds NC–C Second 10 Rounds NC–NC

Figure 24.2. Contributions in a Public Goods (PGs) Game Source: Isaac and Walker (1988a). The left panel shows the proportion of contributions to the group fund in a PG experiment, by round, for three treatments. Rounds were broken into two halves of ten rounds each. In the C–NC treatment, communication was allowed in each of the first ten rounds, but no communication was allowed in the last ten rounds. In the NC–C treatment, no communication was allowed in any of the first ten rounds, but communication was allowed in each of the last ten rounds. In the NC–NC treatment, no communication was allowed in any round. The right panel shows the proportion of contributions to the group fund by halves of the experiment for each treatment, as well as ninety-five percent confidence intervals for those means. are able to communicate with one another dicted to have no effect on outcomes in the at least from time to time, either in formally PG game (Harsanyi and Selten 1988, 3). constituted meetings or at social gatherings. Figure 24.2 clearly shows, however, that In an effort to take one step at a time toward communication has a profound effect. Take the fuller situations faced by groups provid- the case where communication was allowed ing PGs, researchers have tried to assess the in the last ten rounds of play, that is, the effects of communication. dashed line in the left panel of Figure 24.2. Figure 24.2 shows the results from two It appears for the first ten rounds that the additional treatments in Isaac and Walker subjects are on a similar trajectory as those (1988b). In the treatment C–NC, subjects who are never allowed to communicate; in were allowed to communicate at the begin- other words, mean contributions to the pub- ning of each of the first ten rounds, but were lic good are steadily falling. After communi- not allowed to communicate thereafter. In the cation in round 10, however, mean contri- treatment NC–C, subjects were not allowed butions increase substantially. In the second to communicate for the first ten rounds, but half of the game, contributions are near 100 starting in round 11 were allowed to com- percent of the total endowments. The right municate in every round thereafter. In the panel of Figure 24.2 shows that the mean baseline treatment (NC–NC), described in contributions are significantly higher in the the previous section, no communication was second ten rounds. Perhaps more astonish- allowed in any round. Nonbinding commu- ing is that when communication is allowed in nication is referred to as cheap talk and is pre- the first ten rounds, contributions continue 344 Eric Coleman and Elinor Ostrom to remain high in the second ten rounds of other participants’ responses. In particu- (the solid line in the left panel of Figure lar, if a participant believes that other par- 24.2), although this tends to taper off in the ticipants are of a cooperative type (i.e., will last three rounds. Still, mean contributions cooperate in response to cooperative play), remain essentially the same across ten-round then that participant may play cooperatively increments, as indicated in the right panel to induce cooperation from others. In this of Figure 24.2. Such strong effects of com- case, cooperating can be sustained as rational munication have been found in many studies play in the framework of incomplete infor- (Sally 1995). In fact, Miettinen and Suetens mation regarding participant types. (2008) argue that “researchers have reached a rather undisputed consensus about the prime Leadership driving force of the beneficial effect of communication on cooperation” (945). Political leadership has long been linked Subject communication tends to focus to the provision of PGs (Frohlich, Oppen- around strategies for the game. Often, sub- heimer, and Young 1971). Economists have jects will agree on some predetermined traditionally modeled leadership in PG set- behavior. Although they frequently do what tings as “leading by example” (Levati, Sutter, they promise, some defections do occur. If and van der Heijden 2007). That is, one sub- promises were not kept, subjects use the ject is randomly selected from the experiment aggregated information on the outcomes to be a first mover in the PG game. Other from the previous round to castigate the subjects are then hypothesized to take cues unknown participant(s) who does not keep from the first mover. Evidence suggests that to their agreement. Subjects can be indignant this type of leadership increases cooperation about evidence of defection and express their in PG games (Levati et al. 2007). anger openly. Not only does the content of Leading by example, however, seems to communication matter, but the medium of be a coarse operationalization of leadership. communication is also important. Frohlich Experimental research would benefit from and Oppenheimer (1998, 394) found that more thoughtful insights from political lead- those allowed to communicate face-to-face ership theory, and these insights could be reach nearly 100 percent contribution to carefully examined by endowing leaders with the public good, whereas communication different capabilities. For example, character- via e-mail improves contributions to about istics of Machiavelli’s prince might be manip- 75 percent.3 ulated in the laboratory. Is it truly better to Although the findings show that commu- be feared than loved? Do leaders who devise nication makes a major difference in out- punishments for those who do not contribute comes, some debate exists as to why commu- to the public good fare better than those who nication alone leads to better results (Buchan, offer rewards?4 Johnson, and Croson 2006). A review by In addition, although much research has Shankar and Pavitt (2002) suggests that voic- been conducted on the election process ing of commitments and development of of political leadership (see Morton and group identity and norms seem to be the Williams’ chapter in this volume), the effects best explanations for why communication of such institutions on PG provision have makes a difference. Another reason may be not been thoroughly explored. One notable the revelation of a participant’s type, which exception is a recent paper by Hamman, is one source of incomplete information Woon, and Weber (2008). The authors in experimental games. For example, face- investigate the effects of political leadership to-face communication and verbal commit- by forcing groups to delegate authority to ments may change participants’ expectations one elected (majority rule) leader, who then

3 See Sally (1995) for a meta-analysis relating commu- 4 For the PG game, see Sefton, Shupp, and Walker nication treatments to cooperation in experimental (2007); for the CPR game, see van Soest and games. Vyrastekova (2006). Experimental Contributions to Collective Action Theory 345 determines the contributions of each mem- ket 2, which functioned like a CPR so that the ber to the public good. The delegate is then return was determined in part by the actions reelected in subsequent periods, ensuring of all participants in the experiment. some accountability to other group members. Each participant i could choose to invest ω The authors find that under delegated PG a portion xi of his or her endowment of provision, groups elect delegates who ensure in the common resource market 2, and the ω that the group optimum is almost always remaining portion – xi is then invested in met. market 1. The payoff function used in Ostrom et al. (1994)is ⎧ . · ω = 3. Common Pool Resource ⎨⎪ 0 05 if xi 0 Experiments 0.05 · ω − > 0 u (x) = ( xi )ifxi (3) i ⎩⎪ + / · Common pool resources such as lakes, (xi xi ) F( xi ) forests, fishing grounds, and irrigation systems are resources from which one person’s where use subtracts units that are then not avail- 8 able to others, and it is difficult to exclude or = 23 · − 0.25· limit users once the resource is provided by F xi xi i=1 ⎞ nature or produced by humans (Ostrom et al. 2 1994). When access to a CPR is open to all, 8 ⎠ /100. 4 anyone who wants to use the resource has an xi ( ) incentive to appropriate more resource units i=1 when acting independently than if he or she could find some way of coordinating his or According to this formula, the payoff of her appropriation activities with others. someone investing all ω tokens in market 1 = × ω CPR games are different from PG games (xi 0)is0.05 . The payoff from market in two ways: 1) the decision task of a CPR is 1 is like a fixed wage paid according to the removing resources from a joint fund instead hours devoted to working. Investing part or > of contributing, and 2) appropriation is rival- all tokens in market 2 (xi 0) yields an out- rous. This rivalry can be thought of as an come that depends on the investments of the externality that occurs because the payout other participants. rate from the CPR depends nonlinearly on Basically, if appropriators put all their total group appropriation. Initially, it pays assets into the outside option (working for to withdraw resources from the CPR, but a wage rather than fishing), then they are subjects maximize group earnings when they certain to receive a fixed return equal to invest some, but not all, of their effort to the amount of their endowment times an appropriate from the CPR. unchanging rate. If appropriators put some of The first series of CPR experiments was their endowed assets into the CPR, then they initiated at Indiana University to complement received part of their payoff from the outside ongoing fieldwork. The series started with a option and the rest from their proportional static, baseline situation that was as simple as investment in the CPR. The participants possible while keeping crucial aspects of the received aggregated information after each problems that real harvesters face. The pay- round, so they did not know individual off function used in these experiments was a actions. Each participant was endowed with quadratic function similar to the theoretical a new set of tokens in every round of play. function specified by Gordon (1954). The ini- Their outside opportunity was valued at tial resource endowment of each participant $0.05 per token. They earned $0.01 on each consisted of a set of tokens that the partici- outcome unit they received from investing pants could allocate between two situations: tokens in the CPR. The number of rounds in market 1, which had a fixed return, and mar- each experiment varied between twenty and 346 Eric Coleman and Elinor Ostrom thirty rounds, but participants were informed approach the symmetric Nash equilibrium that they were in an experiment that would level in later rounds. last no more than two hours. The solid line in Figure 24.3 shows the Voting relationship between total group investments in market 1, the fixed wage rate, and group If subjects are allowed to make binding agree- earnings from that market. The dashed ments about their behavior in the CPR game, line shows the relationship between group they might overcome the free rider problem. investments in market 2, the CPR, and its People may be willing to voluntarily precom- group earnings. Wage earnings are inter- mit to limit the choices available to the group preted as the opportunity costs of investing in the future in order to achieve a more pre- in the CPR. Total earnings, represented ferred group outcome (Elster 1977). by Equation (3), are maximized when the Ostrom et al. (1994) investigated whether CPR earnings minus wage earnings are subjects were willing to precommit to binding maximized. Given the parameterization of contracts in the CPR game and whether those Equation (4), this occurs at total investment contracts would produce efficient results. in the CPR of thirty-six tokens. The authors gave groups of seven subjects The symmetric Nash equilibrium for this an opportunity to use simple majority rule to finitely repeated game (if subjects are not dis- develop an appropriation system for them- counting the future, and each participant is selves. In the lab, they found people moving assumed to be maximizing his or her own toward a minimum winning coalition. Sub- monetary returns) is for each participant to jects knew the computer numbers and began invest eight tokens in the CPR for a total to make proposals such as “Let’s give all of sixty-four tokens (see Ostrom et al. 1994, the optimal resources to computer number 111–12). They could, however, earn consid- one, two, three, and four.” Of course, this erably more if the total number of tokens came from somebody who was using com- invested were thirty-six tokens (rather than puter number one, two, three, and four; and sixty-four tokens). The baseline experiment is they zeroed out five, six, and seven. When the an example of a commons dilemma in which voting rule was changed to require unanim- the Nash equilibrium outcome involves sub- ity, the subjects also went to the optimum, stantial overuse of a CPR, while a much better but they allocated it across the entire group. outcome could be reached if participants were to lower their joint use relative to the Nash Sanctioning equilibrium. In the field, many users of CPRs do monitor and sanction one another (Coleman and Baseline Common Pool Resource Steed 2009). Engaging in costly monitoring Experiments and sanctioning behavior is not consistent Participants interacting in baseline experi- with the theory of norm-free, full rational- ments substantially overinvested as predicted. ity (Elster 1989, 40–41). At the individual level, participants rarely To test whether participants would use invested eight tokens – the predicted level their own resources to sanction others, of investment at the symmetric Nash equi- Ostrom, Walker, and Gardner (1992) con- librium. Instead, all experiments provided ducted a modified CPR game. Individual evidence of an unpredicted and strong investments in each round, as well as the pulsing pattern in which individuals appear total outcomes, were reported.5 Participants to increase their investments in the CPR were then told that in the subsequent rounds, until there is a strong reduction in yield, they would have an opportunity to pay a at which time they tend to reduce their investments, leading to an increase in yields. 5 See Fehr and Gachter¨ (2000) for an application in PG At an aggregate level, behavior begins to games. Experimental Contributions to Collective Action Theory 347

Efficient Level of Symmetric Harvest Nash Equilibrium 0123456 0204060 80 Group Investment Private Returns CPR Returns

Figure 24.3. A Common Pool Resource (CPR) Game The dashed black line represents total group earnings from the CPR for all levels of group investment, and the solid black line represents earnings from the private fund. Earnings from the private fund are the opportunity costs of earnings from the CPR. The efficient level of investment in the CPR, that is, that maximizes group earnings from the CPR (CPR earnings minus the opportunity costs from the private fund), is a group investment of thirty-six tokens (see Ostrom et al. 1994). Note that net earnings from the CPR are negative when group investment equals seventy-two tokens. The symmetric Nash equilibrium is to invest eight tokens each, or sixty-four total tokens, in the CPR. fee in order to impose a fine on the payoffs to investigate collective action theory. Many received by another participant. In brief, the collective action dilemma experiments have finding from this series of experiments was been conducted in developed countries with that much sanctioning occurs. Most sanctions undergraduate students from university set- were directed at subjects who had ordered tings. The initial reasons for this selected high levels of tokens. Participants react both sample of participants were their accessibil- to the cost of sanctioning and to the fee– ity, control for the experimenters, and lower fine relationships. They sanction more when overall costs.6 Experiments have now been the cost of sanctioning is less and when the conducted with nonstudent populations and ratio of the fine to the fee is higher (Ostrom with more salient frames of the decision tasks, et al. 1992). Participants did increase benefits and there are often striking differences in through their sanctioning, but reduced their behavior across these populations (Henrich net returns substantially due to the high use et al. 2004). of costly sanctions. Because of the increased costs and logistical problems associated with these types of experiments, researchers should think 4. Bringing the Lab to the Field 6 Even though laboratory experiments conducted in a We believe that laboratory experiments in university setting usually pay participants more than they would earn in a local hourly position, the total field settings hold a challenging, yet poten- costs of the experiment itself are usually substantially tially fruitful avenue for political scientists less than experiments conducted in field settings. 348 Eric Coleman and Elinor Ostrom carefully about the reasons for extending their game, a subject is given some sum of money research to field settings. Harrison and List and is simply asked to divide the money with (2004) argue that key characteristics of sub- a partner. The subject can give all, none, or jects from the experimental sample need to anything in between.) match the population for which inferences Second, they tested the effects of techno- will be generalized. That is, if age, education, logical advantages of homogeneous groups. or some political or cultural phenomenon Such groups can draw on common language unique to students is not a key characteris- and culture to produce PGs and are bet- tic of the theory being tested, then a stu- ter able to identify noncooperative members. dent sample may be appropriate to test the To test the first proposition, that coethnics theory (see Druckman and Kam’s chapter in work well together, the authors had subject this volume). However, if one wants to inves- pairs solve puzzles. They found that whereas tigate the effects of communism, for exam- coethnic pairs were more likely to solve the ple, then a sample of U.S. students would puzzle than noncoethnic pairs, the difference not be appropriate; rather, an older age sam- was not significant. The second submecha- ple from a postcommunist country would nism is that members of homogeneous groups be needed for the experiment (Bahry et al. can find and identify noncooperative mem- 2005). bers through social networks. The authors had subjects locate randomly selected nonexperimental subjects as “targets” in Kampala. Ethnic Diversity and the Mechanisms Those of the same ethnicity as their target of Public Goods Provision found the target forty-three percent of the Habyarimana et al. (2007) were interested in time, whereas those of different ethnicities why ethnic heterogeneity leads to decreased found the target only twenty-eight percent of investments in PGs. To test a number of the time. possible mechanisms, the authors conducted Third, ethnicity might serve to coordinate a set of surveys and experiments using 300 strategies through social sanctioning. To test randomly selected subjects recruited from a this mechanism, the authors had subjects play slum in Kampala, Uganda. It was necessary to a nonanonymous dictator game. The authors use such subjects because “ethnicity is highly reason that in such a game, if subjects give salient in everyday social interactions,” and nothing to their partner, then they might still the subjects had almost exclusive responsibil- be subject to social shame for acting non- ity for supplying local PGs (712). cooperatively. The authors found that cer- The authors identified three different tain types of subjects discriminate their giving potential mechanisms. First, they tested the based on ethnicity when play is not anony- effects of ethnic heterogeneity of tastes – mous. That is, they give less to noncoethnics the extent to which different ethnic groups than they do to coethnics. care about different types of PGs and their In this study, experimental methods preferences that PGs are provided to their allowed the researchers to carefully parse own ethnic group and not others. Using a out and test different causal mechanisms survey instrument, the authors found that that explain why ethnically heterogeneous there was little difference in tastes both as groups provide fewer PGs than homoge- to which types of PGs subjects preferred neous groups. Experimental methodology (drainage, garbage collection, or security) or was needed to explore these mechanisms to the means of their provision (government because all three seem equally plausible when vs. local). The authors then had subjects play analyzing observational data. In addition, an anonymous dictator game to test if non- the field setting allowed the authors to test coethnic pairs have different tastes for income these theories with samples from a popula- distribution than coethnic pairs and found tion where ethnic diversity was a major factor that this was not the case. (In the dictator in public goods provision. Experimental Contributions to Collective Action Theory 349

should spend no more than the optimal level Social Norms and Cultural Variability of time in the forest during each round (Car- in Common Pool Resources denas et al. 2000). Subjects were also told that An interesting series of replications and ex- there would be a 50 percent chance that some- tensions of CPR experiments has been con- one would be monitored during each round. ducted by Cardenas and colleagues (Cardenas The experimenter rolled a dice in front of 2000, 2003; Cardenas, Stranlund, and Willis the participants during each round to deter- 2000) using field laboratories set up in villages mine whether the contributions of any partic- in rural Colombia. The villagers whom Car- ipant would be monitored. If an even num- denas invited were actual users of local forests. ber appeared, someone would be inspected. He wanted to assess whether experienced vil- The experimenter then drew a number from lagers, who were heavily dependent on local chits numbered from one to eight placed in forests for wood products, would behave in a a hat to determine who would be inspected. manner broadly consistent with that of under- Thus, the probability that anyone would be graduate students in a developed country. inspected was one sixteenth per round – a The answer to this first question turned low but realistic probability for monitoring 7 out to be positive. Cardenas asked villagers forest harvesting in rural areas. The monitor to decide on how many months a year they checked the investment of the person whose would spend in the forest gathering wood number was drawn. A penalty was subtracted products in contrast to using their time oth- from the payoff of anyone over the limit, erwise. Each villager had a copy of an iden- and no statement was made to others as to tical payoff table. In the baseline, no com- whether the appropriator was complying with munication experiments, Cardenas found a regulations. pattern similar to previous findings from the The participants in this experiment with baseline CPR experiments. Villagers substan- a rule to withdraw the “optimal” amount tially overinvested in appropriation from the imposed on them actually increased their with- resource. drawal levels in contrast to behavior when no Face-to-face communication enabled the rule was imposed and face-to-face communi- villagers to increase total earnings on average cation was allowed. Thus, participants who from 57.7 percent to 76.1 percent of optimal. were simply allowed to communicate with Subjects filled in surveys after completing the one another on a face-to-face basis were able experiments; Cardenas used these to explain to achieve a higher joint return than those the considerable variation among groups. who had an optimal but imperfectly enforced He found, for example, that when most external rule imposed on them. members of the group were already familiar with resources, they used the commu- Some Considerations nication rounds more effectively than when most members of the group were depen- Although investigating the differences in dent primarily on individual assets. Carde- experimental behavior across societies holds nas also found that “social distance and group the potential for important new insights into inequality based on the economic wealth of collective action theory, one would be remiss the people in the group seemed to constrain without mentioning some of the ethical con- the effectiveness of communication for this cerns attendant to such research. Generally, same sample of groups” (Cardenas 2000, 317; payments for participation in these experi- see also Cardenas 2003). In five other exper- ments are large compared to local wage rates. iments, subjects were told that a new regula- Average payments in these games generally tion would go into force mandating that they range from one half of a day’s wage to a week’s wage, although in some instances the stakes 7 Although see Henrich and Smith (2004) for a coun- are even much greater. Researchers should terexample. consider both benefits that subjects receive 350 Eric Coleman and Elinor Ostrom from participating, as well as the potential for Bahry, Donna, Michail Kosoloapov, Polina conflict if some subjects are dissatisfied with Kozyreva, and Rick K. Wilson. 2005. “Ethnic- the results. Every effort should be taken to ity and Trust: Evidence from Russia.” American 99 521 32 ensure that earnings remain anonymous. Political Science Review : – . Bentley, Arthur F. 1949. The Process of Government. Evanston, IL: Principia Press. 5. Conclusion Buchan, Nancy R., Eric J. Johnson, and Rachel T. A. Croson. 2006. “Let’s Get Personal: An Even though much important work has International Examination of the Influence of already been done in collective action Communication, Culture and Social Distance on Other Regarding Preferences.” Journal of experiments, interesting questions remain. It 60 373 98 is perhaps not surprising that considerable Economic Behavior and Organization : – . Camerer, Colin F. 2003. Behavioral Game Theory: variation in behavior is recorded in these Experiments in Strategic Interaction. Princeton, experiments across different societies. How- NJ: Princeton University Press. ever, what is unclear is explaining the cultural Cardenas, Juan-Camilo. 2000. “How Do Groups and political dimensions driving these differ- Solve Local Commons Dilemmas? Lessons ences. Political scientists can make important from Experimental Economics in the Field.” contributions to understanding such behavior Environment, Development and Sustainability 2: by reference to variation in political phenom- 305–22. ena at the local and national level. Political Cardenas, Juan-Camilo. 2003. “Real Wealth and corruption, for example, may be very impor- Experimental Cooperation: Evidence from Field Experiments.” Journal of Development Eco- tant for determining why subjects in some 70 263 89 societies are more cooperative than subjects nomics : – . Cardenas, Juan-Camilo, John K. Stranlund, and in others (Rothstein 2005). Cleve E. Willis. 2000. “Local Environmen- Important advances can also be made in tal Control and Institutional Crowding-Out.” understanding the role of different political World Development 28: 1719–33. structures and the incentives that they pro- Coleman, Eric A., and Brian Steed. 2009.“Mon- vide in CPR and PG games. We do not itoring and Sanctioning on the Commons: An understand, for example, what effect differ- Application to Forestry.” Ecological Economics ent voting rules have on the propensity to 68: 2106–13. delegate punishment authority or allocative Cox, James C., Daniel Friedman, and Steven Gjer- 2007 authority and what effects this may have on stad. . “A Tractable Model of Reciprocity and Fairness.” Games and Economic Behavior 59: cooperation. Research has yet to be done 17 45 that examines the effects of oversight, a – . Dawes, Robyn M., John M. Orbell, Randy T. Sim- third-order collective action dilemma, on the mons, and Alphons J. C. van de Kragt. 1986. propensity to sanction and the subsequent “Organizing Groups for Collective Action.” collective action outcome. There is still much American Political Science Review 80: 1171–85. work to be done examining the role of differ- Elster, Jon. 1977. “Ulysses and the Sirens: A The- ent institutional arrangements on collective ory of Imperfect Rationality.” Social Science action. Information 16: 469–526. Elster, Jon. 1989. Solomonic Judgments: Studies in the Limitations of Rationality. New York: Cam- References bridge University Press. Fehr, Ernst, and Simon Gachter.¨ 2000. “Cooper- Alchian, Armen A. 1950. “Uncertainty, Evolu- ation and Punishment in Public Goods Exper- tion, and Economic Theory.” Journal of Political iments.” American Economic Review 90: 980–94. Economy 58: 211–21. Fehr, Ernst, and Klaus M. Schmidt. 1999.“A Axelrod, Robert. 1984. The Evolution of Cooperation. Theory of Fairness, Competition, and Coop- New York: Basic Books. eration.” Quarterly Journal of Economics 114: Axelrod, Robert, and William D. Hamilton. 1981. 817–68. “The Evolution of Cooperation.” Science 211: Frohlich, Norman, and Joe A. Oppenheimer. 1390–96. 1998. “Some Consequences of E-Mail vs. Experimental Contributions to Collective Action Theory 351

Face-To-Face Communication in Experi- Isaac, R. Mark, and James M. Walker. 1988b. ment.” Journal of Economic Behavior and Orga- “Group Size Effects in Public Goods Pro- nization 35: 389–403. vision: The Voluntary Contribution Mecha- Frohlich, Norman, Joe A. Oppenheimer, and Oran nism.” Quarterly Journal of Economics 103: 179– R. Young. 1971. Political Leadership and Collec- 99. tive Goods. Princeton, NJ: Princeton University Levati, M. Vittoria, Matthias Sutter, and Eline van Press. der Heijden. 2007. “Leading by Example in a Fudenberg, Drew, and Jean Tirole. 1991. Game Public Goods Experiment with Heterogeneity Theory.Cambridge,MA:MITPress. and Incomplete Information.” Journal of Con- Gordon,H.Scott.1954. “The Economic Theory ﬂict Resolution 51: 793–818. of a Common Property Resource: The Fish- Miettinen, Topi, and Sigrid Suetens. 2008. ery.” Journal of Political Economy 62: 124–42. “Communication and Guilt in a Prisoner’s Habyarimana, James, Macartan Humphreys, Dilemma.” Journal of Conﬂict Resolution 52: Daniel N. Posner, and Jeremy M. Weinstein. 945–60. 2007. “Why Does Ethnic Diversity Under- Muller, Laurent, Martin Sefton, Richard Stein- mine Public Goods Provision?” American Polit- berg, and Lise Vesterlung. 2008. “Strategic ical Science Review 101: 709–25. Behavior and Learning in Repeated Voluntary Hamman, John, Jonathan Woon, and Roberto Contribution Experiments.” Journal of Economic A. Weber. 2008. “An Experimental Investi- Behavior and Organization 67: 782–93. gation of Delegation, Voting, and the Pro- National Research Council (NRC). 1986. Pro- vision of Public Goods.” Paper presented ceedings of the Conference on Common Prop- at the Graduate Student Conference on erty Resource Management. Washington, DC: Experiments in Interactive Decision Making, National Academies Press. Princeton University, Princeton, NJ. Retrieved National Research Council (NRC). 2002. The from http://myweb.fsu.edu/jhamman/doc/Pub Drama of the Commons, eds. Elinor Ostrom, Goods.pdf. Accessed February 28, 2011. Thomas Dietz, Nives Dolsak,ˇ Paul Stern, Susan Hardin, Garrett. 1968. “The Tragedy of the Com- Stonich, and Elke Weber. Committee on the mons.” Science 162: 1243–48. Human Dimensions of Global Change. Wash- Hardin, Russell. 1982. Collective Action. Baltimore: ington, DC: National Academies Press. The Johns Hopkins University Press. Olson, Mancur. 1965. The Logic of Collective Action: Harrison, Glenn W., and John A. List. 2004. Public Goods and the Theory of Groups. Cam- “Field Experiments.” Journal of Economic Lit- bridge, MA: Harvard University Press. erature 42: 1009–55. Ostrom, Elinor. 1990. Governing the Commons: The Harsanyi, John C., and Reinhard Selten. 1988. A Evolution of Institutions for Collective Action.New General Theory of Equilibrium Selection in Games. York: Cambridge University Press. Cambridge, MA: MIT Press. Ostrom, Elinor. 1998. “A Behavioral Approach Henrich, Joseph, Robert Boyd, Samuel Bowles, to the Rational Choice Theory of Collective Colin Camerer, Ernst Fehr, and Herbert Gin- Action.” American Political Science Review 92: 1– tis, eds. 2004. Foundations of Human Social- 22. ity: Economic Experiments and Ethnographic Evi- Ostrom, Elinor. 2010. “Beyond Markets and dence from Fifteen Small-Scale Societies. Oxford: States: Polycentric Governance of Complex Oxford University Press. Economic Systems.” American Economic Review Henrich, Joseph, and Natalie Smith. 2004.“Com- 100: 641–72. parative Experimental Evidence from the Ostrom, Elinor, Roy Gardner, and James Walker. Machiguenga, Mapuche, Huinca, and Amer- 1994. Rules, Games, and Common-Pool Resources. ican Populations.” In Foundations of Human Ann Arbor: University of Michigan Press. Sociality: Economic Experiments and Ethnographic Ostrom Elinor, James Walker, and Roy Gardner. Evidence from Fifteen Small-Scale Societies, eds. 1992. “Covenants with and without a Sword: Joseph Henrich, Robert Boyd, Samuel Bowles, Self-Governance Is Possible.” American Political Colin Camerer, Ernst Fehr, and Herbert Gin- Science Review 86: 404–17. tis. Oxford: Oxford University Press, 125–67. Poteete, Amy R., Marco A. Janssen, and Eli- Isaac, R. Mark, and James M. Walker. 1988a. nor Ostrom. 2010. Working Together: Collective “Communication and Free-Riding Behavior: Action, the Commons, and Multiple Methods in The Voluntary Contribution Mechanism.” Practice. Princeton, NJ: Princeton University Economic Inquiry 26: 585–608. Press. 352 Eric Coleman and Elinor Ostrom

Rabin, Matthew. 1993. “Incorporating Fairness Shankar, Anisha, and Charles Pavitt. 2002. into Game Theory and Economics.” American “Resource and Public Goods Dilemmas: A New Economic Review 83: 1281–302. Issue for Communication Research.” Review of Rothstein, Bo. 2005. Social Traps and the Problem of Communication 2: 251–72. Trust. New York: Cambridge University Press. Smith, Vernon L. 1991. Papers in Experimental Sally, David. 1995. “Conversation and Coopera- Economics. New York: Cambridge University tion in Social Dilemmas: A Meta-Analysis of Press. Experiments from 1958 to 1992.” Rationality Taylor, Michael. 1987. The Possibility of Coop- and Society 7: 58–92. eration. New York: Cambridge University Satz, Debra, and John Ferejohn. 1994. “Rational Press. Choice and Social Theory.” Journal of Philoso- Truman, David B. 1958. The Governmental Process. phy 91: 71–87. New York: Knopf. Schelling, Thomas. 1960. The Evolution of Coop- van Soest, Daan, and Jana Vyrastekova. 2006. eration. Cambridge, MA: Harvard University “Peer Enforcement in CPR Experiments: The Press. Relative Effectiveness of Sanctions and Trans- Sefton, Martin, Robert Shupp, and James Walker. fer Rewards and the Role of Behavioral Types.” 2007. “The Effect of Rewards and Sanctions in In Using Experimental Economics in Environmen- Provision of Public Goods.” Economic Inquiry tal and Resource Economics, ed. John List. Chel- 45: 671–90. tenham, UK: Edward Elgar, 113–36. CHAPTER 25

Legislative Voting and Cycling

Gary Miller

Discoveries regarding the scope and mean- ity. In general, majority rule may not be able ing of majority rule instability have informed to produce a majority rule winner (an out- debate about the most fundamental ques- come that beats every other in a two-way tions concerning the viability of democracy. vote). Rather, every possible outcome could Are popular majorities the means of serv- lose to something preferred by some major- ing the public interest or a manifestation ity coalition. McKelvey (1976) showed that of the absence of equilibrium (Riker 1982)? the potential for instability was profound, Should majority rule legislatures be suspect not epiphenomenal. A population of voters or even avoided in favor of court decisions with known preferences might easily choose and bureaucratic delegation? Are the machi- any outcome, or different outcomes at differ- nations of agenda setters the true source of ent times. If this were true, then how could what we take to be the legislative expression scholars predict the outcome of a seemingly of majority rule? Are rules themselves subject arbitrary and unconstrained majority rule to the vagaries of shifting majority coalitions institution, even with perfect knowledge of (McKelvey and Ordeshook 1984)? preferences? Was literally anything possible? These and other questions were raised as a result of explorations in Arrovian social Limitations of Field Work choice theory, which visualized group deci- on Legislative Instability sions as being the product of individual preferences and group decision rules such as Political scientists tried to use legislative data majority rule. The biggest challenge to the to answer fundamental questions about the research agenda was majority rule instabil- scope and meaning of majority rule instability. Riker (1986) uses historical examples to illustrate the potential for majority Thanks to William Bottom, Jamie Druckman, Raymond rule instability. A favorite case was the 1956 Duch, Dima Galkin, Gyung-Ho Jeong, Yanna Krup- nikov, Arthur Lupia, Michael Lynch, Itai Sened, Jennifer House debate on federal aid to education. Nicoll Victor, Robert Victor, and Rick Wilson. Despite the fact that the majority party was

353 354 Gary Miller prepared to pass the original bill over lations” (Druckman et al. 2006, 629). Exper- the status quo, Republicans and northern imental research offered the prospect of nail- Democrats preferred the bill with the Pow- ing down the causation that was inevitably ell Amendment, which would send no aid to obscured by field data. segregated schools. Once amended, a third majority (southern Democrats and Republi- Early Spatial Experiments: cans) preferred no bill at all. This seemed Fiorina and Plott to be a graphic example of cyclic instability because some majority was ready to vote It did not take long after the emergence down any of the three alternatives (origi- of early rational choice models of legislative nal bill, Powell Amendment, status quo) in decision making for the advantages of experi- favor of a different outcome. The outcomes ments to become apparent. Fiorina and Plott “cycled,” and none of the three outcomes (1978) set out to assess McKelvey’s (1976) seemed to have a privileged position in terms demonstration that voting in two dimen- of legitimacy, manipulability, or likelihood. sions could cycle to virtually any point in the The determinant of the final outcome seemed space. “McKelvey’s result induces an inter- to have more to do with manipulation of the esting either–or hypothesis: ‘if equilibrium agenda than anything else. exists, then equilibrium occurs; if not, then Nevertheless, this case and others like it chaos’” (Fiorina and Plott 1978, 590). In set- were open to debate. Riker (1986) made ting out to examine this hypothesis, Fiorina assumptions about the preferences of the leg- and Plott were the precursors of an exper- islators that were open to different inter- imental research agenda assessing the pre- pretations. Wilkerson (1999) believed that dictability of majority rule. the possibility of instability as manifested by In addressing this question, Fiorina and “killer amendments” was minimal. In general, Plott (1978) created what became the canon- political scientists could only guess about the ical design for majority rule experiments. connection between voting behavior and the They used students as subjects, presenting underlying preferences of legislators. With- them with two-dimensional sets of possible out a way to measure the independent vari- decision “outcomes” – the crucial dependent ables (preferences and rules) or the dependent variable. The dimensions were presented in variables (legislative outcomes), rational actor an abstract way intended to render them neu- models seemed singularly handicapped. tral of policy or personal preferences. Of course, the effect of a shift in prefer- The two dimensions were salient only for ences or rules change might offer a “natural their financial compensation; the students experiment” on the effect of such a change saw payoff charts showing concentric circles on policy outcomes, but usually such historic around their highest-paying “ideal point.” changes were hopelessly confounded with One student might receive a higher payoff other historical trends that might affect the in the upper right-hand corner, whereas oth- outcome. Were the 1961 rule changes gov- ers would prefer other areas in the space. The erning the House Rules Committee respon- students were quite motivated by the payoffs, sible for the liberal legislation of the next a fact that gave the experimenters control decade or the manifestation of a change in over the key independent variable – prefer- preferences that would have brought that ences. legislation into being in any case? Was the The experimenter deliberately presented Republican ascendancy of 1994 the cause a passive, neutral face while reading instruc- of welfare reform or the vehicle for pub- tions that incorporated a carefully specified lic pressure that would have brought welfare regime of rules, determining exactly how a reform about in any case? Research on readily majority of voters could proceed to enact a observable features of legislatures – partisan- policy change or to adjourn. They were rec- ship, committee composition, constituency, ognized one at a time to make proposals, etc. – led to “ambiguous and debated corre- and each proposal was voted on against the Legislative Voting and Cycling 355 most recent winning proposal. They could discuss alternatives and seek supporters for particular proposals. Subjects had few con- Player 4 straints beyond a prohibition against side payments and (famously) “no physical threats.” Player 3 This procedure provided rigorous control over both preferences and rules. Player 1 However, did the students behave in a way y–coordinate Player 2 that could generalize to real legislature – and was it important if they did not? As Fior- 1978 Player 5 ina and Plott ( ) put it, “What makes us 20 40 60 80 100 120 believe [ . . . ] that we can use college students to simulate the behavior of Congress mem- 0 50 100 150 bers? Nothing” (576). They made no claims x–coordinate about the generalizability of the results, but Series 3 they did make claims about the implications of the outcomes for theory. “[I]f a given Figure 25.1. Outcomes of Majority Rule Experiments without a Core model does not predict well relative to others 1978 under a specified set of conditions [designed Source: Fiorina and Plott ( ) to satisfy the specifications of the theory], why should it receive preferential treatment as an explanation of non-laboratory behavior ...” move to the Northeast. The other treatment (576)? thus provided the crucial test of what would 1 Fiorina and Plott (1978) designed two happen when anything could happen. experimental settings: one with an equilib- However, Fiorina and Plott’s (1978) non- rium and one without. The equilibrium con- core experiments showed a great deal more cept of interest was the majority rule equi- “clustering” than could have been expected, librium or core – an alternative that could given McKelvey’s (1976) result (Figure 25.1). defeat, in a two-way vote, any other alter- The variance in the outcomes was greater native in the space. The core existed as a without a core than it was with a core – but result of specially balanced preferences by the differences were not as striking as they the voters – the core was the ideal point had expected. They concluded, “The pattern of one voter, and the other pairs of vot- of experimental findings does not explode, a ers had delicately opposing preferences on fact which makes us wonder whether some either side of the core. The Fiorina-Plott unidentified theory is waiting to be discov- results provided significant support for the ered and tested” (Fiorina and Plott 1978, predictive power of the core, when it existed. 590). Outcomes chosen by majority rule clus- Fiorina-Plott’s (1978) invitation to theo- tered close to the core. This conclusion was rize on the apparent constraint of noncore supported by subsequent experiments (Mc- majority rule was taken up promptly by at Kelvey and Ordeshook 1984). Wilson (2008a, least two schools of thought. One school held 875) analyzed experimental decision trajec- that subjects acting on their preferences in tories demonstrating the attractive power of reasonable ways produced constrained, cen- the core. Majorities consistently proposed trist results – corelike, even without a core. new alternatives that moved the group choice The other school emphasized that institu- toward the core. tional structure and modifications of major- The other treatment did not satisfy the ity rule generated the predictable constraint fragile requirements for a core (Figure 25.1). on majority rule. The first school examined For example, player 1’s ideal point, although a centrist outcome, could be defeated by a coalition of players 3, 4, and 5 preferring a 1 The gray shaded area is explained later in the chapter. 356 Gary Miller hypotheses about the effects of preferences Series 2 changes (holding rules constant), and the second, the effects of rule changes (holding pref- Player 3 erences constant). Experimenters could randomly assign subjects to legislative settings that varied by a tweak of the rules or a tweak of Player 2 the preferences, allowing conclusions about Player 4 causation that were impossible with natural legislative data. y–coordinate Player 1

1. Institutional Constraints on Player 5

Majority Rule Instability 0 20406080

The behavioral revolution of the 1950s and 0204060 80 100 1960 s consciously minimized the importance x–coordinate of formal rules in social interaction. In light Open of that, probably the most innovative and Figure 25.2. Majority Rule with Issue-by-Issue far-reaching idea that came out of Arrovian Voting social choice was neoinstitutionalism – the Source: McKelvey and Ordeshook (1984) claim that rules can have an independent and sometimes counterintuitive effect on legisla- ing does not seem to constrain outcomes tive outcomes. to the proposed structure-induced equilib- Once again, natural legislative settings did rium, as long as subjects can communicate not supply much definitive evidence one way openly. In Figure 25.2, player 5 is the median or the other. Even if scholars could point voter in the X dimension, as is player 4 to a significant rule change – for example, in the Y dimension. The results indicate a the change in the Senate cloture rule in good deal of logrolling, for instance, by the 1975 – and even if that change coincided 1, 2, and 5 coalition, that pulls outcomes with a change in the pattern of legislation, away from the structure-induced equilibrium. it was impossible to sort out whether the rule They conclude that theorists who “seek to change was causal, spurious, or incidental to uncover the effects of procedural rules and the policy change. One research agenda that institutional constraints must take cognizance followed from Arrovian social choice was to of incentives and opportunities for people to examine the effect of rules themselves, while disregard those rules and constraints” (201). holding preferences constant. The germaneness rule does not seem a sturdy source of majority rule stability. Procedural Rules: Structure-Induced Equilibrium Forward and Backward Agendas The institutional approach was kicked off Wilson (1986, 2008b) ran experiments on by Shepsle (1979), who initiated a flores- a different procedural variation – forward- cence of theory about institutions as con- moving agendas versus backward-moving straints on majority rule instability. For exam- agendas. A forward-moving agenda consid- ple, Shepsle argues that germaneness rules, ers the first proposal against the status quo, which limited voting to one dimension at the second alternative against the winner of a time, would induce a structure-induced the first vote, and so on. Each new proposal is equilibrium located at the issue-by-issue voted on against the most recent winner. Pre- median. sumably, the first successful proposal will be McKelvey and Ordeshook (1984)ran in the winset of the status quo, where the win- experiments showing that issue-by-issue vot- set of X is the set of alternatives that defeat X Legislative Voting and Cycling 357

350 2

300

1 250 Status Quo

200 3 A voting cycle under the forward moving agenda. 150 A backward moving agenda with myopic voting ending at the status quo. 100

50 4

0 050100 150 200 250 300 350

X Axis

Path under Backward Agenda manipulation Path under Forward Agenda manipulation

Ideal points of subjects Initial status quo

Figure 25.3. Effect of Backward and Forward Agendas Source: Wilson (2008b) by majority rule. A core has an empty winset, Figure 25.3 shows one typical voting tra- but when there is no core, every alternative jectory for each treatment. The soft gray line has a nonempty winset. The winset of the sta- shows a typical forward-moving agenda. The tus quo is the propeller-shaped figure shown first proposal was in the winset of the status in Figure 25.3. quo, backed by voters 2, 3, and 4. Subsequent An alternative is a backward-moving moves were supported by coalitions 3, 4, and agenda, in which alternatives are voted on 5;then1, 2, and 5; and then 2, 3, and 4 to in backward order from the order in which restore the first successful proposal and com- they were proposed. If alternatives 1, 2, and plete a cycle. A forward-moving agenda did 3 are proposed in that order, then the first nothing to constrain majority rule instability. vote is between 2 and 3, with the winner The dark line shows that the first alterna- against 1, and the winner of that against the tive introduced was not in the winset of the status quo. With this agenda, the final out- status quo, so the final vote resulted in the come should be either the status quo or an imposition of the status quo. This could have alternative in the winset of the status quo. been avoided with strategic voting by player 5 Theoretically, a backward-moving agenda is on the penultimate step, leaving the commit- more constrained – more predictable – than tee with an outcome closer to 5’s ideal point a forward-moving agenda. than the status quo. 358 Gary Miller

Overall, Wilson (2008b, 886) reports that in the other, a monopoly agenda setter. In this eight of twelve experiments run with the latter case, the agenda setter’s ideal point was backward-moving agenda treatment were at the unique core. Wilson showed that the out- the initial status quo, and the other four tri- comes in the open agenda had high variance; als were in the winset of the status quo. This the outcomes with an agenda setter had lower contrasted sharply with the forward-moving variance and were signiﬁcantly biased toward agenda, which never ended at the original sta- the agenda setter’s ideal point. tus quo and frequently cycled through the Figure 25.4 shows the trajectory for a typi- policy space. cal agenda setter experiment. The agenda set- The conclusion is that forward-moving ter, player 5, consistently plays off the coali- agendas do not constrain majority rule insta- tion with 1 and 2 against the coalition with 3 bility or provide the leverage necessary for and 4. The power to do so means, of course, accurate prediction. However, the backward- that majority rule instability can be replaced moving agenda is an institution that does by coherence – at the cost of making one effectively constrain majority rule. player a dictator.

Monopoly Agenda Control 2. Preference-Based Constraints on In simple majority rule, every majority coali- Majority Rule Instability tion has the power and motivation to move an outcome from outside its Pareto-preferred Shepsle’s (1979) original hypothesis – that set to some point inside. No point outside institutional variations of majority rule can the Pareto set of every majority coalition can sharply constrain majority rule instability and be in equilibrium. When the Plott symmetry allow prediction of experimental outcomes – conditions hold, a single internal voter’s ideal has proven both true and of the utmost point is included in every majority coalition’s significance for studying democracy. Rules Pareto set. Because there is no point that is defining control over the agenda, the size of internal to the Pareto sets of all decisive coali- the majority, or bicameralism have all been tions, there is no core. Instability is the result shown to lead to an improvement in predic- of too many decisive majority coalitions. tion accuracy. The rules can create stability by mandating However, the patterning of outcomes in that some majority coalitions are not decisive. simple majority rule experiments, as illus- For example, the rules may specify that every trated in Figure 25.1, reveals that institutional proposal to be considered must be approved rules are a sufficient, but not necessary, con- by a single actor – the agenda monopolist. dition for constraint. Experimental outcomes In other words, every majority coalition that cluster with simple majority rule – even with- does not include the agenda monopolist is not out monopoly agenda control, germaneness decisive. rules, or a backward-moving agenda. This greatly reduces the number of deci- Despite the fact that McKelvey (1976)was sive majority coalitions. In particular, the the author of what came to be known as intersection of the Pareto sets for all deci- the “chaos” theorem, he himself was an early sive coalitions is guaranteed to include only advocate of finding a preference-based solu- one point – the agenda setter’s ideal point. As tion concept. That is, he believed that the a result, the core of a game with an agenda actions of rational voters, negotiating alter- monopolist necessarily includes the agenda native majority coalitions to advance their setter’s ideal point. own preferences, would somehow constrain To test the effect of this institutional majority rule to a reasonable subset of the feature on majority rule instability, Wilson entire policy space – without requiring the (2008b) ran experiments with constant pref- constraint of rules other than simple major- erences and no simple majority rule core. In ity rule. McKelvey, Ordeshook, and Winer one treatment, there was an open agenda, and (1978) advanced the solution concept known Legislative Voting and Cycling 359

300 Status Quo 2

250

200

3 150

100

5 50 4 Agenda Setter

0 0 50 100 150 200 250 300 X Axis

Agenda path for one trial Ideal points for subjects Initial status quo

Figure 25.4. Effect of Monopoly Agenda Setting Source: Wilson (2008b) as the “competitive solution” for simple the context of discrete alternatives by Miller majority rule games. By understanding coali- (1980). It is a solution concept that identiﬁes tion formation as a kind of market that estab- a set of moderate outcomes in the “center” of lished the appropriate “price” for coalitional the space of ideal points as the likely outcome pivots, McKelvey et al. generated predictions of strategic voting and the coalition formation that worked rather well for ﬁve-person spatial process. games. However, the authors gave up on the Outcomes that are far from the “center” competitive solution when other experimen- of the ideal points are certain to be covered, tal results, using discrete alternatives, proved where a covered alternative B is one such that to be sensitive to cardinal payoffs (McKelvey there is some alternative A that beats B, and and Ordeshook 1983). every alternative X that beats A also beats B. If A covers B, then it implies that B is a relatively unattractive alternative with a large enough The Uncovered Set 2 winset to encompass the winset of A. An alternative preference-based solution concept was the uncovered set, developed in 2 More centrist outcomes have smaller winsets. 360 Gary Miller

An alternative is in the uncovered set if it free” – the ideal points of voters provide is not covered by any other alternative. If D enough information to predict where out- is uncovered, then, for every C that beats D, comes should end up, even without knowing then there is some alternative X such that D exactly which of the three institutions would beats X and X beats C. This means that an be used to select the outcome. uncovered alternative can either defeat every The problem was that neither McKelvey other alternative directly or via an intermedi- nor anyone else knew exactly how much the ate alternative. The uncovered set is the set of uncovered set constrained majority rule deci- centrist outcomes that constitute the (unsta- sion making because no one had a way to ble) center of the policy space. characterize the uncovered set for a given set Early theoretical results showed that the of preferences. uncovered set had several striking characteristics. For one thing, the uncovered set was Looking Backward with the Uncovered Set shown to be a subset of the Pareto set. For another, it shrank in size as preference pro- The recent invention of an algorithm for pre- files approximated those producing a core, cise estimation of the uncovered set (Bianco, and it collapsed to the core when the core Jeliazkov, and Sened 2004) has allowed the existed (Cox 1987). testing of that solution concept against pre- The uncovered set has proven to be of viously reported experimental results (Bianco interest to both noncooperative game theory et al. 2006) and with new data (Bianco et al. and cooperative game theory. The reason is 2008). Figure 25.1 is a case in point because that, as McKelvey (1986) argues, the uncov- it shows the Fiorina-Plott (1978) noncore ered set contains the noncooperative equi- experiments. The uncovered set for their libria arising under a variety of institutional experimental configuration of preferences is rules. shown as the small shaded region. In Fig- Shepsle and Weingast (1984) propose ure 25.1, the uncovered set is a relatively that “the main conclusion is that institu- precise and promising predictor of the non- tional arrangements, specifically mechanisms core experiments. The same is true for the of agenda construction, impose constraints uncovered set shown (as a gray shaded region) on majority outcomes” (49). McKelvey (1986) for the McKelvey-Ordeshook experiments on took away a quite different interpretation. germaneness and communication – nearly all In an article provocatively titled “Covering, outcomes were in the uncovered set (Figure Dominance and Institution-Free Properties 25.2). For the McKelvey-Ordeshook experi- of Social Choice,” McKelvey (1986) argues ments, with different proposal rules and dif- that if a single solution concept encompasses ferent degrees of constraint on communi- the equilibrium results of a variety of institu- cation, the uncovered set performs equally tions, then the choice process is “institution well. free.” That is, “the actual social choice may We can do the same with other major- be rather insensitive to the choice of institu- ity rule experiments run in two-dimensional tional rules” (McKelvey 1986, 283). policy space with simple majority rule. The In the article, McKelvey (1986) demon- results for a series of simple majority rule strates that various distinct institutions theo- experiments are shown in Table 25.1. Out of retically lead to equilibrium outcomes inside 272 total majority rule experiments admin- the uncovered set. He confirmed the result istered by eight different teams of experi- that legislative voting under a known, fixed mentalists, ninety-three percent were in the agenda should lead inside the uncovered set. uncovered set. Cooperative coalition formation should lead to outcomes in the uncovered set, as should Testing the Uncovered Set two-candidate elections. Hence, McKelvey could argue, constraint on simple majority Although the results in Table 25.1 are note- rule instability seemed to be “institution- worthy, the experiments reported there were Legislative Voting and Cycling 361

Table 25.1. Testing the Uncovered Set with Previous Majority Rule Experiments

Total % in Article Experiment Outcomes Uncovered Set

Fiorina and Plott (1978) Series 31580.00 McKelvey, Ordeshook, Competitive Solution 8 100.00 and Winer (1978) Laing and Olmsted (1978) A2–The Bear 19 89.47 B–Two Insiders 19 94.74 C1–House 19 73.68 C2–Skewed Star 18 83.33 McKelvey and Ordeshook PH–Closed Communication 17 100.00 (1984) PH–Open Communication 16 93.75 PHR–Closed Communication 15 100.00 PHR–Open Communication 18 100.00 Endersby (1993) PH–Closed Rule Closed Communication 10 100.00 PH–Open Rule Closed Communication 10 100.00 PH–Open Rule Open Communication 10 100.00 PHR–Closed Rule Closed Communication 10 100.00 PHR–Open Rule Closed Communication 10 100.00 PHR–Open Rule Open Communication 10 100.00 Wilson (2008b) Forward Agenda 12 91.67 Backward Agenda 12 100.00 Wilson and Herzberg Simple Majority Rule 18 94.44 (1987) King (1994) Nonvoting Chair 6100.00 Total 272 93.75

Source: Bianco et al. (2006). not designed to test the uncovered set. change in player 1’s ideal point shifted the In particular, several of these experiments uncovered set dramatically. typically imposed maximal dispersion of ideal The alternative hypothesis is what may be points, resulting in quite large uncovered called the partisan hypothesis, based on the sets – perhaps an “easy test” of the uncov- obvious clustering of ideal points. Poole and ered set. Consequently, Bianco et al. (2008) Rosenthal (1997), Bianco and Sened (2005), designed computer-mediated, five-person, and others have estimated the preferences of majority rule experiments with two treat- real-world legislatures – finding that they are ments creating relatively small and nonover- organized in two partisan clusters. So, the lapping uncovered sets – designed to be a dif- differences between the two configurations ficult case for the uncovered case. could be thought of as a shift of majority party The two treatments were based on two control with a change in representation of configurations of preferences shown in Fig- district 1. The members of the majority clus- ures 25.5a and 25.5b. In each case, the prefer- ter in either configuration could easily and ences were “clustered” rather than maximally quickly pick an alternative within the convex dispersed; this had the effect of producing hull of their three ideal points and, resisting smaller uncovered sets. Configurations 1 and the attempts by the members of the minority 2 are identical except for the location of player cluster, vote to adjourn. 1’s ideal point. In configuration 1, player 1 It is worth noting that the uncovered set was clustered with players 4 and 5; in con- in this setting is primarily located between figuration 2, player 1 was in an even tighter the Pareto sets for the majority and minority majority cluster with players 2 and 3.The parties, and thus will only occur if there is a 362 Gary Miller

Player 2 Player 2 100806040 100806040

Player 3 Player 1 Player 3 y–coordinate y–coordinate

Player 5 Player 1 200 200 Player 5 Player 4 Player 4

0 20406080100 0 20406080100 x–coordinate x–coordinate (a) (b) Figure 25.5. Sample Majority Rule Trajectory for Configuration 1 Source: Bianco et al. (2008), used with permission of The Society for Political Methodology. significant amount of cross-partisan coalition to pull outcomes out of the 1-2-3 Pareto tri- formation and no party solidarity. In other angle toward the minority cluster. The result words, in configuration 1, if players 4 and is cycling within the smaller uncovered set. 5 can offer player 3 an outcome that is more Twenty-eight experiments were done with attractive than that offered by players 1 and 2, each treatment. Figure 25.6a shows the final then the uncovered set has a chance of being outcome in the twenty-eight configuration realized. But if player 3, for example, refuses 1 experiments. The percentage of final out- offers especially made to move him or her comes in the uncovered set was 100 percent. away from his or her “natural” allies, then the Figure 25.6b shows the final outcomes in the outcome should be well within the Pareto set twenty-eight configuration 2 experiments. In of the partisan coalition, rather than in the four committees, the outcome seemed to be uncovered set. influenced by fairness considerations. Figure 25.5a shows a sample committee In seven of the committees, the oppo- trajectory for configuration 1. As can be seen, site occurred – the partisan 1-4-5 coalition there was a great deal of majority rule insta- formed and imposed an outcome in their bility. A variety of coalitions formed, includ- Pareto triangle but outside the uncovered set. ing coalitions across clusters. However, the In either case, the presence of an extremely instability was constrained by the borders of tight cluster of three ideal points seemed to the uncovered set. Despite frequent success- decrease the likelihood of the kind of mul- ful moves to outcomes close to the contract tilateral coalition formation that could pull curve between players 1 and 3, players 4 and outcomes into the uncovered set. Overall, the 5 were repeatedly able to pull the outcome proportion of configuration 2 outcomes in the modestly in their direction by offering player uncovered set was still 60.7 percent. 3 more than player 1 had offered. Although fairness considerations or parti- Configuration 2 is more difficult; any out- san solidarity can result in outcomes outside come in the Pareto set of the tight cluster of the uncovered set, it seems fair to say that, 1, 4, and 5 is very attractive to these three as long as the coalition formation process is voters – making it hard for 2 and 3 to offer cross-partisan and vigorous, the outcome will proposals that will break up the 1-4-5 coali- likely be within the uncovered set. Overall, tion. Yet, even here, players 2 and 3 occasion- the uncovered set experiments suggest that ally make proposals that attract support from the majority rule coalition formation pro- members of the majority cluster. This tends cess does constrain outcomes, as argued by Legislative Voting and Cycling 363

Configuration 1 Configuration 1

100806040 Player 2 100806040 Player 2 Player 1 Player 3 Player 3 y–coordinate y–coordinate

Player 1 200 200 Player 5 Player 5 Player 4 Player 4

0 20406080100 0 20406080100 x–coordinate x–coordinate

(a) (b) Figure 25.6. (a) Uncovered Set and Outcomes for Conﬁguration 1 (b) Uncovered Set and Outcomes for Conﬁguration 2 Source: Bianco et al. (2008), used with permission of The Society for Political Methodology.

McKelvey (1986). Even more important, out- erative games associated with particular insti- comes tend to converge to centrist, compro- tutional rules are in fact located in the uncov- mise outcomes. ered set. One institution McKelvey (1986)was interested in was that in which amendments 3. Challenges and Opportunities are generated by an open amendment pro- for Further Research cess from the ﬂoor, in the absence of complete information about how the amendments Even though the past generation of majority might be ordered or what additional motions rule experiments has largely tested either an might arise. The proposal stage would be fol- institutional effect or preference-based effect lowed by a voting stage in which all voters on majority rule outcomes, the McKelvey would know the agenda. Viewing this insti- (1986) hypothesis offers a research agenda tution as an n-person, noncooperative game, that involves both institutions and prefer- the equilibrium should be contained in the ences, both noncooperative and cooperative uncovered set as long as voters vote sophisti- game theory. catedly. We know that voters sometimes make mistakes (i.e., fail to vote in a sophisticated man- The McKelvey Challenge: Endogenous ner) (Wilson 2008b). So, the outcome of such Agendas in Legislatures endogenous agenda institutions is an open Since McKelvey (1986) wrote his arti- question for experimental research. Given cle “Covering, Dominance and Institution- McKelvey’s (1986) result, there are three log- Free Properties of Social Choice,” scholarly ical possibilities: 1) outcomes will be at the research on legislative institutions has ﬂow- noncooperative equilibrium (and therefore in ered, especially with the aid of noncoopera- the uncovered set); 2) outcomes will be in the tive game theory (e.g., Baron and Ferejohn uncovered set, but not at the noncooperative 1989). However, little of that research has equilibrium; or 3) outcomes will be outside served to respond to McKelvey’s challenge to the uncovered set (and therefore not at the examine whether the equilibria of noncoop- noncooperative equilibrium). 364 Gary Miller

There is a large and established psycho- left the ordinal payoffs unchanged. Eavey logical literature on negotiation, touching on (1991) ran simple majority rule experiments the effect of such factors as risk preferences, with the same ordinal preferences as in Fio- cognitive biases, trust, egalitarian norms, cul- rina and Plott (1978), but Eavey constructed tural considerations, and ethical considera- less steep payoff gradients for the voters to tions. Because implementation of majority the west, creating a benign Rawlsian alterna- rule ultimately boils down to negotiating tive to the east; that is, the point that maxi- majority coalitions, it is important to begin mized the payoff of the worst-paid voter lay to incorporate insights from that literature east of the core and gave all five members of into the design of majority rule experiments. the committee a moderate payoff. Although For example, the core is a cooperative solu- the attraction of the core was still apparent tion concept that assumes a contract enforce- (Grelak and Koford 1997), the new cardi- ment mechanism, which is uniformly lacking nal payoffs tended to pull outcomes in the in majority rule experiments. Why does the direction of the fair point because participants core work so well in experiments that uni- in these face-to-face committees seemed to formly lack any contract enforcement mech- value outcomes supported by supermajori- anism? One answer is suggested by Bottom, ties, rather than a minimal winning coali- Eavey, and Miller (1996). In this experiment tion. Further research is needed to explore related to examining an institution of decen- the sensitivity of computer-mediated exper- tralized agenda control, getting to the core iments to cardinal payoffs. Understanding from some status quos required forming and the degree of sensitivity to cardinal values is then reneging on a coalition – actions that potentially important for evaluating our abil- many subjects were unwilling to undertake. ity to control subjects’ induced valuation of Groups were “constrained by a complicated alternatives. set of social norms that prevents the fric- One challenge facing students of major- tionless coalition formation and dissolution ity rule has been persistently voiced since assumed by cooperative game theory” (318). Fiorina and Plott (1978). Their defense of The net result is that informal social processes experiments was grounded in an acknowl- may substitute for formal contract enforce- edged need for parallel experimental and ment, resulting in experimental support for field research: “we reject the suggestion cooperative solution concepts such as the core that the laboratory can replace creative field and the uncovered set. researchers” (576). Since that time, parallelism in research has been advocated a good many more times than it has been attempted. Fairness and Other Nonordinal The recent development of techniques Considerations for estimating the spatial preferences of The Fiorina-Plott (1978) experiments were real-world legislators offers the prospect designed in such a way that subject payoffs fell of parallel research using laboratory and off very quickly from ideal points. As a result, real-world data. An ideal point estimation there was no single outcome that would give method called “agenda-constrained” estima- three voters a significant payoff; at least one tion (Clinton and Meirowitz 2001; Jeong majority coalition voter had to vote for an 2008) relies on the knowledge of the agenda outcome that yielded only pennies. And there and legislative records on roll call votes was certainly no outcome that could provide on amendments; with this information, they a lucrative payoff for all five voters. obtain estimates of both legislative prefer- In one sense, this was a difficult test for ences and the alternative outcomes legislators the core. It proved a good predictor, even are voting on. This information is just what though it did not create a gleeful majority is needed to test the uncovered set with real coalition. However, it also raised the ques- legislative data. tion of whether the choice of the core was For example, Figure 25.7 shows estimates sensitive to changes in cardinal payoffs that of senators’ ideal points and a trajectory of Legislative Voting and Cycling 365

The Civil Rights Act of 1964

Democrats Republicans

Dirksen Kennedy Saltonstall

Fong B D A C E > Expand – Scope

Limit < SQ

Russell –8 –6 –4 –2 0 2 4 6 –8 –6 –4 –2 0 2 4 6 Enforcement Less Strict <–> More Strict Figure 25.7. Senatorial Ideal Points and Proposed Amendments for Civil Rights Act of 1964 Source: Jeong et al. (2009)

winning outcomes using Senate voting on weakest supporters in one dimension or the 109 roll call amendments for the Civil Rights other (Jeong, Miller, and Sened 2009). Act of 1964. It also depicts the estimated The coalitional negotiations in the Sen- uncovered set given the locations of the senate were much like that in experiments: ators over the two key dimensions of the new coalitions were formed to propose and bill – scope and enforcement. The uncovered vote on new amendments, and as these set lies between the cluster of ideal points succeeded or failed, coalitional negotiations of the strongest civil rights supporters and continued to generate yet more amendments. the strongest opponents. As in the laboratory The administration bill, as modified by the experiments, the senators created a variety of House, was located in the uncovered set. An cross-cluster coalitions; the civil rights oppo- amendment to guarantee a trial by jury for nents repeatedly tried to weaken the enforce- those state and local officials found in con- ment or limit the scope by picking off the tempt for their opposition to civil rights was 366 Gary Miller popular enough to generate a majority coali- rule instability. As Wilson (2008b) noted, tion that moved the bill to location B to given appropriate institutions, “voting cycles, the left of the civil rights bill. A leadership rather than being rare events, are common” substitute form of the bill was much stricter (887). Given an open, forward agenda and in enforcement at point C, but a weaken- minimally diverse preferences, cycles can be ing amendment protecting southern officials readily observed. from double jeopardy brought the location of Nevertheless, experiments have also the bill back inside the uncovered set, where it shown that cycles are contained within the remained despite a slight weakening of scope. uncovered set and can be tamed by institu- The second to last vote pitted the adminis- tional rules and procedures. There is a place tration bill as amended against the leader- for more theoretical endeavors and further ship substitute as amended; the final vote ran experimental research on ideological (spatial) the leadership substitute against the perceived decision making – both in the lab and in paral- status quo. The final bill, located at E, was lel fieldwork on questions generated by exper- well within the uncovered set. imental research. Indeed, the results of major- What does the date in Figure 25.7 sug- ity rule experiments have both informed the gest for an integrated research agenda involv- political science debate about the meaning ing both Senate data and experiments? One and limits of majority rule (McKelvey and possibility is that preferences estimated from Ordeshook 1990, 99–144) and guided the- real-world legislators on actual legislation orists as they seek explanations for both the may be replicated in the laboratory; thus, a observed instabilities of majority rule and the unique legislative history can potentially be observed constraints on that instability. If, as repeated many times over. The debate on the McKelvey (1986) hypothesized, a variety of civil rights bill can be replayed by inducing institutional rules can only manipulate out- preferences similar to those of the senators to comes within the uncovered set, then the determine whether a similar outcome occurs. degree to which behind-the-scenes agenda We can find out whether, given the prefer- setters can manipulate the outcome of major- ences of legislators, the outcome was in some ity rule processes is itself limited. sense inevitable or whether a dispersion of final outcomes could have been the basis for alternative histories. References Modifications in real-world preferences can be examined to examine counterfactuals Baron, David, and John Ferejohn. 1989.“Bargain- such as 1) what would have happened to this ing in Legislatures.” American Political Science bill if Midwestern Republicans had been less Review 83: 1181–206. 2 supportive of the civil rights act? or ) could Bianco, William T., Ivan Jeliazkov, and Itai Sened. the bill have been passed if Tennessee’s sen- 2004. “The Uncovered Set and the Limits of ators had been more opposed? Legislative Action.” Political Analysis 12: 256– The same preferences can be examined 76. under different institutional rules to exam- Bianco, William T., Michael S. Lynch, Gary J. ine what might have occurred if the legis- Miller, and Itai Sened. 2006. “‘Theory Wait- lature had operated under a different set of ing To Be Discovered and Used’: A Reanaly- sis of Canonical Experiments on Majority Rule rules. What if the Senate had used a different 68 837 agenda procedure or had enacted the 1975 Decision-Making.” Journal of Politics : – 50. cloture reform before 1964? Bianco, William T., Michael S. Lynch, Gary Miller, and Itai Sened. 2008. “The Constrained Instability of Majority Rule: Experiments on 4. Conclusion the Robustness of the Uncovered Set.” Political Analysis 16: 115–37. Experimental research has to some extent Bianco, William T., and Itai Sened. 2005. substantiated the concern with majority “Uncovering Conditional Party Government: Legislative Voting and Cycling 367

Reassessing the Evidence for Party Inﬂuence Implications for Agenda Control.” Journal of in Congress and State Legislatures.” American Economic Theory 12: 472–82. Political Science Review 99: 361–72. McKelvey, Richard D. 1986. “Covering, Domi- Bottom, William, Cheryl Eavey, and Gary Miller. nance and Institution Free Properties of Social 1996. “Getting to the Core: Coalitional Choice.” American Journal of Political Science 30: Integrity as a Constraint on the Power of 283–314. Agenda Setters.” Journal of Conﬂict Resolution McKelvey, Richard D., and Peter C. Ordeshook. 40: 298–319. 1983. “Some Experimental Results That Fail Clinton, Joshua D., and Adam Meirowitz. 2001. to Support the Competitive Solution.” Public “Agenda Constrained Legislator Ideal Points Choice 40: 281–91. and the Spatial Voting Model.” Political Analysis McKelvey, Richard D., and Peter C. Ordeshook. 9: 242–59. 1984. “An Experimental Study of the Effects Cox, Gary W. 1987. “The Uncovered Set and the of Procedural Rules on Committee Behavior.” Core.” American Journal of Political Science 31: Journal of Politics 46: 182–205. 408–22. McKelvey, Richard D., and Peter C. Ordeshook. Druckman, James N., Donald Green, James Kuk- 1990. “A Decade of Experimental Research linski, and Arthur Lupia. 2006.“TheGrowth in Spatial Models of Elections and Commit- and Development of Experimental Research tees.” In Advances in the Spatial Theory of Voting, in Political Science.” American Political Science eds. James M. Enelow and Melvin J. Hinich. Review 100: 627–36. Cambridge: Cambridge University Press, 99– Eavey, Cheryl L. 1991. “Patterns of Distribution 149. in Spatial Games.” Rationality and Society 3: McKelvey, Richard D., Peter C. Ordeshook, and 450–74. Mark D. Winer. 1978. “The Competitive Solu- Endersby, James W. 1993. “Rules of Method tion for N-Person Games without Transfer- and Rules of Conduct: An Experimental Study able Utility, with an Application to Committee on Two Types of Procedure and Committee Games.” American Political Science Review 72: Behavior.” Journal of Politics 55: 218–36. 599–615. Fiorina, Morris P., and Charles R. Plott. 1978. Miller, Nicholas. 1980. “A New Solution Set for “Committee Decisions under Majority Rule: Tournament and Majority Voting.” American An Experimental Study.” American Political Sci- Journal of Political Science 24: 68–96. ence Review 72: 575–98. Poole, Keith T., and Howard Rosenthal. 1997. Grelak, Eric, and Kenneth Koford. 1997. “A Re- Congress: A Political-Economic History of Roll- Examination of the Fiorina-Plott and Eavey Call Voting. New York: Oxford University Voting Experiments.” Journal of Economic Press. Behavior and Organization 32: 571–89. Riker, William H. 1982. Liberalism against Pop- Jeong, Gyung-Ho. 2008. “Testing the Predictions ulism. San Francisco: Freeman. of the Multidimensional Spatial Voting Model Riker, William H. 1986. The Art of Political with Roll Call Data.” Political Analysis 16: 179– Manipulation. New Haven, CT: Yale University 96. Press. Jeong, Gyung-Ho, Gary Miller, and Itai Sened. Shepsle, Kenneth A. 1979. “Institutional Arrange- 2009. “Closing the Deal: Negotiating Civil ments and Equilibria in Multidimensional Vot- Rights Legislation.” American Political Science ing Models.” American Journal of Political Science Review 103: 588–606. 23: 27–59. King, Ronald R. 1994. “An Experimental Investi- Shepsle, Kenneth A., and Barry R. Weingast. gation of Super Majority Voting Rules: Impli- 1984. “Uncovered Sets and Sophisticated Vot- cations for the Financial Accounting Standards ing Outcomes with Implications for Agenda Board.” Journal of Economic Behavior and Orga- Institutions.” American Journal of Political Sci- nization 25: 197–217. ence 28: 49–74. Laing, James D., and Scott Olmstead. 1978.“An Wilkerson, John. 1999. “‘Killer’ Amendments in Experimental and Game-Theoretic Study of Congress.” American Political Science Review 93: Committees.” In Game Theory and Political Sci- 535–52. ence, ed. Peter C. Ordeshook. New York: New Wilson, Rick K. 1986. “Forward and Backward York University Press, 215–81. Agenda Procedures: Committee Experiments McKelvey, Richard D. 1976. “Intransitivities in on Structurally Induced Equilibrium.” Journal Multidimensional Voting Models and Some of Politics 48: 390–409. 368 Gary Miller

Wilson, Rick K. 2008a. “Endogenous Properties Handbook of Experimental Economics Results, eds. of Equilibrium and Disequilibrium in Spatial Charles R. Plott and Vernon Smith. Amster- Committee Games.” In Handbook of Experimen- dam: North Holland, 880–8. tal Economics Results, eds. Charles R. Plott and Wilson, Rick K., and Roberta Herzberg. 1987. Vernon Smith. Amsterdam: North Holland, “Negative Decision Powers and Institutional 872–9. Equilibrium: Experiments on Blocking Coali- Wilson, Rick K. 2008b. “Structure-Induced Equi- tions.” Western Political Quarterly 40: 593– librium in Spatial Committee Games.” In 609. CHAPTER 26

Electoral Systems and Strategic Voting (Laboratory Election Experiments)

Rebecca B. Morton and Kenneth C. Williams

It can be complicated to attempt to under- tion that assigns a preference for a particular stand how election mechanisms and other candidate or party. Candidates are typically variables surrounding an election determine rewarded based on whether they win the elec- outcomes. This is because the variables of tion, but sometimes their rewards depend on interest are often intertwined; thus, it is dif- the actions they take after the election. ﬁcult to disentangle them to determine the By randomizing subjects to different treat- cause and effect that variables have on each ments and controlling many exogenous vari- other. Formal models of elections are used ables, a laboratory election experiment is able to disentangle variables so that cause and to establish causality between the variables effect can be isolated. Laboratory election of interest. For example, if a researcher is experiments are conducted so that the causes interested in how different types of infor- and effects of these isolated variables from mation (e.g., reading a newspaper editorial these formal models can be empirically mea- vs. reading a blog online) affect voting deci- sured. These types of experiments are con- sions, then in the laboratory it is possible ducted within a single location, where it is to control all parameters in the experiment, possible for a researcher to control many of such as voter preferences for different candi- the variables of the election environment and dates, parties, or issues, while only varying the thus observe the behavior of subjects under types of information voters receive. Random- different electoral situations. The elections izing subjects to two different types of infor- are often carried out in computer laborato- mation treatments (e.g., editorial vs. blog) ries via computer terminals, and the com- allows the researcher to determine whether munication between the researcher and sub- one type of information (editorial) causes vot- jects occurs primarily through a computer ers to behave differently than other voters interface. In these experiments, subjects are who were given a different type of informa- assigned as either voters or candidates, and, tion (blog) under the same electoral condi- in some cases, they are given both roles. tions and choices. Hence, laboratory election Voters are rewarded based on a utility func- experiments are concerned with controlling

369 370 Rebecca B. Morton and Kenneth C. Williams

they receive the highest payoff on the x axis); $100 each voter prefers policies closer to his or her ideal point than policies further away. Candidate 1 Candidate 2 In this simple theory, it is assumed that two candidates compete in an election by adopting policies on the single dimension and voters vote for candidates who adopt policies

Payoffs to voters Payoffs Voter 1 Voter 2 Voter 3 closer to their own ideal points. To see this, consider that if candidate 1 adopts position 10 and candidate 2 adopts position 60, then voter 1 will vote for candidate 1, but voters 2 and 3 will vote for candidate 2 because his or her 0 20 50 80 100 position is closer to their ideal points than Policy Position is candidate 1’s position. Because both can- Figure 26.1. Median Voter Theorem didates know the distribution of voter ideal points, both candidates realize that the opti- aspects of an election environment and ran- mal location for placement of positions is domizing subjects to treatments in order to at voter 2’s ideal point. Choosing this point determine causality of the election variable(s) guarantees each candidate of receiving at least a researcher has selected. Field experiments fifty percent of the vote. also allow researchers to vary variables, such The theorem shows the importance of a as types of information, but they do not allow central tendency in a single-dimension elec- for the control of other factors, such as voter tion. However, the theorem relies on restric- preferences over candidates, parties, or issues tive assumptions such that voters have full (see Gerber’s chapter in this volume). As information about candidate positions, can- opposed to field experiments, the laboratory didates know the distribution of voter prefer- allows a researcher to control all aspects of ences, and there is no abstention. When the the election environment. theorem is extended to two dimensions, no Laboratory election experiments started equilibrium exists unless other rigid restric- in the early 1980s and were directly de- tions are made. rived from committee experiments because Experiments first considered the assump- the researchers working on some of these tion that voters possess full information about committee experiments, McKelvey and candidate positions, which was challenged by Ordeshook (1982), began conducting the first empirical findings that voters were relatively laboratory election experiments (for a review uninformed about the policies or even the of committee experiments, see Miller’s chap- names of the candidates (Berelson, Lazars- ter in this volume). Similar procedures used in feld, and McPhee 1954; Almond and Verba the committee experiments were carried over 1963; Converse 1975). The purpose of these to test competitive elections where the sub- early experiments was to determine whether stantive questions that led to laboratory elec- the general results would still hold if the tion experiments were questions relating to information assumption were relaxed. Labo- the median voter theorem (Downs 1957). To ratory experiments were an ideal way to study illustrate this theory, consider Figure 26.1.In this question because it was possible to repli- this case, there is a single policy dimension cate the assumptions of the model in an exper- where three voters have single-peaked utility imental setting, allowing for variables, such as functions or preferences over the dimension the information levels that voters have about (the y axis). The property of single peakedness candidates, to be relaxed. Hence, causality of utility functions ensures that each voter has could be directly measured by determining a unique ideal point over the policy dimen- whether candidate positions converged when sion. For each voter, a dashed line represents voters possessed only incomplete informa- their ideal point or their best policy (where tion about candidate positions. We discuss Electoral Systems and Strategic Voting (Laboratory Election Experiments) 371 the results of these types of experiments later to observe learning effects. In some experi- in this chapter. ments, subjects need experience with the election environment to figure out equilibrium behavior, or what is the optimal decision to 1. Methodology make given the environment, and this can often take a number of rounds. Hence, in In terms of incentives, laboratory election most analyses, researchers look at the behav- experiments pay subjects based on their per- ior of subjects in the beginning, middle, and formance during the experiment. For a fuller end of the experiment to determine learning discussion of the use of financial incentives, effects. Finally, conducting multiple elections see Dickson’s chapter in this volume. Finan- during a single experimental session simply cial incentives allow experimenters to oper- gives the researcher more data to analyze. ationalize a monetary utility function that Again, one of the advantages of using labo- establishes performance-based incentives for ratory experiments is that it is easy to measure subjects within the experimental environ- causality about subject behavior within the ment. This procedure follows the principles election environment when different treat- of induced value theory (Smith 1976), which ments are compared and subjects are ran- essentially means that payments awarded dur- domly assigned to treatments. Consequently, ing the experiment must be salient in order experimental controls and randomization of to motivate subjects to choose as if the situa- treatments allow for easy establishment of tion or election were natural. That is, within causality of the variables of interest with- the election environment, voter and candi- out the use of sophisticated statistical meth- date subjects must believe that there are real ods. Laboratory election experiments allow consequences to their actions. the researcher to examine electoral phenom- Another aspect of laboratory elections is ena that observational studies cannot because that a number of repeated trials or elec- there are simply no data or very little data, tions are conducted within a single exper- such as the electoral properties of elec- imental session. In a typical election, vot- tion rules that have not been instituted or ers are usually assigned a type (or a prefer- used very little in real-world elections. Also, ence for a particular candidate); but in the researchers who rely on observational data are next election period, the voter is randomly constrained by the number of election occur- reassigned another type, exposed to a dif- rences, whereas in the laboratory, election ferent treatment, and so on until the com- researchers can conduct hundreds of elections pletion of the experimental session. This is under various manipulations. Finally, and referred to as a “within-subject treatment” more important, laboratory elections allow design that allows researchers to vary treat- the researcher to play the role of God over the ments (election parameters) while holding election environment because manipulations subject identities constant. One reason that are generally easy to induce in a laboratory. voters are randomly assigned different types In this chapter, we present laboratory for each election period is to avoid repeated election experiments from a wide range of game effects. This means that the researcher approaches in which we hope to show that does not want subjects to view each election results from experimental elections have, as a function of the last election, but rather he like results from observational data, provided or she want subjects to believe that they are findings deemed to be fruitful for the body of participating in a single or new election each literature on election behavior and electoral period. Randomly assigning subjects different mechanisms. First, we discuss the early elec- types each period and varying treatments does tion experiments that were concerned with in fact create a new electoral environment for testing the robustness of the median voter subjects at each period because they have dif- theorem. We continue with an examination ferent decisions to make. Also, by having a of experiments on theories that explain can- number of elections, it allows the researcher didate policy divergence in elections. We 372 Rebecca B. Morton and Kenneth C. Williams then examine experiments on multicandidate Prior to the election, a poll was con- elections and the coordination problem that ducted that revealed the percentage of sub- is involved with strategic voting, and dis- jects within a group who reported favor- cuss experiments on sequential elections in ing either candidate. With these two pieces which different sources of information are of information (the interest group endorse- relayed to early and late voters in the vot- ments and the poll results), uninformed voting queue. Finally, we present experiments ers in the theoretical equilibrium behaved on voter turnout. Although we attempt to or voted like informed voters. In the experi- provide a comprehensive review of labora- ment, there was significant evidence in sup- tory experimental work on elections, due to port of the rational expectations equilibrium, space constraints we are not able to discuss in which uninformed voters behaved as if they all experimental work in detail. We encour- were informed eighty percent of the time. age readers to explore this literature further. The experimental results also found that, although candidate subjects did not know the median voter’s ideal point, their posi- 2. The Literature on Laboratory tions converged near the median after a few Election Experiments rounds of the experiment. As discussed previously, these experimental results were pos- Tests of the Median Voter Theorem sible to observe primarily because of the con- The first significant laboratory election trol that the researchers were able to exert experiments were conducted by McKelvey over voter information. Such control would and Ordeshook (1985a, 1985b), in which they not have been possible in observable elections relaxed the full information condition of the or in a field experiment on an election. median voter theorem.1 These experiments A retrospective voting experiment con- are what is referred to as a “stress test,” which ducted by Collier et al. (1987) severely limited examines how a model’s results hold when the information of voters and candidates as some of its assumptions are relaxed. Mc- well. In this experiment, voter subjects were Kelvey and Ordeshook tested a rational expec- assigned an ideal point over a single dimen- tations theory of markets, which dictates that sion; however, unlike the previous experi- information is aggregated so that informed ment, they did not know the location of their traders transmit information about the price or other voters’ ideal points. Candidates also of commodities to uninformed traders, did not know the distribution of the voters’ and, as a result, the market behaves as if it ideal points. During the experiment, two sub- were fully informed. In applying this theory jects posed as candidates and one of the can- to elections, they designed an experiment in didate subjects was randomly chosen to be an which there were informed and uninformed initial incumbent. This subject then selected voters who were divided into three groups a position on the issue space, which trans- and had ideal points on three locations on a lated into a monetary payoff for the voter sub- single dimensional space. Uninformed voters jects based on the location of their ideal point only knew the location of their ideal points relative to the incumbent’s position. The (i.e., which voter groups were to the left and amount of their payoffs was revealed to vot- right of their ideal points) and an interest ers, and they voted either to retain the incum- group endorsement that specified which bent or to vote for the challenger. Between candidate was furthest left. Informed voters twenty-three and forty-five elections were knew the precise location of candidates’ conducted. positions. Candidate subjects did not know Some sessions used a within-subjects the location of the median voter’s ideal point. design, in that they shifted the voter ideal points by thirty-five units in round twenty- 1 McKelvey and Ordeshook conducted an earlier elec- one in order to determine whether candi- tion experiment in 1982, but this experiment only concerned voters and whether they could calculate dates could track the shift in the median. mixed strategy candidate equilibria in their minds. The results showed that candidate positions Electoral Systems and Strategic Voting (Laboratory Election Experiments) 373 did converge to the median voter in both Although the median voter theorem espoused the nonshift and shift experiments. Hence, by Downs was criticized for its simplicity, these experiments show that candidate sub- the experiments showed that this theorem jects largely located the median position, even holds up when its restrictive conditions have when they had no information on its loca- been relaxed, surviving a number of tests that tion. Although voters did not know candidate the laboratory allowed researchers the ability positions based on retrospective evaluations, to implement. Even when voters are unin- they learned within the first ten periods how formed about the exact location of candi- to identify the candidate who was closest to date positions and candidates do not know their ideal points. the exact distribution of voter ideal points, In a variant of this experiment, Collier, the median is still a magnet for electoral Ordeshook, and Williams (1989) allowed outcomes. Although observational data can voter subjects the option of purchasing infor- point to a central tendency of candidate posi- mation about the challenger’s proposed positions in elections, they cannot determine how tion. One hypothesis tested was that when this pull to the median is affected by voter the electoral environment is stable such that information because this variable cannot be policies enacted are invariant from election to controlled, measured, or randomized in real- election, voters will rely on retrospective cues, world elections. but when the political environment is unstable (i.e., when policies vary from election to Models of Candidate Divergence election), voters will invest in information to discover the policy of the challenger. Again, The experiments we discuss emphasize the the researchers used a within-subject design. robustness of the pull of the median for can- To create systems of electoral stability and didates in elections. Although these tenden- instability, “dummy” candidates were used cies exist, it is also well documented that where candidate positions remained constant there are policy differences in candidates and for a fixed number of periods and fluctu- parties; in fact, a number of formal mod- ated for a fixed number of periods.2 Again, els and experimental tests have been pro- the research question was whether voters can posed to explain these differences. One of track shifting candidates’ positions and under the first formal models to explore why can- what conditions voters will purchase informa- didates might diverge in policy positions is tion (stable vs. unstable environments). The Wittman’s (1977) model of candidates with experiments showed that when the electoral divergent policy preferences. Calvert (1985) system was stable, voters tended to purchase demonstrates that for such policy preferences less information and relied more on retro- to lead to divergence of candidates’ posi- spective evaluations, and when the electoral tions in elections, candidates must also be system was unstable, they tended to pur- uncertain about the ideal point of the median chase more information about the incum- voter. Morton (1991) presents an experimen- bent’s positions. In real elections, it is difficult tal test of Calvert’s proposition. In her exper- to measure the costs of being informed for iment, subjects were assigned as candidates, individuals because these costs vary by indi- and their payments depended solely on the vidual, but in the laboratory this cost can be policy position chosen by the winning candi- explicitly measured. date. In the treatment with incomplete infor- The experiments discussed in this sec- mation, the median voter’s ideal point was tion revealed, as the pundits have observed a random draw each period. This treatment in observational elections, that the tendency was compared to one in which the median toward the median distribution of voter ideal voter’s ideal point was constant across peri- points is a powerful pull in electoral politics. ods. Morton finds that indeed, as Calvert predicts, candidates choose more divergent 2 Unbeknownst to voter subjects, candidate subjects were not used; instead, a researcher manually input positions when the median voter ideal point candidate positions. is randomly determined in each period, but 374 Rebecca B. Morton and Kenneth C. Williams converge in positions when the median voter demonstrate that the results are robust to ideal point is fixed. variations in subject pools (the experiments Morton’s (1991) experiment illustrates were conducted at the California Institute how laboratory elections can provide tests of Technology [Caltech] in Pasadena, Cali- of theories that are nearly impossible using fornia, and at Universitate Pompeu Fabra in observable data or field experiments. The Barcelona, Spain) and in framing (some treat- comparative static prediction would be dif- ments at Caltech used a game framing where ficult, if not impossible, to evaluate in obser- subjects made choices in a payoff matrix and vational elections because a researcher would others used a political context). Thus, they only be able to estimate the knowledge of establish not only that the theory is supported candidates and parties about voter prefer- in the laboratory, but also that their results are ences (as well as only be able to estimate robust to a number of external validity tests. those preferences him- or herself) and is Related to the experimental research on unlikely to have exogenous variation in that theories of why candidates might diverge knowledge. Furthermore, Morton’s experi- in candidate policy positions is the work of ment also illustrates how laboratory exper- Houser and Stratmann (2008), who consider iments can provide researchers with new, how candidates with fixed, divergent policy unexpected results that are not observable positions but uncertain quality differences if we only rely on observable studies and might use campaign advertisements to con- field experiments. That is, Morton finds that vey information on their qualities to voters, subjects converge more than is theoretically testing a theory of campaign contributions predicted in the uncertainty condition, sug- posited by Coate (2004). In their experiment, gesting that subjects value winning indepen- subjects are both voters and candidates. Can- dent of their payoffs. Given the ability in the didates are rewarded solely based on whether laboratory to control both the information they win an election, and different schemes subjects had and their monetary or finan- for paying for campaign advertisements are cial payoffs from the election, it was pos- considered (i.e., sometimes campaign adver- sible to discern that the subjects received tisements reduce voter payoffs if the adver- intrinsic, nonmonetary payoffs from winning, tiser wins, as if the campaign spending was something that would be extremely tough to funding by special interests, and other times observe outside the lab. campaign advertisements are not costly to An alternative explanation for candidate voters, although always costly to candidates). divergence is the existence of valence quality Note that they use a within-subject design so differences between candidates (differences they are able to measure the effects of the dif- that are independent of policy positions), ferent finance schemes, while controlling for posited in Londregan and Romer (1993), specific subject effects. Groseclose (2001), and Aragones and Palfrey Houser and Stratmann (2008) find that (2004). Aragones and Palfrey present experi- indeed candidates use advertisements to con- mental tests of a theory that suggests that can- vey policy information and that voters use didates who are perceived by voters to have that information to choose candidates who lower valence quality advantages will adopt provide them with greater payoffs. Further- more extreme positions and that, similar to more, they find that less information is pro- Calvert’s (1985) results, the more uncertain vided to voters when campaign contribu- the position of the median voter, the greater tions are costly in terms of voter payoffs. the divergence of the candidates in policy Houser and Stratmann, because they can positions. Aragones and Palfrey find that control and randomize how campaign con- these theoretical predictions are supported in tributions directly affect voter payoffs, pro- the laboratory elections.3 Furthermore, they vide useful information on how campaign

candidate quality differences in experiments on the 3 Aragones and Palfrey’s experimental design builds principal-agent problem between voters and candi- on earlier work of Dasgupta and Williams (2002)on dates. Electoral Systems and Strategic Voting (Laboratory Election Experiments) 375

Table 26.1: Payoff Schedule the same). Also, assume that there are four type 1 voters, four type 2 voters, and six type Election Winner 3 voters. Voter Total No. In this configuration, candidate C is a Con- Type A B C of Each Type dorcet loser because he or she would lose in 1 (A) $1.20 $0.90 $0.20 4 a two-way contest against either of the other 2 (B) $0.90 $1.20 $0.20 4 candidates. However, if each type voted their 3 (C) $0.40 $0.40 $1.40 6 sincere preference in a plurality election, then candidate C, the Condorcet loser, would be Source: Forsythe et al. (1993). the winning candidate. The problem is how can type 1 and 2 voters coordinate their votes advertisements financed by interest groups to prevent candidate C, their least preferred might affect voter information, voter choices, candidate, from winning? Should type 1 and and electoral outcomes. Such control and 2 voters vote strategically for candidate B or randomization are largely impossible outside A? Without a coordination device, subjects the laboratory. Furthermore, a within-subject in the type 1 and 2 groups behave poorly and design, which allows for the ability to control fail to coordinate, so the Condorcet loser wins for subject-specific effects, cannot be done about 87.5 percent of the time. outside the laboratory. Three types of coordination devices were instituted in the experiments: polls, history of past elections, and ballot location. Polls in Strategic Voting and Coordination the experiment were implemented by allow- in Three-Candidate Elections ing subjects to vote in a nonbinding election, Strategic voting in three-candidate elections where the results were revealed to all sub- means that a voter realizes that his or her most jects and then a binding election took place. preferred candidate will lose the election so The theory of Myerson and Weber (1993) he or she votes for his or her second preferred provided no predictions as to whether any candidate to prevent his or her least preferred of these coordination devices would work or candidate from winning the election. The whether one would be more successful than problem is that a substantial number of other the others. Thus, the experiments provided voters in the same situation must also decide new information for our understanding of to vote strategically so that their least pre- voter coordination. The researchers found ferred candidate does not win. Hence, it is a that with the use of polls the Condorcet loser coordination problem among voters whereby only won thirty-three percent of the time. they figure out that supporting their second Also, when either A or B was leading in the most preferred candidate leads to a better out- poll results, the Condorcet loser only won six- come (i.e., their least preferred candidate is teen percent of the time. The experiment also not elected). found a small bandwagon effect in which the Forsythe et al. (1993) consider experimen- candidate who won in a past election garnered tally the voter coordination problem in strate- more support, and there was also a small gic voting situations using an example from ballot location effect. Thus, the researchers the theoretical work of Myerson and Weber found that polls were only weak coordination (1993). They present a simple example that devices, albeit stronger than the other alter- illustrates the problem voters confront, as natives. detailed in Table 26.1.InTable 26.1, there Using a similar design, Rietz, Myerson, are three types of voter preferences where and Weber (1998) consider another coordi- type 1 prefers A to B to C because the mone- nation device, campaign contributions where tary amount associated with each candidate is subjects can purchase ads for candidates. higher. Type 2 voters prefer B to A to C, and They find that campaign contributions are type 3 voters prefer C and are indifferent to more successful than polls as a coordina- B and A (because the monetary amounts are tion device. Note that, unlike the Houser 376 Rebecca B. Morton and Kenneth C. Williams and Stratmann (2008) experiments, the ads voters (six are the minority type) only get provided no information to other voters utility from C. Hence, like the experiment about voter payoffs but were merely ways previously described, candidate C is the Con- in which voters could attempt to coordi- dorcet loser. In straight voting, voters can cast nate before an election. Morton and Rietz one vote for up to two candidates, whereas in (2008) analyze majority requirements as coor- cumulative voting they can cast two votes for dination devices. In contrast to the other one candidate. Voters can also abstain. They devices, majority requirements theoretically find that minority representation increases should result in full coordination because with the use of cumulative voting because the requirements theoretically eliminate the minority voters can cumulate their votes on equilibria where the Condorcet loser wins. the minority candidate. Again, the use of the They find that, indeed, majority require- laboratory allows for the researchers to make ments are far more effective as coordination comparisons, holding voter preferences con- devices than the others studied. stant; again, this is not possible in observa- Forsythe et al. (1996) conducted a related tional data or in field experiments. experiment using the same preference profile These experiments illustrate how, by using that replicates the “Condorcet loser” para- laboratory elections, it is possible to vary dif- dox, but they varied the voting rules. They ferent aspects of an election – whether there altered three alternative voting rules: plural- is a poll, whether campaign contributions are ity, approval voting, and the Borda count. allowed, whether majority requirements are Under the plurality rule, voters voted for just instituted, and whether alternative voting sys- one candidate; under approval voting, sub- tems are instituted – and thus measure the jects voted for as many of the candidates as causal effects of these different aspects, again they wanted to; and under the Borda rule, something not possible using observational voters were required to give two votes to their data or field experiments. Real-world elec- two most preferred candidates and one vote tions have not provided sufficient observa- to their second preferred candidate. Although tional data on these electoral mechanisms, yet there were multiple equilibria, in general, laboratory elections are able to test the effi- under the plurality rule, they predicted the ciencies of these mechanisms. Condorcet loser would win more elections; under approval voting, the Condorcet loser Sequential Voting would win less often; and under the Borda rule, the Condorcet loser would lose even Morton and Williams (1999, 2001) examine more elections. The results generally sup- multicandidate elections to test the difference port the equilibrium predictions. Hence, the between simultaneous and sequential voting Borda and approval rules were more efficient in terms of their impact on voter behavior in defeating the Condorcet loser. The abil- and electoral outcomes. Simultaneous vot- ity to hold preferences of voters over can- ing represents the American general presi- didates constant and to vary the electoral dential elections where voters vote once, the rule provided researchers with the opportu- results are revealed, and a winning candidate nity to gain causal information on the effects is announced. Sequential voting represents of these rules, which would not be possible American presidential primaries where voting using observational data or field experiments. takes place over time in stages, and the win- Gerber, Morton, and Rietz (1998) used a ning candidate is the one who accumulates similar candidate profile to examine cumu- the most votes over the stages. The two main lative versus straight voting in multimem- hypotheses that Morton and Williams test are ber districts (i.e., three candidates compete whether sequential elections give candidates for two seats). In their setup, there are three who are well known to voters an electoral voter types with the following preferences: advantage and whether sequential elections type 1 voters (4) prefer A to B to C, type lead to more informed voter decisions. The 2 voters (4) prefer B to A to C, and type 3 voters in the experiment have preferences Electoral Systems and Strategic Voting (Laboratory Election Experiments) 377 over three candidates, where group x has ten Table 26.2: Payoff Schedule members and prefers x to y to z, group z has ten members and prefers z to y to x, and group Voter Type Is Green y has four members and prefers y and is indif- G Wins Election R Wins Election ferent to x and z. In this case, y is the Con- dorcet winner and will beat the other candi- Vote G 1.50 0.0 dates in a pairwise competition, but will lose Vote R 0.25 1.0 if all voters vote sincerely, in which case x and z will tie. Source: Dasgupta et al. (2008). In the simultaneous treatment, the identity of one candidate was revealed to voters voted (i.e., the first voter does not have any prior to the casting of votes. This was an vote information, and the last voter knows incomplete information treatment in which how everyone else has voted). In this experi- one candidate is better known to voters. In ment, voters maximize their payoff when they the sequential treatment, voters were divided vote for the alternative that garners a majority into two groups, A and B, where group A of the votes. This experiment examines situ- voted first, followed by group B, and then ations where a voter is concerned with voting the results were tallied across the two groups. for the majority preferred alternative, or what There were treatments that varied the level is referred to as “conformity voting” (Cole- of information had by voters, but the pri- man 2004). Subjects were randomly assigned mary treatment involved letting group A vot- one of two types (they either preferred green ers know the identity of the candidate and or red). Subjects in the experiment were group B would have poll results, or they shown a payoff matrix (Table 26.2). would know how group A voted. The results This is a payoff matrix for subjects who show that later voters, or group B voters, were randomly assigned to be a green type. If were able to use poll results to make more a subject is randomly assigned to be a green informed decisions that reflected their pref- type and he or she believes that the green can- erences. The researchers also find that under didate will win the election, then he or she certain conditions when the Condorcet win- should vote for the green candidate because ner, y, is unknown, this candidate does better this will yield him or her $1.50 forthiselec- under sequential voting than under simulta- tion period. However, if he or she believes neous voting. The laboratory provides Mor- that green will lose the election, then he or she ton and Williams a unique opportunity to should vote for the red candidate and receive compare the effects of these two voting sys- $1.00, instead of receiving nothing when he tems, simultaneous and sequential, on the or she votes for the green candidate and election outcomes and to compare voter the green candidate loses. information about their choices. Given that Two information conditions were varied; presidential elections occur only every four one where subjects only knew their type, years and that there is considerable variation and the other where they knew the type of over time in the candidates, voters, and count- all other subjects. For full information, the less other factors that confound such a com- equilibrium prediction is that all voters who parison in observational data, the laboratory are in the majority should vote for the most experiments provide information that cannot preferred candidate. The equilibrium predic- be learned otherwise. tion for the incomplete information treat- In a related setup that is a blend of Morton ment depends on the order in which types and Williams and the Forsythe et al. (1993, are revealed, but generally voters later in 1996) experiments, Dasgupta et al. (2008) the voting queue should switch their votes examine coordinated voting in a five-person more often (i.e., we should see more strate- sequential voting game. In this experiment, gic voting). The results generally confirm the five subjects vote sequentially, and each sub- equilibrium predictions. As in the Morton ject knows how each subject prior to them has and Williams (1999, 2001) experiment, these 378 Rebecca B. Morton and Kenneth C. Williams results show that in elections where sequen- the researchers find that the later voters are tial choice is the chosen voting rule, such as in advantaged by having more information to European Union elections, voters at different determine when to abstain. However, they positions in the voting queue will be voting also find that subjects often make choices at with different available information, and this variance with the theoretical predictions, vot- can have an effect on electoral outcomes. ing more often than theoretically predicted in These aforementioned experiments on the sequential voting elections. Again, these sequential voting elections do not allow experiments illustrate the effect of sequential for abstention. Battaglini, Morton, and Pal- elections in terms of voter turnout. Because frey (2007) examine the difference between sequential elections are not that common, sequential and simultaneous voting with two laboratory election experiments are able to choices under uncertainty when there is a illustrate that these types of elections do affect cost to voting and abstention is allowed. In voting and turnout behavior. the experiment they conduct, there are two voting cost treatments: a low-cost environ- Turnout Experiments ment and a high-cost environment. In this experiment, the researchers used a nonpolit- One of the riddles in the voting literature is ical frame, unlike those previously discussed. the basic question of why people vote. The That is, in this case, subjects were presented voting paradox posits that in large elections with two jars (via a computer image) where the probability of one vote affecting the out- one jar contained six red balls and two blue come is relatively small, almost zero, so that balls, and the other jar contained six blue if there is a cost to voting, then actually going balls and two red balls. The subjects then to the polls to vote will outweigh the benefits selected a ball from one of the jars, which (i.e., being a pivotal vote) and, therefore, it was only revealed to that subject. Subjects is not rational to vote; yet, we observe large then guessed which jar was the correct one; number of voters participating in elections. they could guess jar 1,jar2, or abstain. In the In an early attempt to address this paradox, simultaneous treatment, subjects all guessed Ledyard (1984) and Palfrey and Rosenthal or abstained together, and in the sequential (1985) noted that the probability that a vote treatment, subjects were assigned a position is pivotal is endogenously determined and (first, second, or third) and guessed in that that, when candidates have fixed policy posi- order. If a majority of the subjects guessed tions, equilibria exist with positive turnout for the correct jar, then all subjects received and purely rational voter decision making. a payoff minus the cost of voting. This is because, if everyone assumed that The theory predicted that, in simulta- the probability of being pivotal was smaller neous voting, the probability of abstention than the cost of voting and, consequently, no decreases with the cost of voting. In sequen- one voted, then the probability of being piv- tial voting under low cost of voting, early otal for any one voter would be 100 percent voters should bear the cost and later voters because that one voter would determine the should vote as if their vote is pivotal, whereas outcome. Thus, when electorates are finite with high costs later voters will be forced to and candidate positions are fixed, endogene- bear the cost. In terms of equity of the elec- ity of pivotality means that equilibria with toral mechanism, they show that simultane- positive voting are possible. ous voting is more equitable as opposed to Levine and Palfrey (2007) conducted the sequential elections because all voters derive first direct experimental tests of the Palfrey the same expected utility, although sequen- and Rosenthal (1985) model. They find that tial voting may lead to higher aggregate pay- the comparative statics of the theory are sup- offs and be more economically efficient. As ported in the laboratory – specifically, turnout predicted, the researchers find that absten- decreases when the electorate increases (size tion increases with voting costs in simultane- effect), turnout is higher when the election ous voting. In the case of sequential voting, is close (closeness effect), and supporters of Electoral Systems and Strategic Voting (Laboratory Election Experiments) 379 minority candidates turn out in greater num- when voting is costless, for example, when bers than do those of majority candidates (an voters choose to not vote in some races while underdog effect). Levine and Palfrey also find in the voting booth. Feddersen and Pesendor- that turnout is higher than predicted in the fer (1996) create a model of elections between large elections (and smaller than predicted two alternatives, one in which all voters have in the small elections), which they explain to the same preferences (common preferences) be a consequence of voter errors. Levine and but are asymmetrically informed and one in Palfrey’s experiments demonstrate the advan- which some voters are more informed than tage of the laboratory, in that they can both other voters. They show that when the poorly control and manipulate a number of impor- informed voters are indifferent between the tant theoretical independent variables that are alternatives, in equilibrium it is optimal for difficult to control or measure in the field, these voters to abstain and let the more such as voter costs, electorate size, and size of informed voters vote because their misguided the majority, while holding voter preferences choices may sway the election in the wrong constant. direction. Consequently, even if there is no An alternative model of voter turnout is cost to voting, it is rational for the poorly that voters respond to group influences and informed to forgo voting and delegate the are not purely individually motivated as for- decision to the more informed voters. This mulatedinMorton(1987, 1991), Uhlaner phenomenon is referred to as the “swing (1989), and Schram and Van Winden (1991). voter curse” (SVC). One corollary of this the- The Morton and Uhlaner models view ory is that if these uninformed voters know turnout decisions as responses to group leader that some voters are partisans and will vote manipulations, whereas Schram and Van for their favored candidates regardless of the Winden posit that groups have a psychologi- information, then the uninformed voters will cal impact on how individuals vote. Feddersen abstain less often and choose to vote in order and Sandroni (2006) provide a micromodel to cancel out the votes of the partisans, even of how groups might influence turnout, in if doing so is contrary to their prior informa- which some voters turn out even when their tion about which choice is best for them. To votes are not likely to be pivotal due to ethical understand the intuition of this result, sup- concerns. pose that there are two options before voters, A number of experimentalists have con- a and b, and two states of the world, A and B. sidered the group turnout models. For exam- A set of voters (swing voters) prefers option a ple, Schram and Sonnemans (1996a, 1996b) in state of the world A and option b in state and Grosser and Schram (2006) present of the world B. Some of the swing voters are experimental evidence on the impact that psy- informed and know for sure the true state of chological influence and contact with other the world, whereas others are uninformed and voters has on voters’ turnout decisions. Gail- only have probabilistic information about the mard, Feddersen, and Sandroni (2009)pro- true state of the world. Their prior informa- vide an interesting experimental test of the tion is that state of the world A is more likely ethical voter model in that they vary both than state of the world B. However, there is the probability that a subject’s vote is deci- also a group of partisan voters who will always sive in the outcome and the benefit to other vote for option a regardless of the state of the voters from voting. They find support for the world. In the event that an uninformed voter argument that ethical motives might explain is pivotal, then it is likely that all informed turnout decisions. These results may also par- voters are voting for b because partisans vote tially explain the tendency of excessive voting for a. Thus, uninformed voters have an incen- in the large elections found by Levine and tive to offset the partisan votes and vote for b Palfrey (2007). as well, even though their prior information The models we have thus far discussed suggests that the state of the world is A. examine turnout decisions when voting is Battaglini, Morton, and Palfrey (2009) test costly. However, voters often abstain even this theory using a similar procedure as in 380 Rebecca B. Morton and Kenneth C. Williams their experiment on sequential voting (previ- decide. They find that the tendency to dele- ously discussed). In this experiment, subjects gate is so strong that even in a voting game were shown one of two colored jars on their where such delegation by all less informed computer screens, and subjects voted for the voters is not an equilibrium, subjects were jar that was randomly deemed to be the cor- drawn to such behavior. These results sug- rect jar. Some voters were notified as to which gest that the tendency observed by Battaglini jar was the correct jar, and other voters were et al. may reflect a norm of behavior to del- not. The results show that uninformed vot- egate to experts that may not always be opti- ers delegated their vote or abstained about mal. The ability to discern the importance of ninety-one percent of the time. They also such a norm that the laboratory provides is yet find evidence that when uninformed voters another example of what one can learn from a know partisans are voting in the election, they laboratory election that is not generally possi- tend to abstain less often and to vote in order ble using observational or field experimental to cancel out the votes of the partisans. The data alone. results further show that turnout and margin Other experimental research demonstrates of victory tend to increase with the number of that the tendency of uninformed voters to informed voters in the election environment. abstain when voting is costless appears to It is noteworthy that uninformed voters voted be related to experimental context and other to cancel out the votes of the partisans, even aspects of the voting situation. Houser, Mor- though doing so involved voting against their ton, and Stratmann (2008) extend the afore- own prior information in some of the treat- mentioned Houser and Stratmann (2008) ments. As Lassen (2005) explains, this type experiments to allow voters to abstain. Thus, of nuanced voting behavior is unlikely to be their experiments consider the extent that measurable in observational elections, even in uninformed voters abstain in a situation in the best of circumstances (when information which campaign information is endogenous is provided to voters arguably exogenously). and may be ambiguous (i.e., when the adver- Thus, the laboratory provides an important tising is funded at a cost to voters). They environment for testing these sorts of pre- find that abstention rates of uninformed vot- cise predictions that are largely unobservable ers are higher, as is theoretically predicted, outside the laboratory. but are much lower than those found by Morton and Tyran (2008),usinganexper- Battaglini et al. (2009). They also find that imental design similar to that of Battaglini all voters, both informed and uninformed, et al. (2009), examine voting cases where vot- abstain significantly more when campaign ers vary in information quality, but no voter information is costly to voters. These results is completely uninformed and no voter is provide unique information on the effects completely informed. In the games explored, of campaign financing of information on there are multiple equilibria: equilibria where voter turnout that is difficult to observe in only highly informed voters participate (SVC observational elections or to manipulate in Equilibria), and equilibria where all voters field experiments because such an experiment participate (All Vote Equilibria). In some would require manipulating how candidates cases, the Pareto-optimal equilibrium (i.e., finance their campaigns – something to which the equilibrium that provides subjects with we would expect only a rare viable candidate the highest expected payoffs) is SVC, and in in an election to consent. others, it is All Vote, in contrast to Battaglini et al., where SVC is always Pareto optimal. Morton and Tyran find that the tendency of 3. Conclusion less informed voters to abstain is so strong that, even in the cases where it is Pareto opti- As we illustrate, laboratory election experi- mal for all to vote and share information, ments allow researchers to conduct tests of less informed voters delegate their votes to predictions from formal models, both rela- the more informed voters, letting the experts tionship and dynamic predictions, and to Electoral Systems and Strategic Voting (Laboratory Election Experiments) 381 conduct stress tests where our theory does can be generalized to variations in treat- not give us a guide. The laboratory provides ments and target populations; this can only researchers with the ability to control many be shown empirically through replication, not aspects of an election environment, enabling through logic or supposition. In behavioral researchers to better measure the causal economics, experimental economists have effects of different election variables, such engaged in an increasing amount of such as information, timing of voting, variations replication, taking many of the basic exper- in voting rules, coordination devices such as iments in economics to new target popula- polls, and ethical motivations on voter and tions across the world, allowing for a better candidate choices. Laboratory experiments understanding of the validity of the results have provided significant support for the the- from these experiments.5 We believe that the oretical predictions of the median voter the- next challenge facing experimental political orem, even in situations where voter and scientists is to similarly consider the exter- candidate information is limited. Further- nal validity of laboratory election experiments more, laboratory experiments have allowed by conducting such experiments with new for researchers to investigate voting mecha- and different target populations, as in the nisms that are difficult to study observation- Aragones and Palfrey (2004) experiment dis- ally because they are rarely used and their cussed in this chapter. In other experimental effects on individual voters are often impos- disciplines, it is common to engage in repli- sible to determine. cation and to use such replications to con- One concern about laboratory election duct meta-analyses and systematic reviews to experiments is the costs associated with con- determine which results are valid across the ducting these types of experiments. Because variety of experimental treatments and target these experiments assume induced value the- populations. It is time for political scientists ory, subjects must be financially rewarded to investigate whether these important results based on their performance in the exper- from laboratory experimental elections are iment, and these costs can be prohibitive externally valid the only way that such an for some researchers. However, there are investigation can be done – by conducting many methods to satisfy induced value the- more such experiments but varying the sub- ory and keep the costs low, such as paying jects used and the treatments, as is done in subjects only for randomly selected rounds other experimental disciplines. (see Morton and Williams [2010] for other In addition to replication issues, other techniques.) Programming costs must also future developments for laboratory election be taken into account because most of these experiments are the various extensions that experiments are conducted on a computer are applicable for the experiments discussed network, and software must be developed to in this chapter. For example, varying the pref- operationalize the experiment in the labo- erences and motivations of voters, informa- ratory. However, there is free software that tion that voters and candidates possess about allows researchers to design network-based the election environment, number of vot- experiments, even with little programming ers in the elections, and different types of knowledge.4 electoral mechanisms are all valid research Laboratory election experiments have agendas. We also want to see more collab- been criticized for failing to provide exter- orative projects. Currently, the number of nally valid claims because of the artificiality subjects who participate in an election exper- of the laboratory. However, external validity iment tends to be rather small due to the does not refer to whether the laboratory elec- costs mentioned previously. However, we can tion resembles some election in the observ- imagine, as a result of Internet collaboration, able world, but rather to whether the results 5 See, for example, the recent meta-analysis of ultimatum game experiments conducted using a variety of 4 See www.iew.uzh.ch/ztree/index.php (November 15, subject pools in Oosterbeek, Sloof, and Van de Kulien 2010). (2004). 382 Rebecca B. Morton and Kenneth C. Williams projects where laboratories across the nation Dasgupta, Sugato, and Kenneth C. Williams. or around the world could participate in 2002. “A Principal-Agent Model of Elections large-scale experiments and share in the with Novice Incumbents.” Journal of Theoreti- 14 409 38 costs. cal Politics : – . Downs, Anthony. 1957. An Economic Theory of Democracy. New York: Harper & Row. References Feddersen, Timothy J., and Wolfgang Pesendor- fer. 1996. “The Swing Voter’s Curse.” Ameri- Almond, Gabriel, and Sidney Verba. 1963. The can Economic Review 86: 408–24. Civic Culture. Princeton, NJ: Princeton Uni- Feddersen, Timothy, and Alvaro Sandroni. 2006. versity Press. “A Theory of Participation in Elections with Aragones, Enriqueta, and Thomas R. Palfrey. Ethical Voters.” American Economic Review 964: 2004. “The Effect of Candidate Quality 1271–82. on Electoral Equilibrium: An Experimental Forsythe, Robert, Roger B. Myerson, Thomas Study.” American Political Science Review 90: 34– Rietz, and Robert Weber. 1993. “An Exper- 45. imental Study on Coordination in Multi- Battaglini, Marco, Rebecca B. Morton, and Candidate Elections: The Importance of Polls Thomas R. Palfrey. 2007. “Efficiency, Equity, and Election Histories.” Social Choice and Wel- and Timing in Voting Mechanisms.” American fare 10: 223–47. Political Science Review 101: 409–24. Forsythe, Robert, Roger B. Myerson, Thomas Battaglini, Marco, Rebecca B. Morton, and Rietz, and Robert Weber. 1996. “An Experi- Thomas R. Palfrey. 2009.“TheSwingVoter’s mental Study of Voting Rules and Polls in a Curse in the Laboratory.” Review of Economic Three-Way Election.” International Journal of Studies 77: 61–89. Game Theory 25: 355–83. Berelson, Bernard R., Paul F. Lazarsfeld, and Gailmard, Sean, Timothy Feddersen, and Alvaro 2009 William N. McPhee. 1954. Voting. Chicago: Sandroni. . “Moral Bias in Large Elec- The University of Chicago Press. tions: Theory and Experimental Evidence.” Calvert, Randall L. 1985. “Robustness of the Mul- American Political Science Review 103: 175– tidimensional Voting Model, Candidate Moti- 92. vations, Uncertainty and Convergence.” Amer- Gerber, Elisabeth, Rebecca B. Morton, and 1998 ican Journal of Political Science 29: 69–95. Thomas Rietz. . “Minority Representation Coate, Stephen 2004. “Pareto-Improving Cam- in Multimember Districts.” American Political 92 127 44 paign Finance Policy.” American Economic Science Review : – . 2001 Review 94: 628–55. Groseclose, Tim. . “A Model of Candidate Coleman, Stephen. 2004. “The Effect of Social Location When One Candidate Has a Valence Conformity on Collective Voting Behavior.” Advantage.” American Journal of Political Science 45 862 86 Political Analysis 12: 76–96. : – . Collier, Kenneth E., Richard D. McKelvey, Grosser, Jens, and Arthur Schram. 2006. “Neigh- Peter Ordeshook, and Kenneth C. Williams. borhood Information Exchange and Voter Par- 1987. “Retrospective Voting: An Experimental ticipation: An Experimental Study.” American 110 235 48 Study.” Public Choice 53: 101–30. Political Science Review : – . Collier, Kenneth E., Peter Ordeshook, and Ken- Houser, Daniel, Rebecca B. Morton, and Thomas neth C. Williams. 1989. “The Rationally Unin- Stratmann. 2008. “Turned Off or Turned Out? formed Electorate: Some Experimental Evi- Campaign Advertising, Information, and Vot- dence.” Public Choice 60: 3–39. ing.” Working Paper 1004, George Mason Converse, Phillip E. 1975. “Public Opinion and University, Interdisciplinary Center for Eco- Voting Behavior.” In Handbook of Political Sci- nomic Science. ence, eds. Fred I. Greenstein and Nelson W. Houser, Daniel, and Thomas Stratmann. 2008. Polsby. Reading, MA: Addison-Wesley, 75– “Selling Favors in the Lab: Experiments on 169. Campaign Finance Reform.” Public Choice 136: Dasgupta, Sugato, Kirk A. Randazzo, Regi- 215–39. nald S. Sheehan, and Kenneth C. Williams. Lassen, David. 2005. “The Effect of Information 2008. “Coordinated Voting in Sequential and on Voter Turnout: Evidence from a Natural Simultaneous Elections: Some Experimental Experiment.” American Journal of Political Sci- Results.” Experimental Economics 11: 315–35. ence 49: 103–18. Electoral Systems and Strategic Voting (Laboratory Election Experiments) 383

Ledyard, John O. 1984. “The Pure Theory of Morton, Rebecca B., and Kenneth C. Williams. Large Two-Candidate Elections.” Public Choice 2001. Learning by Voting. Ann Arbor: University 44: 7–41. of Michigan Press. Levine, David K., and Thomas R. Palfrey. 2007. Morton, Rebecca B., and Kenneth C. Williams. “The Paradox of Voter Participation: A Labo- 2010. Experimental Political Science and the Study ratory Study.” American Political Science Review of Causality: From Nature to the Lab. New York: 101: 143–58. Cambridge University Press. Londregan, John, and Thomas Romer. 1993. Myerson, Robert B., and Robert J. Weber. 1993. “Polarization, Incumbency, and the Personal “A Theory of Voting Equilibria.” American Vote.” In Political Economy: Institutions, Competi- Political Science Review 87: 102–14. tion, and Representation, eds. William A. Barnett, Oosterbeek, Hessel, Randolph Sloof, and Gijs Van Melvin J. Hinich, and Norman J. Schoﬁeld. de Kulien. 2004. “Cultural Differences in Ulti- Cambridge: Cambridge University Press, 355– matum Game Experiments: Evidence from a 78. Meta-Analysis.” Experimental Economics 7: 171– McKelvey, Richard D., and Peter C. Ordeshook. 88. 1982. “Two-Candidate Elections without Palfrey, Thomas R., and Howard Rosenthal. 1985. Majority Rule Equilibria.” Simulation and “Voter Participation and Strategic Uncer- Games 3: 311–35. tainty.” American Political Science Review 79: 62– McKelvey, Richard D., and Peter C. Ordeshook. 78. 1985a. “Elections with Limited Information: Rietz, Thomas A., Robert B. Myerson, and Robert A Fulﬁlled Expectations Model Using Con- J. Weber. 1998. “Campaign Finance Levels temporaneous Poll and Endorsement Data as as Coordinating Signal in Three-Way Exper- Informational Sources.” Journal of Economic imental Elections.” Economics and Politics 10: Theory 36: 55–85. 185–217. McKelvey, Richard D., and Peter C. Ordeshook. Schram, Arthur, and Joep Sonnemans. 1996a. 1985b. “Sequential Elections with Limited “Voter Turnout as a Participation Game: Information.” American Journal of Political Sci- An Experimental Investigation.” International ence 29: 480–512. Journal of Game Theory 25: 385–406. Morton, Rebecca B. 1987. “A Group Major- Schram, Arthur, and Joep Sonnemans. 1996b. ity Voting Model of Public Good Provision.” “Why People Vote: Experimental Evidence.” Social Choice and Welfare 4: 117–31. Journal of Economic Psychology 17: 417– Morton, Rebecca B. 1991. “Groups in Rational 42. Turnout Models.” American Journal of Political Schram, Arthur, and Frans Van Winden. 1991. Science 3: 758–76. “Why People Vote: Free Riding and the Pro- Morton, Rebecca B., and Thomas A. Rietz. 2008. duction and Consumption of Social Pres- “Majority Requirements and Minority Repre- sure.” Journal of Economic Psychology 12: 575– sentation.” New York University Annual Survey 620. of American Law 63: 691–726. Smith, Vernon L. 1976. “Experimental Eco- Morton, Rebecca B., and Jean-Robert Tyran. nomics: Induced Value Theory.” American Eco- 2008. “Let the Experts Decide: Asymmet- nomic Review 66: 274–79. ric Information, Abstention and Coordina- Uhlaner, Carole Jean. 1989. “Relational Goods tion in Standing Committees.” Unpublished and Participation: Incorporating Sociability manuscript, New York University. into a Theory of Rational Action.” Public Choice Morton, Rebecca B., and Kenneth C. Williams. 62: 253–85. 1999. “Information Asymmetries and Simul- Wittman, Donald. 1977. “Candidates with Pol- taneous versus Sequential Voting.” American icy Preferences: A Dynamic Model.” Journal of Political Science Review 93: 51–67. Economic Theory 14: 180–89. CHAPTER 27

Experimental Research on Democracy and Development

Ana L. De La O and Leonard Wantchekon

Expectations about the role of democracy macropolitical examinations (e.g., are democ- in development have changed considerably racies better at producing development than in recent years. In principle, the exercise of are authoritarian regimes?) to microexpla- political rights sets democracies apart from nations (e.g., under what circumstances can other political regimes in that voters can voters limit bureaucrats’ rent-seeking behav- pressure their representatives to respond to ior?). However, the bulk of empirical evi- their needs. It has been argued that such dence in this respect is inconclusive (Prze- pressure “helps voters constrain the confisca- worski and Limongi 1997; Boix and Stokes tory temptations of rulers and thereby secure 2003; Keefer 2007). Is democracy a require- property rights; increases political account- ment for development, or is it the other way ability, thus reduces corruption and waste; around? Are formal institutions the causes or and improves the provision of public goods the symptoms of different levels of develop- essential to development” (Boix and Stokes ment? Which should come first – property 2003, 538). Thus, the argument follows, rights or political competition? Civil liberties democracy is development enhancing. Yet, or public service provision? Why are elec- deprivations such as malnutrition, illiteracy, tions compatible with rampant corruption? and inequalities in ethnic and gender rela- As critical as these questions are to the dis- tionships have proven to be resilient, even cipline, what we know thus far is plagued within the nearly two thirds of the world’s by problems of simultaneous causality, spu- countries ranked as electoral democracies. rious correlations, and unobserved selection The persistence of deprivations is a reminder patterns. that there is still a great deal to be learned Recently, experimental research on the about the relationship between democracy political economy of development has blos- and development. somed. Despite its novelty, progress has Not surprisingly, scholars have explored been rapid and continues apace. As experi- numerous ways in which democracy can ments in this field have evolved, several fea- be related to development, ranging from tures distinguish them from earlier empirical

384 Experimental Research on Democracy and Development 385 contributions. First, scholars have started to Because experimentation is still a novel address central debates in the field by map- research tool in the field, throughout this ping broad theoretical issues to more spe- chapter we review some of the ongo- cific and tractable questions (Humphreys and ing and published research projects that Weinstein 2009). For example, instead of illustrate how random assignment is being asking how different political regimes shape used to tackle questions about the political development, recent studies ask whether var- economy of development. We begin Sec- ious methods of preference aggregation pro- tion 1 by considering examples of pioneer- duce divergent provisions of public goods. ing field experiments executed in collabo- Second, unlike previous macrostudies based ration with nongovernmental organizations on cross-country regressions, recent work (NGOs). Section 2 describes two unique field has focused on the subnational level. Third, experiments performed in partnership with researchers are increasingly making use of political parties. Section 3 presents several field experiments to study how politics affects studies that took advantage of natural exper- development and how development shapes iments, whereas Section 4 introduces the politics in developing countries. use of laboratory and laboratory-in-the-field Throughout this chapter, as in the rest of experiments. Section 5 discusses some of the this volume, when we speak of experiments challenges faced by researchers conducting we mean research projects where the subexperiments on development and democracy, jects under study are randomly assigned to such as internal and external validity, as well different values of potentially causal variables as ethical issues. This section also presents (i.e., different treatment and control groups). practical solutions to some of these challenges For example, a researcher might assign two drawing from recent experimental designs. groups of households to each receive a cash In Section 6, we conclude that, despite transfer, making one of the transfers condi- the challenges, experiments are a promis- tional on parents investing in their children’s ing research tool that have the potential to education. In some designs, there is also a make substantial contributions to the study control group that does not receive any treat- of democracy and development, not only by ment. As Druckman et al. explain in the intro- disentangling the causal order of different duction to this volume, random assignment components of democracy and development, means that each entity being studied has an but also by providing evidence that other equal chance to be in a particular treatment empirical strategies cannot produce. Mov- or control condition. ing forward, we argue that the best of the Experimentation in the field of politi- experimental work in the field of democracy cal economy of development has taken sev- and development should reflect well-chosen eral forms: 1) the increasingly popular field populations and a deep understanding of the experiments take place in a naturally occur- interaction of the interventions with their ring setting; 2) laboratory experiments occur contexts. It should also test theoretical mech- in a setting controlled by the researcher; 3) anisms such that scientific knowledge starts laboratory experiments in the field resem- to accumulate. ble field experiments more generally in that interventions take place in a naturally occurring setting, but researchers have more con- 1. Field Experiments in trol over the setting and the treatment; 4) Collaboration with survey experiments involve an intervention Nongovernmental Organizations in the course of an opinion survey; and 5) some interventions of theoretical interest are Olken’s (2010) study of two political mecha- randomly assigned, not by researchers but nisms – plebiscites and meetings – in Indone- by governments. We group studies that take sia illustrates the use of field experiments advantage of this type of randomization in the to test a particular angle of the relation- category of natural experiments. ship between democracy and development. 386 AnaL.DeLaOandLeonardWantchekon

Although most previous work on the topic ences in the type and location of projects can takes institutions as a given and studies their be adjudicated with certainty to the political effects (Shepsle 2006), Olken’s study starts mechanism in place. from the recognition that, in countless exam- Olken’s (2010) experiment is an example ples, institutions and the public policies that of a growing trend in political science and follow them are endogenous. development economics where researchers Olken (2010), with support from the collaborate with NGOs in order to imple- World Bank and UK’s Department for Inter- ment an intervention and evaluate its effects. national Development, conducted a field This type of partnership has proven fruit- experiment in forty-eight Indonesian villages, ful for the study of a vast array of topics each of which was preparing to petition for central to our understanding of the rela- infrastructure projects as part of the Indone- tionship between democracy and develop- sian Kecamatan Development Program. All ment. For example, Humphreys, Masters, villages in the experiment followed the same and Sandbu (2006) explore the role of lead- agenda-setting process to propose two infra- ers in democratic deliberations in Sao˜ Tome´ structure projects – one general project deter- and Prıncipe;´ Bertrand et al. (2007) collab- mined by the village as a whole, and one orate with the International Finance Corpo- women’s project. The experiment randomly ration to study corruption in the allocation assigned villages to make the final decision of driver’s licenses in India; Blattman, Fiala, regarding the projects either through a meet- and Martinez (2008) study the reintegra- ing or through a plebiscite. Olken examined tion of ex-combatants in northern Uganda; the impact of meetings and plebiscites on elite Collier and Vicente (2008) test the effec- capture along two main dimensions. First, he tiveness of an antiviolence intervention in examined whether the types of projects cho- Nigeria; Moehler (2008) investigates the sen moved closer to the preferences of villages role of private media in the strengthening elites. Second, he tested whether the location of accountability; Levy Paluck and Green of projects moved toward wealthier parts of (2009) examine how media broadcasts affect the villages. interethnic relations in a postconflict con- The experiment’s findings paint a mixed text; and Fearon, Humphreys, and Weinstein picture. Whether there was a meeting or a (2009) collaborate with the International Res- plebiscite had little impact on the general cue Committee to evaluate the impact of project, but the plebiscite did change the loca- a community-driven reconstruction program 1 tion of the women’s project to the poorer in Liberia. These studies were made possible areas of a village. The type of project chosen in large part through collaboration with local by women, however, was closer to the stated and international NGOs. preferences of the village elites than to poor Interventions led by NGOs can shed villagers’ preferences. Olken (2010) explains much light on social phenomena in con- that because the experiment left agenda set- texts where the involvement of independent ting unchanged, the elite’s influence over actors comes naturally, such as in the experi- decision making regarding the type of project ments described previously. There are cases, remained unchallenged. The experiment thus however, where one must give special consid- confirms previous arguments on the rele- eration to the effect that an NGO’s involve- vance of political mechanisms to aggregate ment may itself have on the social phenomena preferences. At the same time, it shows the at hand. Ravallion (2008) writes, “The very resilience of political inequalities. nature of the intervention may change when The persuasiveness of the results comes it is implemented by a government rather from the research design, which guaranteed than an NGO. This may happen because that plebiscites and meetings were allocated of unavoidable differences in (inter alia) the to villages, regardless of their social and polit- 1 For two excellent summaries of these studies, see ical configuration or any other observed or Humphreys and Weinstein (2009) and Moehler unobserved characteristic. Therefore, differ- (2010). Experimental Research on Democracy and Development 387 quality of supervision, the incentives facing public policy appeals produce mixed results. service providers, and administrative capac- Beyond confirming these arguments, the ity” (17). Moreover, there are social contexts Benin experiment presents a wide range of where an NGO’s involvement is not eas- new results that are counterintuitive and ily justified. In such cases, researchers have could not likely have been derived from any two options. First, they can undertake the other form of empirical research because enterprise of forging alliances with the rel- in Benin we almost never observe a can- evant actors, such as government officials or didate campaigning on public policy. For politicians, required to randomize an inter- instance, the experiment shows that 1) clien- vention of substantive interest. Second, they telist appeals reinforce ethnic voting (not the can take advantage of the growing number of other way around); 2) voters’ preference for cases where natural experiments are already clientelist or public goods messages depends in place due to policymakers’ decisions to ran- largely on political factors, such as incum- domize an intervention of interest. bency, and on demographic factors, such as gender; and 3) the lack of support for programmatic platforms is not due to opposing 2. Field Experiments in preferences among groups, level of education, Collaboration with Politicians or poverty, but instead to the fact that public policy platforms lack credibility, presumably Wantchekon’s (2003) study of clientelism because they tend to be vague. and its electoral effectiveness in Benin is an In a follow-up experiment implemented example of a unique collaboration between in the context of the 2006 presidential elec- researchers and politicians to implement a tions, Wantchekon (2009) finds that broad- treatment. He worked directly with presi- based platforms can be effective in generating dential candidates to embed a field experi- electoral support when they are specific and ment in the context of the first round of the communicated to voters through town hall March 2001 presidential elections. Together meetings. As a result of these experiments, with the candidates, Wantchekon randomly discussions of how to promote broad-based selected villages to be exposed to purely clien- electoral politics in Benin now have an empir- telist or purely public policy platforms. ical basis. Prior to this study, scholars had given little attention to the effectiveness of clientelist and programmatic mobilization strategies. Stokes 3. Natural Experiments (2007) notes that “most students and casual observers of clientelism assume that it works Although experiments like Wantchekon’s as an electoral strategy – that, all else equal, (2003) are still rare, scattered throughout a party that disburses clientelist benefits will the literature on development are examples win more votes than it would have had it of randomized interventions where assign- not pursued this strategy. In general we do ment of treatment is outside researchers’ not expect parties to pursue strategies that control. Chattopadhyay and Duflo’s (2004) are ineffective. And yet we have some theo- study of the quota system for women’s polit- retical reasons for believing that conditions ical participation and the provision of public are not always ripe for clientelism” (622). goods in India is such an example. The nat- The challenge of estimating the effectiveness ural experiment was facilitated by the 73rd of clientelism, patronage, and pork barrel as Amendment, which required that one third mobilization strategies rests in the possibility of Village Council head positions be ran- that electoral performance can shape spend- domly reserved for women. Chattopadhyay ing decisions (Stokes 2007). and Duflo’s evidence confirms that correcting The Benin experiment empirically vali- unequal access to positions of representation dates the argument that clientelist appeals leads to a decrease in unequal access to public are a winning electoral strategy, whereas goods. To begin with, the quota system was 388 AnaL.DeLaOandLeonardWantchekon effective. In the two districts studied (West audited before and after the 2004 municipal Benagal and Rajasthan), all positions of chief elections. in local village councils (Gram Panchayats, Ferraz and Finan (2008) find that, con- henceforth GPs) reserved for women were, ditional on the level of corruption exposed in fact, occupied by females. In turn, having by the audit, incumbents audited before the a woman chief increased the involvement of election did worse than incumbents audited women in GPs’ affairs in West Bengal, but after the election. Furthermore, in those had no effect on women’s participation in municipalities with local radio stations, the GPs in Rajasthan. Moreover, the increase in effect of disclosing corruption on the incum- women’s nominal representation translated bent’s likelihood of reelection was more into substantive representation. severe. This finding is in line with previous The study of the quota system shows that contributions that show how access to infor- women invest more in goods that are relevant mation affects the responsiveness of govern- to the needs of local women: water and roads ments. Moreover, it also corroborates find- in West Bengal and water in Rajasthan. Con- ings that the media is important to diffuse versely, they invest less in goods that are not information and discipline incumbents for as relevant to the needs of women: nonformal poor performance (Besley and Burgess 2002; education centers in West Bengal and roads Stromberg 2004). in Rajasthan. The evidence from this study De La O’s (2008) study of the electoral confirms that some classic claims of repre- effects of the Mexican conditional cash trans- sentative democracy, such as the relevance fer program (Progresa) is a third example of rules and the identity of representatives, of the use of a natural experiment. Finding hold true. Subsequent studies, however, show the electoral effectiveness of programmatic that despite institutional innovations, polit- spending presents similar challenges to the ical inequalities and prejudice continue to ones previously discussed. To evaluate the bias the representation system against minor- causal effect of spending, one needs to find ity and disadvantaged groups. In particular, exogenous variation on it. De La O empir- once the GPs’ chief position was no longer ically examines whether Progresa influenced reserved for women, none of the chief women recipients’ voting behavior by taking advan- was reelected, even though villages reserved tage of the fact that the first rounds of the for women leaders have more public goods program included a randomized component. and the measured quality of these goods is at Five hundred and five villages were enrolled least as high as in nonreserved villages (Duflo in the program twenty-one and six months and Topalova 2004). before the 2000 presidential election. De La In Latin America, Ferraz and Finan (2008) O finds that the longer exposure to program make use of a natural experiment to study the benefits increased turnout and increased the effects of the disclosure of local government incumbent’s vote share in 2000. corruption practices on incumbents’ electoral outcomes in Brazil’s municipal elections. The research design takes advantage of the fact 4. Lab and Lab-in-the-Field that Brazil had initiated an anticorruption Experiments program whereby the federal government began to randomly select municipal govern- Research opportunities such as the ones ments to be audited for their use of federal described in previous sections are becoming funds. To promote transparency, the out- more common as governments, NGOs, and comes of these audits were then disseminated sponsors around the world are giving priority publicly to the municipality, federal prosecu- to the systematic evaluation of interventions. tors, and the general media. Ferraz and Finan There are, however, other questions central compare the electoral outcomes of mayors to the field of political economy of devel- eligible for reelection between municipalities opment that require a deeper understanding Experimental Research on Democracy and Development 389 of the microfoundations of social processes. 5. Challenges for Experiments For example, what determines preferences Internal Validity over redistribution? Why do some individuals behave in a self-interested way while others The advantage of experiments compared to seem to be altruistic? Why do some commu- observational research is that random assign- nities prefer private over public goods? Why ment ensures that, in expectation, the treat- is inequality tolerated more in some places ment groups have the same observable and than others? What triggers reciprocity? unobservable baseline characteristics. As the Political scientists have found experimen- editors of this volume note in the intro- tation in the laboratory useful to study duction, however, random assignment alone these and many other questions. The lab- does not guarantee that the experimental out- oratory gives researchers complete control come will speak convincingly to the theoret- over assignment to treatment, the treatment ical question at hand. The interpretation of itself, and – perhaps most alluring – control the experimental result is a matter of internal over the setting where subjects are exposed validity – whether the treatment of interest to the treatment. The price that researchers was, in fact, responsible for changing out- pay for the internal validity of experimental comes. For example, in a pioneering field results produced in a laboratory is a well- experiment, Levy Paluck and Green (2009) known critique about external validity. Con- seek to gauge the causal effect of listening to cerns about generalizability, however, are a radio program aimed at discouraging blind not a dismissal of laboratory experiments. obedience and reliance on direction from Rather, they are an opportunity for creative authorities, and at promoting independent researchers (Camerer 2003). Indeed, recent thought and collective action in problem solv- studies have shown that lab-based experimen- ing in postgenocide Rwanda. Research assis- tation does not need to be confined to univer- tants played the radio program on a portable sities. stereo for the listener groups. The challenge Habyarimana et al. (2007), for example, of this experimental design in terms of inter- take the experimental laboratory to Uganda nal validity is that listener groups often lin- to study the mechanisms that link high lev- gered to chat after the radio program finished. els of ethnic diversity to low levels of public Therefore, the effect of the radio program goods provision. In this study, subjects are could be conflated with the effect of social- naturally exposed to high ethnic diversity on ization. Levy Paluck and Green successfully a daily basis. Thus, the conclusions drawn dealt with this challenge by recording on stan- from the dictator, puzzle, network, and pub- dardized observation sheets the lengths and lic goods games played by Ugandan subjects subjects of discussions during and after the speak directly to the social phenomenon of program. With this information, they could interest. test whether groups exposed to a particu- The games in Uganda show that labora- lar radio program socialized more than other tory experimentation enables researchers to groups. adjudicate among complex mechanisms that The interpretation of experimental results in less controlled settings would be con- also depends on what the control group founded. For example, Habyarimana et al. receives as treatment. In the experiment in (2007) find that ethnic diversity leads to Rwanda, for example, the control group lis- lower provision of public goods, not because tened to an educational entertainment radio coethnics have similar tastes or are more soap opera, which aimed to change beliefs altruistic, but because people from different and behaviors related to reproductive health ethnic groups are less linked in social net- and HIV. The average treatment effect is works. Therefore, the threat of social sanc- therefore the relative influence of the dif- tion for people who do not cooperate is less ferent content of the radio programs. This credible. comparison is different from a comparison 390 AnaL.DeLaOandLeonardWantchekon between those who listen to a radio program treatment. In other applications, however, the and those who do not listen to anything at degree to which an experiment can shed light all. A comparison between a group of lis- onto a theoretical question will depend on teners and a control group, however, would how the individual components of bundled be problematic in terms of internal validity treatments map onto theoretically relevant because the treatment group would not only variables. be exposed to radio program content, but also The second challenge faced by experimen- to group meetings, interactions with research tal researchers is that logistical difficulties of assistants, and so on. working in the field often compromise com- More generally, researchers in political pliance with research protocols. One form of economy of development face three chal- noncompliance occurs when those assigned lenges. First, because of the nature of the sub- to the treatment group do not receive the ject, researchers in development and democ- treatment. In this case, the randomly assigned racy need to forge alliances with relevant groups remain comparable, but the differ- decision makers to study social phenomena. ence in average outcomes does not measure These alliances make research not only more the average treatment effect. For example, realistic, but also more challenging. Policy- De La O et al. (2010) design an informa- makers, both in government and NGOs, are tional campaign in Mexico where households interested in maximizing the effect of a spe- in randomly selected polling precincts receive cific intervention, and it is natural for them a flyer with information about their munic- to endorse treatments that consist of a bun- ipal government’s use of a federal transfer dle of interventions. For example, Green et al. scheme aimed at improving the provision of (2010), in partnership with the Sarathi Devel- public services. Complying with the research opment Foundation, implemented a field protocol was more challenging in some of experiment in India during the 2007 elec- the experimental sites than in others because tion to examine how voters in rural areas some of the polling precincts were more would respond to messages urging them not isolated. Naturally, easy-to-access precincts to vote on caste lines but to vote for devel- are different from harder-to-access precincts; opment. The treatment consisted of puppet that is, they are more urban and wealthier shows and posters. This bundle of interven- than the other precincts. These sociodemo- tions is attractive from the NGO perspective, graphic differences are directly correlated to but it is challenging for researchers who want partisanship. Thus, in this example, noncom- to estimate the average treatment effect of an pliance in the form of failure to treat could educational campaign. greatly compromise the experimental design. To make the challenge more explicit, De La O et al. circumvent the problem of assume that in the example of Green et al.’s noncompliance by including several mech- (2010) Indian field experiment, compliance anisms of supervision in the distribution of with the research protocol was perfect. If the flyers, including the use of GPS receivers and effects of posters and puppet shows are inde- unannounced audits. pendent from each other, then the effect of An alternative form of noncompliance the bundled intervention is equal to the sum occurs when a treatment intended for one of the effects of the individual components of unit inadvertently treats a unit in another the intervention. In contrast, if the effects of group. The risk of spillover effects is preva- posters and puppet shows are not indepen- lent in the study of politics of development. dent, then there are four possibilities: posters In the Rwanda experiment, for example, might magnify the effect of puppet shows the radio program was also being nation- and vice versa, or, alternatively, posters might ally broadcasted, so listeners in both treat- cancel out the effect of puppet shows (and ment groups could listen to the program inde- vice versa). In this particular application, it pendent of the study. To minimize spillover might not be theoretically relevant to iso- effects, Levy Paluck and Green (2009)use late the effects of the two components of the strategies such as offering to give participants Experimental Research on Democracy and Development 391 in both groups the cassettes containing the had been affected by war because they either radio program that they were not supposed to experienced violence or were displaced. listen to at the end of the study. An alternative Second, the context of an experiment strategy to deal with problems generated by must resemble the context of the social spillover is for researchers to choose a unit of phenomenon of interest. For example, in analysis that enables them to estimate overall the experiment in Mexico, De La O et al. treatment effects. For example, Miguel and (2010) distribute to households the informa- Kremer (2004) design a field experiment in tion about municipal spending of the infras- Kenya where deworming drugs are randomly tructure fund close to the election day. An phased into schools, rather than provided to alternative design would be to recruit indi- individuals. With this design, they can take viduals for a study where similar information into account the fact that medical treatment would be distributed in informational meet- at the individual level has positive externali- ings directed by the researchers. This design, ties for nontreated individuals in the form of however, comes less naturally than that of reduced disease transmission.2 flyer distribution – a widely used communication technique in developing countries. Third, researchers must find creative ways External Validity to design treatments that resemble the vari- Field experiments are valuable tools for the ables of interest in the real world. In this study of development and democracy, but sense, not only the treatment, but also the designing and executing an experiment that scale of a field experiment must be taken speaks convincingly to theoretical questions into account when thinking about exter- of interest to the field presents some chal- nal validity. Consider the recent trend in lenges, in addition to the ones discussed in the the field where researchers collaborate with previous section. Just like field researchers, policymakers to evaluate an intervention in experimental researchers face a trade-off its pilot phase. Within these partnerships, between the depth of knowledge that comes policymakers welcome researchers’ interven- from studying a particular population and tions in small-scale versions of larger pol- the generalizability of their findings (Wood icy projects. Yet, as Deaton (2009) explains, 2007). “small scale projects may operate substan- To address challenges to external valid- tially different than their large scale ver- ity, researchers must design their experiments sion. A project that involves a few villagers with four things in mind. First, it is often or a few villages may not attract the atten- the case that researchers need to exert great tion of corrupt public officials because it effort to include in a study the subset of the is not worth their while to undermine or population worth studying, rather than the exploit them, yet they would do so as soon subset of the population that is most read- as any attempt were made to scale up. So that ily available to participate in a randomized there is no guarantee that the policy tested trial. For example, Habyarimana et al. (2007) by the randomized controlled trial will have recruit their subjects from an area in Uganda the same effects as in the trial, even on the characterized by high levels of ethnic diver- subjects included in the trial” (42). Finally, sity and low levels of public goods provision. researchers must find ways to measure out- In the Rwandan experiment, Levy Paluck and comes that resemble the actual outcomes of Green (2009) include two genocide survivor theoretical interest. Indeed, experiments have communities and two prisons in their four- in some cases started to revolutionize the teen experimental sites. Fearon et al.’s (2009) field by presenting alternative measures of study includes communities in postconflict key concepts, such as corruption and vote Liberia where the majority of the population buying. Consider Olken’s (2007) field experiment in 608 Indonesian villages where treat- 2 For more details on Miguel and Kremer’s (2004) ments were designed to test the effective- experiment, see Nickerson’s chapter in this volume. ness of top-down and bottom-up monitoring 392 AnaL.DeLaOandLeonardWantchekon mechanisms to reduce corruption. Unlike Moving forward, researchers will be con- much of the empirical work that measures fronted with the challenge of designing field corruption based on perceptions, Olken mea- experiments in a way that enables the accu- sured corruption more directly, by comparing mulation of knowledge. According to Mar- two measures of the same quantity, one before tel Garcia and Wantchekon (2010), there are and one after corruption. With this innova- two ways to achieve this goal. One option is to tive measure, Olken found that bottom-up replicate as much as possible the relationship interventions were successful in raising par- between two variables under different condi- ticipation levels. However, when compared tions (the robustness approach). The ongo- to the top-down intervention, the bottom-up ing research on the role of information in interventions proved to be less successful at community development projects illustrates reducing corruption. this approach. Banerjee et al. (2010) find Nickerson et al. (2010) present another that in India a randomly assigned informa- example where a field experiment innovation campaign was not effective at fostering tively measures a critical concept on the field. community involvement in Village Education Numerous qualitative studies of vote buying Committees and, ultimately, had no impact have concluded that the exchange of votes for on teacher effort or student learning out- gifts or cash is a prevalent practice around the comes. In contrast, a similar study in Uganda world. Yet, studies based on survey research reveals that, as a result of an informational have consistently found surprisingly little evi- campaign, people became more engaged in dence of vote buying. Nickerson et al. mea- community-based organizations and began to sured the frequency of vote buying in the monitor the health units more extensively. 2008 Nicaraguan municipal elections using This community-based monitoring increased a survey-based list experiment. All respon- the quality and quantity of primary health dents were asked how many activities from a care provision (Bjorkman and Svensson list were carried out by candidates and party 2007). operatives during the elections. The con- The examples provided in this section trol group was given a list of four activities, show that, even in cases where similar experi- including typical campaign activities such as ments are executed across two different popu- hanging posters, visiting homes, and placing lations, contextual differences could cause the advertisements in the media, as well as not same intervention to have different effects. An so typical activities, such as making threats. alternative to replicating similar treatments The treatment group was given the same list in different contexts is to use an analytic of activities, with the addition of vote buying. approach that makes the theoretical founda- Because respondents were not asked which tions of an experiment more explicit (Martel of the activities they witnessed but rather Garcia and Wantchekon 2010). This analytic how many, a certain degree of anonymity approach brings front and center the mech- when reporting vote buying was guaranteed. anisms that link a causal variable to an out- The proportion of respondents receiving a come. By being explicit about mechanisms, gift or favor in exchange for their vote was researchers can develop trajectories of exper- then measured as the difference in responses iments that are suitable to test theoretically between the treatment and the control group. informed hypotheses. Based on the list experiment, the authors esti- Consider, for example, the Benin electoral mated that nearly one fourth of respondents experiments (Wantchekon 2003, 2009). One received a gift or favor in exchange for their of the findings of the 2001 experiment is that vote. In contrast, fewer than three percent of voters are more likely to react positively to respondents reported that they had received a public goods message when it comes from a gift or favor when asked directly.3 a coethnic candidate. A possible explanation for this finding is that voters trust a candi- 3 For more details on the origins of the list experiment, date from their ethnic group more than they see Sniderman’s chapter in this volume. trust a candidate from another group. This Experimental Research on Democracy and Development 393 means that the mediating variable between ical than the standard opportunistic tailor- ethnic ties and votes is trust, or the credi- ing of messages to what voters want to bility of the candidate. By testing the rela- hear. tionship between credibility of candidates and A similar concern is raised by experimen- voting behavior in a follow-up experiment tal designs where subjects in one group are in 2006, Wantchekon (2009) improves the denied a treatment that they would ordinar- external validity of the results of the 2001 ily seek. For example, a study undertaken to experiment. As the Benin electoral experi- examine the effect of international aid, where ments illustrate, to make scientific progress some villages are randomly selected to receive in this field, new experimental designs should aid and some equally needy villages are ran- not only take into consideration the context domly selected to be denied aid, is bound of current experiments, but should also focus to raise ethical questions. Practical consid- on testing various aspects of a theory in a erations, however, can help researchers mit- coherent way. igate these concerns. For example, in most cases, NGOs and governments have limited budgets that force them to make decisions On the Ethics of Getting Involved regarding where to start an educational cam- in Elections paign, a social policy, or any other interven- One of the most striking features of exper- tion of interest. Random assignment in these iments on democracy is that they require cases provides policymakers with a trans- researchers to work directly with policymak- parent and fair way to decide the order in ers, politicians, or government officials. In which subjects are, for example, enrolled in a some cases, they also require researchers to program. get involved with running elections, gov- An ongoing field experiment in Uganda ernment programs, or education campaigns. illustrates this empirical strategy. Annan et al. Embedding experiments in the context of real (2010), in collaboration with Innovations for elections and programs brings a great degree Poverty Action and the Association of Vol- of realism to the treatments. However, what unteers in International Service, are evaluat- is gained in terms of the external validity of ing the Women’s Income Generating Sup- the experimental results may not sufficiently port program, which provides women with offset ethical concerns. grants and business training. To find whether We are far from having a consensus on small grants empower women and shape their where to draw the line between interven- political participation, Annan et al. will enroll tions that are ethical and interventions that women to the program in different phases are not. Nevertheless, there are several guide- over three years. The order of enrollment lines that researchers can follow when design- is randomly assigned. This design enables ing an experiment. First, an intervention will causal inferences, but no vulnerable house- raise fewer ethical concerns if the units under hold contacted by researchers will be left out study are exposed to a treatment they would of the program. ordinarily seek. In the Benin experiments, for A second way to think about ethical issues example, the clientelist treatment could at is to ask: what are the costs to subjects of first glance be a source of concern. Candidates participating in an experiment? In the Benin in Benin, however, typically run campaigns examples, if there were a cost to voters for based on clientelist appeals, regardless of being exposed to clientelist messages, then researchers’ presence. In such experiments, this cost is already routinely incurred in all the researcher was merely acting as an unpaid elections. In fact, the whole purpose of the campaign advisor to the candidate or civic experiment was to lower the cost of this cam- educator. The researcher’s main contribu- paign strategy for voters in future elections. tion was to suggest random assignment of More generally, experimental designs must campaign messages to districts. If anything, take into account the costs of exposing sub- random assignment of messages is more eth- jects to treatments, including, but not limited 394 AnaL.DeLaOandLeonardWantchekon to, material costs (e.g., the opportunity costs paign, which was sponsored by the National of spending time in the study), psychological Electoral Commission. costs, and even physical costs. A third set of ethical issues that researchers must take into account is the degree to 6. Concluding Remarks which interventions alter the outcomes and the costs associated with such departures. The rise of experiments as one of the most For example, in the experimental study prominent empirical strategies has led to new of elections, one common concern is that advances in the study of democracy and devel- researchers change the result of an election. opment. So far, some experimental results A 2002 New York Times article comment- have confirmed previous arguments, such as ing on the 2001 Benin experiment stated: the effectiveness of clientelism as a mobi- “There are some major ethical concerns with lization strategy and the prevalence of polit- field experiments in that they can affect elec- ical and social inequalities despite institution results and bring up important consid- tional innovations. Other experiments have erations of informed consent.”4 Wantchekon revealed relationships that only a random- (2003), however, suppressed this possibility ized control trial could uncover, like the fact by including in the experiment only safe dis- that clientelist appeals reinforce ethnic voting tricts, where candidates collaborating in the and not the other way around. Finally, some study had a stronghold. experiments are revolutionizing the measure- In this particular example, the subset of ment of core concepts in the field. For exam- districts where ethical concerns are manage- ple, we now know that vote buying measured able coincided with the subset of districts that experimentally is more prevalent than what were theoretically relevant to study because observational studies suggested. clientelism is more resilient in districts where Going forward, field experiments in col- one political machine has a monopoly than in laboration with policymakers, governments, districts where there is more political com- and NGOs are a promising line of research. petition. In other applications, restricting the The next round of experiments, however, experiment to certain subpopulations where faces considerable challenges, including those ethical concerns are manageable may com- we highlight throughout this chapter. First, promise the external validity of the experi- researchers must find creative ways to design ment’s results. interventions that are attractive to poten- Finally, many research questions in the tial partners but that still speak convinc- political economy of development, like the ingly to theoretically relevant questions. In effect of violence on development, involve doing so, researchers must pay special atten- interventions that are difficult to study tion to internal validity issues. Second, a through experimentation without raising eth- more analytic approach would help guide ical concerns. Creative experimental designs, researchers to design experiments that enable however, can enable researchers to study significant accumulation of knowledge to take social phenomena that at first glance seem out place. Finally, as the scope of experimentation of reach. For example, Vicente (2007) con- expands, the trade-off between external valid- ducted a field experiment in Sao˜ Tome´ and ity and ethical concerns will become more Prıncipe´ to study vote buying. As in many salient. other countries, buying votes is illegal in Sao˜ Despite these challenges, experimental Tome.´ Thus, Vicente randomly assigned sub- research on development and democracy is a jects to be exposed to an antivote buying cam- productive and exciting endeavor. As insightful as the experimental research has been up until now, numerous substantive questions 4 Lynnley Browning. 2002. “Professors Offer a Real- remain unanswered. Hopefully, the selection ity Check for Politicians.” New York Times, August 31. Retrieved from www.nytimes.com/2002/08/31/ of studies covered in this chapter illustrates arts/31FIEL.html (November 15, 2010). how experiments can be used as a research Experimental Research on Democracy and Development 395 tool to study broader and more central ques- De La O, Ana L. 2008. “Do Conditional Cash tions about the relationship between democ- Transfers Affect Electoral Behavior? Evidence racy and development. from a Randomized Experiment in Mexico.” Unpublished manuscript, Yale University. De La O, Ana L., Alberto Chong, Dean Karlan, References and Leonard Wantchekon. 2010. “Information Dissemination and Local Governments’ Elec- Annan, Jeannie, Christopher Blattman, Eric toral Returns, Evidence from a Field Experi- Green, and Julian Jamison. 2010. “Uganda: ment in Mexico.” Paper presented at the con- Enterprises for Ultra-Poor Women after ference on Redistribution, Public Goods and War.” Unpublished manuscript, Yale Univer- Political Market Failures, Yale University, New sity. Retrieved from http://chrisblattman.com/ Haven, CT. projects/wings/ (November 15, 2010). Deaton, Angus. 2009. “Instruments of Develop- Banerjee, Abhijit, Rukmini Banerji, Esther Duflo, ment: Randomization in the Tropics, and Rachel Glennerster, and Stuti Khemani. 2010. the Search for the Elusive Keys to Economic “Pitfalls of Participatory Programs: Evidence Development.” Unpublished manuscript, from a Randomized Evaluation in Education Princeton University. in India.” American Economic Journal: Economic Duflo, Esther, and Petia Topalova. 2004. “Unap- Policy 2: 1–30. preciated Service: Performance, Perceptions, Bertrand, Marianne, Simeon Djankov, Rema and Women Leaders in India.” Working paper, Hanna, and Sendhil Mullainathan. 2007. Massachusetts Institute of Technology. “Obtaining a Driver’s License in India: An Fearon, James D., Macarthan Humphreys, and Experimental Approach to Studying Corrup- Jeremy Weinstein. 2009. “Can Development tion.” Quarterly Journal of Economics 122: 1639– Aid Contribute to Social Cohesion after Civil 76. War? Evidence from a Field Experiment Besley, Timothy, and Robin Burgess. 2002.“The in Post-Conflict Liberia.” American Economic Political Economy of Government Responsive- Review: Papers and Proceedings 99: 287–91. ness: Theory and Evidence from India.” Quar- Ferraz, Claudio,´ and Frederico Finan. 2008. terly Journal of Economics 117: 1415–52. “Exposing Corrupt Politicians: The Effect of Bjorkman, Martina, and Jakob Svensson. 2007. Brazil’s Publicly Released Audits on Electoral “Power to the People: Evidence from a Ran- Outcomes.” Quarterly Journal of Economics 123: domized Field Experiment of a Community- 703–45. Based Monitoring Project in Uganda.” World Green, Jennifer, Abhijit Banerjee, Donald Green, Bank Policy Research Working Paper No. and Rohini Pande. 2010. “Political Mobiliza- 4268. tion in Rural India: Three Randomized Field Blattman, Christopher, Nathan Fiala, and S. Experiments.” Paper presented at the annual Martinez. 2008. “Post-Conflict Youth Liveli- meeting of the Midwest Political Science Asso- hoods: An Experimental Impact Evaluation ciation, Chicago. of the Northern Uganda Social Action Fund Habyarimana, James, Macarthan Humphreys, (NUSAF).” Monograph. Washington, DC: Daniel N. Posner, and Jeremy Weinstein. 2007. World Bank. “Why Does Ethnic Diversity Undermine Pub- Boix, Carles, and Susan Stokes. 2003. “Endoge- lic Goods Provision?” American Political Science nous Democratization.” World Politics 55: 517– Review 101: 709–25. 49. Humphreys, Macartan, William A. Masters, and Camerer, Colin F. 2003. Behavioral Game Theory: Martin E. Sandbu. 2006. “The Role of Leaders Experiments in Strategic Interaction. Princeton, in Democratic Deliberations: Results from a NJ: Princeton University Press. Field Experiment in Sao˜ Tome´ and Principe.” Chattopadhyay, Raghabendra, and Esther Duflo. World Politics 58: 583–622. 2004. “Women as Policymakers: Evidence Humphreys, Macartan, and Jeremy Weinstein. from a Randomized Policy Experiment in 2009. “Field Experiments and the Political India.” Econometrica 72: 1409–43. Economy of Development.” Annual Review of Collier, Paul, and Pedro C. Vicente. 2008.“Votes Political Science 12: 367–78. and Violence: Experimental Evidence from a Keefer, Philip. 2007. “The Poor Performance of Nigerian Election.” Households in Conflict Poor Democracies.” In The Oxford Handbook of Network (HiCN) Working Paper No. 50. Comparative Politics, eds. Carles Boix and Susan 396 AnaL.DeLaOandLeonardWantchekon

Stokes. New York: Oxford University Press, Experiment in Indonesia.” American Political 886–909. Science Review 104: 243–67. Levy Paluck, Elizabeth, and Donald P. Green. Przeworski, Adam, and Fernando Limongi. 1997. 2009. “Deference, Dissent, and Dispute Res- “Modernization: Theories and Facts.” World olution: An Experimental Intervention Using Politics 49: 155–83. Mass Media to Change Norms and Behavior in Ravallion, Martin. 2008. “Evaluation in the Prac- Rwanda.” American Political Science Review 103: tice of Development.” Policy Research Work- 622–44. ing Paper No. 4547. Washington, DC: World Martel Garcia, Fernando, and Leonard Want- Bank. chekon. 2010. “Theory, External Validity, and Shepsle, K. 2006. “Old Questions and New Experimental Inference: Some Conjectures.” Answers about Institutions: The Riker Objec- Annals of the American Academy of Political and tion Revisited.” In The Oxford Handbook of Polit- Social Science 628: 132–47. ical Economy, eds. Barry R. Weingast and Don- Miguel, Edward, and Michael Kremer. 2004. ald A. Wittman. New York: Oxford University “Worms: Identifying Impacts on Education Press, 1031–49. and Health in the Presence of Treatment Stokes, Susan. 2007. “Political Clientelism.” In Externalities.” Econometrica 72: 159–217. The Oxford Handbook of Comparative Politics, Moehler, Devra C. 2008. “Tune in to Governance: eds. Carles Boix and Susan Stokes. New York: An Experimental Investigation of Radio Cam- Oxford University Press, 604–27. paigns in Africa.” Paper presented at the con- Stromberg, David. 2004. “Radio’s Impact on Pub- ference on Field Experiments in Comparative lic Spending.” Quarterly Journal of Economics Politics and Policy, University of Manchester, 119: 189–221. Manchester, UK. Vicente, Pedro C. 2007. “Is Vote Buying Effec- Moehler, Devra C. 2010. “Democracy, Gover- tive? Evidence from a Randomized Experiment nance, and Randomized Development Assis- in West Africa.” Economics Series Working tance.” Annals of the American Academy of Polit- Paper No. 318. Oxford: University of Oxford, ical and Social Science 628: 30–46. Department of Economics. Nickerson, David, Ezequiel Gonzalez-Ocantos, Wantchekon, Leonard. 2003. “Clientelism and Chad Kiewiet de Jonge, Carlos Melendez,´ and Voting Behavior: Evidence from a Field Exper- Javier Osorio. 2010. “Vote Buying and Social iment in Benin.” World Politics 55: 399– Desirability Bias: Experimental Evidence from 422. Nicaragua.” Unpublished manuscript, Univer- Wantchekon, Leonard. 2009. “Can Informed sity of Notre Dame. Public Deliberation Overcome Clientelism? Olken, Benjamin A. 2007. “Monitoring Corrup- Experimental Evidence from Benin.” Working tion: Evidence from a Field Experiment in paper, New York University. Indonesia.” Journal of Political Economy 115: Wood, Elizabeth. 2007. “Field Research.” In The 200–49. Oxford Handbook of Comparative Politics, eds. Olken, Benjamin A. 2010. “Direct Democracy and Carle Boix and Susan Stokes. New York: Local Public Goods, Evidence from a Field Oxford University Press, 123–46. Part VIII

ELITE BARGAINING

CHAPTER 28

Coalition Experiments

Daniel Diermeier

The literature on coalition experiments is literature has tested noncooperative models blessed by a close connection between theory of coalition government. Although there has and empirical work, using both experimen- been a substantial amount of research on solu- tal and field data.1 In political science, the tion concepts from cooperative game theory,2 main research goal has been to understand the its performance to explain experimental data formation of coalition governments in mul- has long been considered unsatisfactory, tiparty democracies. Although other aspects leading some researchers such as Gamson of coalitions could be considered (e.g., coali- (1964) to propose an “utter confusion” tion stability or the creation and maintenance theory. of military alliances), they have not been the In addition to testing of theoretical mod- focus of much existing research. We therefore els, experiments can also be used to supple- focus on experiments in coalition formation. ment field studies. For example, field stud- The goal of this chapter is not to pro- ies (Martin and Stevenson 2001; Diermeier, vide an accurate description of the complete Eraslan, and Merlo 2003) have indicated the history of coalition experiments, but rather importance of constitutional features on gov- to discuss some of the key questions that ernment formation, specifically the presence continue to occupy researchers today. For of an investiture vote. Yet, the equilibria pre- example, much of the recent experimental dicted by noncooperative models in general depend on institutional detail, such as the The author wants to thank Randy Stevenson, Jamie protocol in which offers can be made and Druckman, and Skip Lupia for their helpful comments. so on. However, there is no straightforward He is grateful to Alison Niederkorn and especially to match between (existing) theoretical mod- Justin Heinze for his research support, as well as the Ford Motor Company Center for Global Citizenship at els and field data. For example, the predic- the Kellogg School of Management for additional fund- tions of a model may depend on the protocol ing. All remaining errors and omissions are my own. 2 For overviews, see Gamson (1964) or Burhans (1973). 1 For overviews on the development of the field, see See also Fiorina and Plott (1978) for experiments in Laver and Schofield (1990) and Diermeier (2006). spatial settings.

399 400 Daniel Diermeier by which proposers of possible cabinets are sequential bargaining models under major- selected (so-called formateurs), but such pro- ity rule, specifically reflected in the Baron- tocols may be based on implicit conventions Ferejohn (1989) model. In all variants of the that are not easily inferred from publicly Baron-Ferejohn model, a proposer is selected available data. To address these issues, some according to a known rule and then proposes researchers (Diermeier et al. 2003; Dier- an alternative to a group of voters. Accord- meier, Eraslan, and Merlo 2007) have pro- ing to a known voting rule, the proposal is posed structural estimation techniques, but either accepted or rejected. If the proposal is these models are difficult to construct and still accepted, the game ends and all actors receive limited by existing field data. By using exper- payoffs as specified by the accepted proposal. iments, we can fill these gaps and directly Otherwise, another proposer is selected and study the consequences of institutional dif- so on.3 This process continues until a pro- ferences. Until very recently, the literature posal is accepted. on coalition experiments has largely followed Consider a simple version of the model this research agenda; its goal has been to test where there are three political parties that carefully theories of coalition formation, usu- need to decide how to split $1. Suppose that ally formal theories created in the language no party has a majority of seats. The Baron- of noncooperative game theory. The goal Ferejohn (1989) model predicts that the party then has been to implement these theories with proposal power will propose a mini- as faithfully as possible in a laboratory set- mal winning coalition consisting of itself and ting, following the methodology developed one other member, leaving the third party in experimental economics and game the- with zero. The proposing party will offer its ory (Roth 1995): experimental subjects inter- coalition partner just the amount necessary act anonymously via computer terminals, and to secure acceptance. This amount (or con- payments are based on performance with pay- tinuation value) equals the coalition partner’s ment schedules carefully constructed to avoid expected payoff if the proposal was rejected any contamination from factors such as risk and the bargaining continued. Proposals are preferences, desire to please experimenters, thus always accepted in the first round. Note and so on. Although this approach has been that the proposing party will always choose very fruitful and led to important insights, as its coalition partner the party with the recent research has also pointed to its lim- lowest continuation value. The division of itations. After reviewing the research that spoils will, in general, be highly unequal, has followed the experimental economics, we especially if the parties’ discount factors introduce a different research tradition that are low. introduced some methods from social psy- The Baron-Ferejohn (1989) model is chology into the study of coalition formation. attractive for the study of cabinets, as coali- This approach is emphatically context rich, tion bargaining usually has a strong dis- allowing face-to-face interaction with little tributive component (e.g., control of gov- restriction on communication or negotiation ernment portfolios) and provides predictions protocols. Of particular promise is the abil- even when the game lacks a core. Moreover, ity to study the role of language in coalition the model provides both point and compar- negotiations, which opens the possibility to ative statics predictions about the effects of study communication and framing strategies different institutions on bargaining behavior systematically. and outcomes such as proposer selection and amendment rules. This allows the modeler

1. Baron-Ferejohn Model 3 Baron and Ferejohn (1989) also consider open rules, where (nested) amendments to a proposal are permit- During the late 1980s, noncooperative game ted before the final vote. See also Baron (1989, 1991) for applications to coalition government. There are theory became the dominant paradigm in other variants of the Baron-Ferejohn model, but they the study of coalitions in the form of played no role in the study of coalition government. Coalition Experiments 401 to assess the effects of different constitutional where all individually rational outcomes can factors on coalition outcomes. be supported as equilibria. This means that one cannot test the Baron-Ferejohn model in isolation, but only in conjunction with an 2. Testing the Baron-Ferejohn Model additional equilibrium refinement (here, stationarity). This is important, as McKelvey Given its status as the canonical model of suggests in his discussion of the findings that legislative bargaining, the predictions of the one way of accounting for the discrepancy Baron-Ferejohn (1989) model were soon between experimental outcomes and theoret- tested in controlled laboratory experiments, ical predictions is that subjects may implicitly first by McKelvey (1991). McKelvey uses a try to coordinate on a nonstationary equi- three-voter, closed rule, finite alternative ver- librium. Third, the unique stationary equi- sion of the Baron-Ferejohn model, which librium involves randomization. This implies results in mixed strategy equilibria. McKelvey that the model is predicting a distribution finds that the unique stationary solution to his over outcomes, not a single outcome, which game at best modestly explains the data: pro- requires many observations to detect a sig- posers usually offer too much and (the lower) nificant difference between predicted and equilibrium proposals predicted by the theory observed frequencies. are rejected too frequently. Given the centrality of the Baron- The McKelvey (1991) experiments fol- Ferejohn (1989) model for formal theories lowed the methodological approach of exper- of coalition formation, the McKelvey (1991) imental game theory. Subjects interact results triggered subsequent experimental anonymously through a computer terminal work. The first paper in this direction is Dier- and are paid depending on performance. meier and Morton (2005), which follows the The goal of this approach is to induce the basic setup of the McKelvey experiment, but incentives and knowledge structure specified tries to resolve some of the methodologi- by the game-theoretic model in the labo- cal difficulties of testing the Baron-Ferejohn ratory setting. Yet, a faithful implementa- model. In contrast to McKelvey, they use tion of the Baron-Ferejohn (1989) model in a finite game under weighted majority rule the laboratory is challenging. These prob- where a fixed payoff is divided among three 4 lems are common in experimental game the- actors. This leads to a unique subgame- ory, and the solutions adopted by McKelvey perfect equilibrium without assuming addi- address the problems to a large extent. How- tional stationarity. Second, the subgame- ever, some residual concern should remain. perfect equilibrium involves no randomiza- First, although the Baron-Ferejohn model is tion on the equilibrium path. Third, the of (potentially) infinite duration, it must be weighted majority game allows us to test a implemented in finite time. Participants in a rich set of comparative statics, not just point typical experimental session will know, or at predictions. Comparative statics analyses are least reliably estimate, the maximal duration of particular interest in testing institutional of the game (the recruitment flyer will usu- models where the ability to predict correctly ally state the time they need to make them- behavioral changes in response to institu- selves available for the experiment). This may tional changes is critical. induce endgame effects, especially for later Diermeier and Morton (2005), however, rounds. Second, the model’s prediction is find little support for either point or com- based on the assumption of stationarity, a parative static predictions. First, proposers stronger equilibrium requirement than mere frequently allocate money to all players, not subgame perfection, which rules out depen- just to the members of the minimal winning dence of previous actions and thus eliminates coalition. Second, proposers do not seem to the use of punishment strategies familiar from the study of repeated games. Without it, the 4 Baron and Ferejohn (1989) discuss this case in a foot- Baron-Ferejohn model faces a folk theorem note. 402 Daniel Diermeier select the “cheapest” coalition partner, that is, stand the game’s complex incentives and have the one with the lowest continuation value. insufficient opportunity to learn. In this case, Third, proposers offer too much to their subjects may simply revert to an equal sharing coalition partners. Fourth, a significant per- heuristic and select coalition partners haphaz- centage of first-period proposals above the ardly. continuation value are rejected, sometimes Frechette´ et al. (2003) designed a sec- repeatedly. Diermeier and Morton’s data, ond experiment to address these cognitive however, do reveal some consistent behav- concerns by considering more rounds and ioral patterns. Proposers typically select a by adding a graduate student to the sub- subset of players (sometimes containing all ject pool that used an algorithm to imple- other players) and split the money equally ment the stationary subgame-perfect equilib- 5 among its members. Note that this eliminates rium strategy. In this experiment, proposal the proposer premium. Indeed, players take behavior more closely resembled the Baron- extreme measures to guarantee equal payoffs Ferejohn (1989) predictions: allocations are among the coalition, even “wasting” small less egalitarian, and equal split proposals amounts of money to guarantee an equal split. (among all players) completely vanish. Never- The Diermeier and Morton (2005) find- theless, play does not converge to the alloca- ings confirm and sharpen the McKelvey tion predicted by the Baron-Ferejohn model. (1991) predictions. However, in both cases, Rather, proposers and voters seem to rely on the main focus is on testing the model’s point a “fair” reference point of 1/n share of the 6 predictions. Frechette,´ Kagel, and Lehrer benefits. Offers that are less than the refer- (2003) take a different approach by inves- ence share are consistently rejected, whereas tigating the institutional predictions of the shares that are higher than 1/n are usu- Baron-Ferejohn (1989) model. They com- ally accepted.7 This focal point interpreta- pare open and closed rule versions of the tion may also account for the odd finding in Baron-Ferejohn model with five players. As the open rule case where subjects accepted in McKelvey, but in contrast to Diermeier an amount less than their continuation value, and Morton, play continues until agreement which happened to be significantly higher is reached and the focus is on stationary equi- than the fair reference point. libria. In contrast to previous experiments, The hypothesis that subjects are in part Frechette´ et al. find some qualitative support motivated by fairness concerns is highly confor the Baron-Ferejohn model. In particu- sistent with a related literature in experimen- lar, there are longer delays and more egal- tal economics on bilateral bargaining games, itarian distributions under the open rule, as with the ultimatum game as the best-known predicted. However, some less obvious (but example (Guth,¨ Schmittberger, and Schwarze critical) aspects of the Baron-Ferejohn model 1982). In the ultimatum game, one player are not well supported in the data. For example, in their design, proposers should propose 5 Note, however, that the fact that the presence of a minimal winning coalitions in both the open “selfish” player was announced may have changed and closed rule cases. However, only four per- the nature of the game. 6 The concept of a “fair share” is consistent with cent of proposals correspond to this predic- Bolton and Ockenfels’ (2000a,b) ERC (“equity, reci- tion. Even more troubling, under the open procity, and competition”) theory. Although the rule, subjects accept proposals that offer them ERC approach has been successful in explaining two-player bargaining behavior, recent experimental less than their continuation value. results with three-person games (Kagel and Wolfe A common concern with all experimental 2001; Bereby-Meyer and Niederle 2005) are incon- investigations of the Baron-Ferejohn (1989) sistent with the ERC approach. 7 An alternative approach has been suggested by model is that the specific games under con- Battaglini and Palfrey (2007). The results are not sideration are cognitively highly demand- directly comparable because they analyze a bargaining. Thus, one explanation for the Baron- ing protocol with an endogenous status quo. They also find a significant number of equal distributions, Ferejohn model’s lack of empirical fit may which can be explained by risk aversion in their be that subjects do not initially fully under- framework. Coalition Experiments 403 makes a proposal on the division of a fixed Ferejohn model is like the second player in an amount of money that the other player must ultimatum game. The relationship between either accept or reject, with rejection imply- the proposer and noncoalition member in the ing a zero payoff for both. In experiments last period, however, is also similar to the on ultimatum games, proposers should take dictator game because the votes of the non- (almost) all the money, yet the divisions are coalition members are not necessary to pass a far more equal than predicted. Moreover, if proposal. proposers offer less than a certain amount,8 Diermeier and Gailmard (2006) present an then the other player frequently rejects the approach to separate cognitive from motiva- offer (even if it is a significant amount of tional issues and directly focus on the ques- money) and receives a payoff of zero. Exper- tion of whether and how agents in majori- iments on bargaining games with a series tarian bargaining situations are driven by of alternating offers result in similar out- moral motivations such as fairness. To do comes. Proposers offer more money than sug- this, they use a much simpler version of the gested by their subgame-perfect strategy, and proposer-pivot game that directly resembles bargaining partners consistently reject offers the ultimatum game. Rather than having sub- and forgo higher payoffs (Guth¨ et al. 1982; jects calculate continuation values, players are Ochs and Roth 1989; Davis and Holt 1993; directly assigned an ex ante known disagree- Forsythe et al. 1994;Roth1995). ment value, which is a given amount of money Forsythe et al. (1994) investigated this they will receive if the proposal is rejected. hypothesis by comparing ultimatum and Proposers then make a “take it or leave it dictator games. The dictator game differs offer” on how to split a fixed, known amount from the ultimatum game in that the propos- of money among the players. Disagreement ing player proposes a division between the values are, in essence, a “reduced form” rep- two players and the other player cannot reject resentation of continuation values. Alterna- the proposal. In ultimatum games, almost tively, they can be interpreted as an ultima- sixty percent of the offers observed propose tum game with competing respondents. By an equal division of payoffs. Although there varying the disagreement values as treatment is still a significant percentage of equal divi- variables, competing motivational theories sions in dictator games (less than twenty per- can be tested. Recall that in a model with self- cent), the modal division is the subgame- interested agents, any proposer will select the perfect allocation where the proposer keeps “cheaper” of the other voters and offer that the entire payoff. This result suggests that player his or her disagreement value (perhaps although some of the subjects are primarily with a little security margin), whereas the motivated by egalitarian notions of fairness, other (more expensive) voter receives zero. the high percentage of equal divisions in ulti- Similarly, voters will accept only offers at or matum games cannot be attributed to a simple higher than their respective disagreement val- desire to be fair. ues. Note that this optimal behavior (by pro- The Baron-Ferejohn (1989) bargaining posers and voters) does not depend on the game is similar to these bargaining games in proposer’s disagreement value, other than in the sense that proposers are expected to offer the trivial case where the value is so high that their coalition partner his or her continuation the proposer prefers his or her disagreement value. In the last period of a finite game, this value to any possible proposal. Thus, varying continuation value is zero, as in the ultima- the proposer’s reservation value should not tum game. Hence, the nonproposing coali- have any influence on proposing or voting tion partner in the last period of the Baron- behavior. That is, the tested theory not only makes certain point and comparative statics 8 This amount varies from culture to culture. In exper- predictions, but it also mandates that certain 1991 iments conducted by Roth et al. ( ), the modal aspects of the game should not matter.Ifthey offer varied between forty percent of the payoff in Jerusalem, Israel, and fifty percent in Pittsburgh, do, then the theory simply cannot completely Pennsylvania. account for the findings. 404 Daniel Diermeier

The results of Diermeier and Gailmard tocoalition bargaining (Diermeier and Merlo (2006) are at odds not only with the Baron- 2000; Baron and Diermeier 2001). In a series Ferejohn (1989) model, but also with any of papers, Frechette,´ Kagel, and Morelli of the proposed fairness models (Fehr and (2005a, 2005b, 2005c) test Gamson’s law Schmidt 1999; Bolton and Ockenfels 2000a). (interpreted as a proportionality heuristic) in The key feature is the dependency on the a laboratory setting and compare it to predic- reservation value of the proposer, which should tions from demand bargaining and the Baron- have no strategic impact whether players are Ferejohn (1989) model.10 In demand bargain- selfish or incorporate a fairness consideration ing, players do not make sequential offers in their behavior. But this is not the case. Vot- (as in the Baron-Ferejohn model), but make ers accept lower offers if the proposer has a sequential demands – that is, compensations higher reservation value and proposers pro- for their participation in a given coalition posed significantly more. Interestingly, voters until every member has made a demand or are also more tolerant of higher offers to the until a majority coalition forms. If no accept- other (nonproposing) voter if that voter has a able coalition emerges after all players have higher reservation value. Diermeier and Gail- made a demand, a new first demander is ran- mard interpret their findings as an entitle- domly selected; all the previous demands are ment effect. According to this interpretation, void, and the game proceeds until a com- experimental subjects interpret their exoge- patible set of demands is made by a major- nously given reservation value as an entitle- ity coalition. The order of play is randomly ment that ought to be respected.9 determined from among those who have not yet made a demand, with proportional recognition probabilities. 3. Alternative Bargaining Protocols A simple three-party case provides contrasting predictions generated by the three Much of the existing experimental work has models. Suppose we have three parties and concentrated on testing models of coalition no party has a majority of seats. With equal formation, such as the Baron-Ferejohn (1989) proposal power (and sufficiently high dis- model. Yet, as we discuss in the introduc- count factor), the equilibrium allocation in tion, some experimental work has focused on the Baron-Ferejohn (1989) model will give institutional analysis instead, with an empha- the proposer about two thirds of the pie, with sis on an examination of bargaining proto- one third given to one other party and the cols. One major influence has been Gamson’s third party receiving nothing. Demand bar- (1961) claim that portfolios among cabinet gaining, however, predicts a fifty–fifty split members will be allocated proportionally to between the coalition parties and nothing the parties’ seat shares. This hypothesis has for the out party. The Gamson predictions found so much support in the field studies that depend on the respective seat share. For it has been called “Gamson’s law” (Browne example, in the setting used by Frechette´ et al. and Franklin 1973; Browne and Fendreis (2005a), if one coalition member has nine 1980; Schofield and Laver 1985; Warwick votes and the larger forty-five, then the larger and Druckman 2001). Gamson’s law, how- party would receive about eighty-three per- ever, seems to be at odds with the proposer cent of the share, even if the smaller party is the models because it implies the absence of a proposer. proposer premium, a critical implication of In the laboratory, both the Baron- the Baron-Ferejohn framework. Ferejohn (1989) and demand bargaining, This discrepancy has led to the devel- 10 Asfarasweknow,therehavebeennodirecttestsof opment of alternative bargaining models, protocoalition bargaining as in the model proposed demand bargaining (Morelli 1999), and pro- by Baron and Diermeier (2001), although there have been tests using field data (Carrubba and Volden 2004). Diermeier and Morton (2005) also directly 9 The same effects can be found in bilateral bargaining test the proportionality heuristic in a laboratory set- games (Diermeier and Gailmard n.d.). ting and find no support. Coalition Experiments 405 however, outperform a proportionality simplified environments or when learning is heuristic in the laboratory. Consistent with possible. This suggests that the motivational the results reported in previous experiments, profile of experimental subjects needs to be however, the proposer premium is too small taken seriously, with moral considerations compared to the Baron-Ferejohn predictions, and framing playing an important role. That and voters reject offers that they consider to motivational profile, however, appears to be be too low, even if acceptance would be in complex. It appears to include selfish compo- their self-interest, confirming the previous nents, fairness concerns, and even respect for findings. The existence of a proposer pre- (arbitrary) entitlements. mium, however, even if it is too small com- The second main finding results from the pared to the model, creates a stark contrast study of alternative bargaining institutions. with the field research. Frechetteetal.(´ 2005a, These results point to a potential problem 2005b) provide an intriguing explanation that of using sequential bargaining models as the reconciles this apparent conflict. The idea formal framework to study coalition forma- is to apply the same regression approaches tion: they appear to be “too specific” for used in field studies to the experimental data. the task. Available methods for studying field The results show that the strong support of data cannot distinguish between competing proportionality is due to proportional pro- approaches. poser selection, as identified in Diermeier An important task for future work is thus 1) and Merlo (2000), not bargaining according to allow for a richer set of agent motivations to a proportionality heuristic once a pro- that are not captured by monetary incentives, poser has been selected. Interestingly, the and 2) to try to identify general features of same statistical approach is also unable to dis- sequential bargaining models that do hold tinguish between the Baron-Ferejohn model for various model specifications. One main and demand bargaining. In other words, the insight from sequential bargaining models regression approach used in field data can- is that any current agreement depends on not identify the underlying bargaining pro- the shared expectations of what would occur tocol. Both the Baron-Ferejohn model and if that agreement could be reached or sus- the demand bargaining model can explain tained (e.g., which future agreement would be the striking proportionality regularities in the formed). Such future agreements may be less field data, but existing field data methods favorable to current coalition partners due cannot distinguish between the competing to a shift in bargaining strength, but, most models. important, future coalitions may consist of In summary, experiments on coalitional different parties, relegating at least some of bargaining suggest that sequential bargain- the current coalition members to the much ing models offer a promising framework less desirable role of opposition party. Inter- to understand coalition formation, with the estingly, this “fear of being left out” not only Baron-Ferejohn (1989) bargaining protocol sustains current coalitions as equilibria, but still the leading contender. Yet, some of the it may also lead negotiating parties to accept predictions of the Baron-Ferejohn model are inefficient outcomes out of the fear that the consistently rejected in laboratory experi- current coalition will be replaced by a new ments. Although a proposer premium can one and that they may be left out of the final consistently be identified, it is too small com- deal. This was formally shown by Eraslan and pared to the Baron-Ferejohn prediction. Cor- Merlo (2002). respondingly, voters consistently reject offers These questions are difficult to answer that they consider to be unfair.11 Both find- in the experimental methodology com- ings are consistent with the large literature mon in behavioral game theory that has on ultimatum games and persist in much dominated existing laboratory research on coalitions. Rather, a more contextualized 11 Notice that once voters reject “unfair” offers, proposers may act optimally in offering more than pre- approach seems necessary. Interestingly, such scribed by the Baron-Ferejohn (1989) model. a tradition already exists, but it has been 406 Daniel Diermeier largely ignored by political scientists. It has Table 28.1: Potential Coalitions been developed almost exclusively by psy- and Their Respective Payoffs chologists under the name of “multiparty negotiations” and is almost exclusively exper- Coalition Payoff imental in nature. {A} 0 {B} 0 0 4. Context-Rich Experiments {C} {A,B} $118,000 {A,C} $84,000 One first insight from social psychology is {B,C} $50,000 that the complexity of multiparty negotia- {A,B,C} $121,000 tions may be overwhelming to research subjects, which may lead them to rely on simple heuristics such as equal sharing (Bazerman, Mannix, and Thompson 1988; Messick 1993; agreement, but at least some parties (e.g., A Bazerman et al. 2000). It is instructive to and B) can form a fairly profitable agreement reinterpret the Frechette´ et al. (2003), Dier- without including the third party (here, C). meier and Morton (2005), and Diermeier and So, one possible intuition of how the negoti- Gailmard (2006) results in this context. One ations may proceed is as follows. Parties A and possible interpretation of the Diermeier and B (or some other “protocoalition”) may form Morton findings is that subjects were over- a preliminary agreement on how to split the whelmed by the complexity of the negoti- pie already available to an AB coalition (here, ation task and fell back on an equal shar- 118,000) among themselves, and then only ing heuristic. Interestingly, the heuristic used need to negotiate over the remaining amount appeared to be an equal sharing heuristic, with C. The problem with this intuition is, of consistent with other results in social psy- course, that C will try to break up any pro- chology, not a proportionality heuristic as tocoalition between A and B to avoid being suggested by Gamson’s law. Once a sim- left with a pittance. And attractive offers to pler setting (Diermeier and Gailmard 2006) A or B always exist. That is, for each possi- or opportunities for learning were provided ble split between A and B, C can propose an (Frechette´ et al. 2003), observed behavior allocation that makes either A or B better off. more closely resembled predicted behavior; Hence, A or B may be tempted to abandon its yet, evidence for moral motivations could still protocoalition and team with C instead. Sup- be clearly detected. pose B now forms a new protocoalition with From the point of researchers trained in C. Then A can make a better offer to either game theory, much of the psychological mul- B or C and so on. tiperson negotiation literature may appear In the Baron-Ferejohn (1989) model (or unsatisfactory because too little emphasis is any of the sequential bargaining models con- placed on the incentive structure underly- sidered), this problem is avoided by the fact ing the experiment. Yet, some of the insights that once a coalition is agreed on, then potentially can be blended with a more strate- the game ends. However, in the context gically minded approach. In a recent series of coalition government, this is a problem- of papers, Swaab, Diermeier, and coauthors atic assumption because governing coalitions (Swaab et al. 2002, 2009; Diermeier et al. need to maintain the confidence of the leg- 2008) have proposed such an approach. They islature to remain in power. In other words, consider the following characteristic function during a legislative period, the game never due to Raiffa (1982). The extensive form is “ends” in a strategically relevant sense. intentionally not specified. The key for negotiating parties is thus In Table 28.1, any efficient outcome twofold: 1) they need to settle on a coali- involves the parties reaching a unanimous tional agreement that includes them, and 2) Coalition Experiments 407 they need to make sure that the coalition seventy percent vs. eleven percent). How- is stable. From a political economy point ever, face-to-face communication is a com- of view, this would require identifying some plex phenomenon. In addition to the use self-enforcing mechanism that keeps the cur- of nonverbal cues, it creates a setting that rent coalition in power (Diermeier and Merlo enables the creation of common knowledge 2000). Psychologists, however, have focused through public communication. To separate on alternative means to solve the stability these issues, the authors introduced a privacy problem. The key notion here is the con- variable to the negotiation context, allowing cept of “trust.” Intuitively, parties need to parties in both face-to-face and CMC nego- trust their protocoalition partner to be will- tiations to access private discussion spaces ing to continue their conversation with the separate from the third party. The authors third party because doing so carries the risk expected that the availability of private chat that the third party may be able to break up rooms would decrease efficiency in the nego- the current protocoalition. If such trust can- tiations regardless of communication style not be established, then parties may be better (face to face vs. CMC), but that CMC would off refusing to talk further to the out party still lead to less efficient outcomes compared to avoid giving it an opportunity to break to face-to-face communication. The expec- the current protocoalition apart, which would tations were partially supported. The abil- lead to an inefficient outcome. ity to communicate privately in the CMC Whether such trust can be established setting lowered efficiency from fifty percent may depend on various factors. For exam- to eighteen percent. In the face-to-face con- ple, players may use nonverbal communica- dition, the effect was much smaller (eighty tion to signal an agreement (they may seek percent to seventy-one percent) and not eye contact before speaking to the third party, significant at customary levels of statistical move together on the same side of the table, significance. etc.). Thus, the degree to which nonverbal The effects of communication structure factors can be used (e.g., in a face-to-face can be quite subtle. For example, consider vs. computer-based negotiation) may influ- the differences between using a private chat ence the stability of protocoalitions and nego- room and using instant messaging. In the first tiation efficiency. The idea underlying the case, the content of the conversation is pri- Swaab and Diermeier experiments is that vate, but the fact that private communication by directly manipulating the communication took place is public (the parties are observed structure, we can vary the conditions that lead when they “leave “ the common chat room), to trust among members of protocoalitions, whereas in the instant messaging case, the fact which, in turn, influences whether efficient that private communication took place may outcomes can be reached. So, the “indepen- not be known either (i.e., communication dent variable” in this approach is the commu- is secret). If the intuition that communica- nication structure, the dependent variable the tion structures influence trust between nego- percentage of efficient coalition outcomes, tiating parties is correct, then secret com- and the mediating variable trust in the proto- munication should be particularly destruc- coalition partner. tive because parties can never be sure that Diermeier et al. (2008) consider three such their coalition partner is not secretly try- settings: 1) face-to-face versus computer- ing to double-cross them. Indeed, the mere mediated decision making, 2) public ver- possibility of secret communication taking sus private communication settings, and 3) place may already undermine trust. Dier- private and secret communication settings. meieretal.(2008) find this to be the case. The first finding is unsurprising. Groups When they compare secret communication negotiating face to face were significantly to private communication, not a single group more efficient than groups negotiating via is able to reach an efficient outcome in the computer-mediated communication (CMC; secret condition. 408 Daniel Diermeier

Once the importance of communication a much earlier literature before the noncoop- structures has been established, an investiga- erative revolution. It therefore may be worth- tion of communication strategies is a natu- while to discuss some of these issues in more ral next step. This also may help clarify how detail. negotiators use language to develop a positive common rapport and shared mental models. In unpublished research, Swaab, Dier- 5. Comments on Context-Rich meier, and Feddersen (personal communica- Experiments tion, September 8, 2007) show that merely allowing participants to exchange text mes- Political scientists trained in game theory sages rather than being restricted to numeri- and experimental economists have largely cal offers doubles the amount of efficient out- ignored the extensive psychological literature comes. Huffaker, Swaab, and Diermeier (in on multiperson negotiations. In part, this may press) propose the use of tools from compu- have been due to methodological disagree- tational linguistics to investigate the effect of ments. After all, most of the recent work on language more systematically, using the same coalition bargaining fits squarely within the game form as previously mentioned. Tay- experimental economics research tradition lor and Thomas (2005), for example, point sharing its standards, values, and methods. out that matching linguistic styles and word Yet, psychologists systematically, routinely, choices improve negotiation outcomes. Sim- and intentionally violate many of the tenets ilarly, the use of assents (Curhan and Pent- that experimental (political) economists hold land 2007) and positive emotional content is sacred. Among the many possible violations expected to have a positive impact of negotia- consider the following partial list:12 tion success, whereas the use of negative emotional content may backfire (Van Beest, Van 1. Psychologists usually do not pay their Kleef, and Van Dijk 2008). Huffakeretal.use subjects for performance, but instead use zipping algorithms to measure language sim- large, fictitious monetary values. ilarities and text analytics programs to iden- 2. Psychologists usually do not specify game tify assents and emotional content. They find forms in all but the most rudimentary a significant effect of linguistic similarity and fashion. assent, but no support for positive emotions. 3. Decision problems are not presented in Negative emotions, however, do lead to fewer abstract fashion, but are richly contextu- efficient coalition outcomes. alized using fictitious context. Face-to- These are just some possibilities of includ- face interactions are common. ing richer psychological models into the study of coalition formation. The investiga- For scholars committed to the experimental tion of language is perhaps the most promis- economics paradigm, it is tempting to dis- ing because it connects to the large litera- miss much of the negotiations literature on ture on framing that is increasingly gaining such methodological grounds. But that would traction in political science; however, other be a mistake. To see why, it is important to dimensions should be considered. Examples recall that the main goal of coalition experi- include the explicit study of decision heuris- ments is to better understand coalition forma- tics, as discussed in the context of Dier- tion in real settings such as the negotiations meier and Morton (2005), or the use of over forming a new cabinet. It is not to study moral concerns (Frechette´ et al. 2003; Dier- human behavior in games, even if these games meier and Gailmard 2006). Yet, some of are intended to capture real phenomena. the methodologies used in this context, for instance, the reliance on unstructured inter- 12 Another important difference is the systematic use of deception. Although this is a crucially important action, are alien to most experimental politi- topic in many studies, it does not play a major role in cal scientists, although they do connect with the research on coalition bargaining. Coalition Experiments 409

With this background in mind, let us if–then statement is true. Human subjects are reconsider the approach taken by social psy- notoriously bad at this very elemental logical chologists. The issue of how subjects need to exercise. Yet, performance improves dramati- be paid to properly induce the desired incen- cally when the task is presented in a contextu- tives is an ongoing concern among experi- alized version as a cheater detection problem mental economists. Issues include ensuring (Cosmides and Tooby 1992). trust that payments are actually made, the For our discussion, the main insight is magnitude of the payments, whether they that contextualized versions of an abstract should be paid in cash or lottery tickets, and so decision-making problem yield very differ- on. Second, experimental game theorists take ent predictions than the abstract version. The great care to ensure that subjects are not influ- existence of these differences is nothing new enced by any other aspect of the decision con- and is widely documented in the enormous text than the one specified by the game form. literature on framing effects. Indeed, much Subjects interacting on computer terminals of the care in designing experiments in the are presented with abstract payoff matrices. game-theoretic tradition can be interpreted The motivation is to make sure that these as an attempt to eliminate these factors. But other extraneous factors do not influence sub- the lesson from the Wason test is quite dif- jects’ decision-making processes. Only that, ferent. Here, subjects perform much better it is argued, will allow us to properly test in a contextualized setting compared to the the predictions of a given model. But there is abstract one. Moreover, the contextualized an underlying assumption here. After all, our version of the Wason test (cheater detec- intended domain of application is not anony- tion) is highly relevant to coalition forma- mous agents interacting on a computer screen tion. Indeed, Cosmides and Tooby (1992) and being paid in lottery tickets. (That would argue that the reason subjects perform well be the proper domain for studying behavior in in cheater detection tasks is the evolutionary games.) Our intended domain is professional need for effective coalition formation in early politicians, who know each other very well, human society. So, if our goal is to under- participating in a series of meetings or phone stand reasoning in richly contextualized set- calls over a period of days or weeks and nego- tings, then using very abstract environments tiating over extremely high stakes that may may bias results in the wrong direction. Indeed, determine their professional career. this is precisely what we find when we com- The assumption (and promise) of the pare negotiation performance in an abstract experimental economics approach is that, as setting to a more contextualized setting (e.g., we move from the very abstract, stripped- one where agents are allowed to exchange down context in a game-theoretic setting to text messages). The results by Diermeier the richly contextualized setting of a cab- et al. (2008) further illuminate this insight inet negotiation by professional politicians, as richer communication structures correlate the main insights gained in the abstract set- with better bargaining success in a predictable ting survive. In other words, having proposal manner. power would still be important whether one This approach may open up a potential is deciding on how to split five dollars among blend of the experimental economics and players A, B, and C in a computer lab or social psychology traditions in the context negotiating on the composition of a ruling of coalition bargaining experiments. The key coalition. But recent research, ranging from challenge will be how to strike the right bal- the bias and heuristics literature to evolu- ance between specifying enough context to tionary psychology, suggests that this infer- avoid the Wason test trap, but in a fashion ence is far more problematic than previously to preserve a sufficient level of experimental believed. A well-known example is the Wason control. This blend of strategic models and test (1966), where subjects are asked to turn psychological richness may offer some highly over cards to determine whether a particular promising research directions. Political 410 Daniel Diermeier science with its dual research heritage con- Carrubba, Clifford J., and Craig Volden. 2004. taining both behavioral and formal traditions “The Formation of Oversized Coalitions in seems particularly well positioned to take Parliamentary Democracies.” American Journal 48 521 37 advantage of this opportunity. of Political Science : – . Cosmides, Leda, and John Tooby. 1992. Cogni- tive Adaptations for Social Exchange. New York: References Oxford University Press. Curhan, Jared R., and Alex Pentland. 2007.“Thin Baron, David P. 1989. “A Noncooperative Theory Slices of Negotiation: Predicting Outcomes of Legislative Coalitions.” American Journal of from Conversational Dynamics within the First Political Science 33: 1048–84. Five Minutes.” Journal of Applied Psychology 92: Baron, David P. 1991. “A Spatial Theory of Gov- 802–11. ernment Formation in Parliamentary Systems.” Davis, Douglas D., and Charles A. Holt. 1993. American Political Science Review 85: 137–65. Experimental Economics. Princeton, NJ: Prince- Baron, David P., and Daniel Diermeier. 2001. ton University Press. “Elections, Governments, and Parliaments in Diermeier, Daniel. 2006. “Coalition Govern- Proportional Representation Systems.” Quar- ment.” In Oxford Handbook of Political Economy, terly Journal of Economics 116: 933–67. eds. Barry Weingast and Donald Wittman. Baron, David P., and John A. Ferejohn. 1989. New York: Oxford University Press, 162– “Bargaining in Legislatures.” American Politi- 79. cal Science Review 89: 1181–206. Diermeier, Daniel, Hulya¨ Eraslan, and Antonio Battaglini, Marco, and Thomas R. Palfrey. 2007. Merlo. 2003. “A Structural Model of Govern- “The Dynamics of Distributive Politics.” Social ment Formation.” Econometrica 71: 27–70. Science Working Paper 1273, California Insti- Diermeier, Daniel, Hulya¨ Eraslan, and Antonio tute of Technology. Merlo. 2007. “Bicameralism and Government Bazerman, Max H., Jared R. Curhan, Don A. Formation.” Quarterly Journal of Political Science Moore, and Kathleen L. Valley. 2000. “Negoti- 2: 1–26. ation.” Annual Review of Psychology 51: 279–314. Diermeier, Daniel, and Sean Gailmard. 2006. Bazerman, Max H., Elizabeth A. Mannix, and L. “Self-Interest, Inequality, and Entitlement L. Thompson. 1988. “Groups as Mixed-Motive in Majoritarian Decision-Making.” Quarterly Negotiations.” Advances in Group Processes 5: Journal of Political Science 1: 327–50. 195–216. Diermeier, Daniel, and Sean Gailmard. n.d. “Enti- Bereby-Meyer, Yoella, and Muriel Niederle. 2005. tlements in Bilateral Bargaining.” Mimeo. “Fairness in Bargaining.” Journal of Economic Northwestern University. Behavior and Organization 56: 173–86. Diermeier, Daniel, and Antonio Merlo. 2000. Bolton, Gary E., and Axel Ockenfels. 2000a. “Government Turnover in Parliamentary “ERC: A Theory of Equity, Reciprocity, and Democracies.” Journal of Economic Theory 94: Competition.” American Economic Review 90: 46–79. 166–93. Diermeier, Daniel, and Rebecca Morton. 2005. Bolton, Gary E., and Axel Ockenfels. 2000b. “Experiments in Majoritarian Bargaining.” In “Strategy and Equity: An ERC Analysis of the Social Choice and Strategic Decisions: Essays in Guth¨ van Damme Game.” Journal of Mathe- Honor of Jeffrey S. Banks,eds.DavidAusten- matical Psychology 42: 215–26. Smith and John Duggan. New York: Springer, Browne, Eric C., and John Fendreis. 1980. 201–26. “Allocating Coalition Payoffs by Conventional Diermeier, Daniel, Roderick I. Swaab, Victoria Norm: An Assessment of the Evidence for Cab- Husted Medvec, and Mary C. Kern. 2008. inet Coalition Situations.” American Journal of “The Micro Dynamics of Coalition Forma- Political Science 24: 753–68. tion.” Political Research Quarterly 61: 484–501. Browne, Eric C., and Mark N. Franklin. 1973. Eraslan, Hulya,¨ and Antonio Merlo. 2002. “Major- “Aspects of Coalition Payoffs in European ity Rule in a Stochastic Model of Bargaining.” Parliamentary Democracies.” American Politi- Journal of Economic Theory 103: 31–48. cal Science Review 67: 453–69. Fehr, Ernst, and Klaus M. Schmidt. 1999.“AThe- Burhans, David T., Jr. 1973. “Coalition Game ory of Fairness, Competition, and Coopera- Research: A Reexamination.” American Journal tion.” Quarterly Journal of Economics 114: 769– of Sociology 79: 389–408. 816. Coalition Experiments 411

Fiorina, Morris P., and Charles R. Plott. 1978. McKelvey, Richard D. 1991. “An Experimen- “Committee Decisions under Majority Rule: tal Test of a Stochastic Game Model of An Experimental Study.” American Political Sci- Committee Bargaining.” In Laboratory Research ence Review 72: 575–98. in Political Economy, ed. Thomas R. Palfrey. Forsythe, Robert, Joel L. Horowitz, N. E. Savin, Ann Arbor: University of Michigan Press, and Martin Sefton. 1994. “Fairness in Simple 139–67. Bargaining Experiments.” Games and Economic Messick, David M. 1993. “Equality as a Decision Behavior 6: 347–69. Heuristic.” In Psychological Perspectives on Justice: Frechette,´ Guillaume R., John H. Kagel, and Theory and Applications, eds. Barbara A. Mellers Steven F. Lehrer. 2003. “Bargaining in Legis- and Jonathan Baron. New York: Cambridge latures: An Experimental Investigation of Open University Press, 11–31. versus Closed Amendment Rules.” American Morelli, Massimo. 1999. “Demand Competition Political Science Review 97: 221–32. and Policy.” American Political Science Review 93: Frechette,´ Guillaume R., John H. Kagel, and Mas- 809–20. simo Morelli. 2005a. “Gamson’s Law versus Ochs, Jack, and Alvin E. Roth. 1989. “An Experi- Non-Cooperative Bargaining Theory.” Games mental Study of Sequential Bargaining.” Amer- and Economic Behavior 51: 365–90. ican Economic Review 79: 355–84. Frechette,´ Guillaume R., John H. Kagel, and Raiffa, Howard. 1982. The Art and Science of Nego- Massimo Morelli. 2005b. “Nominal Bargain- tiation. Cambridge, MA: Belknap. ing Power, Selection Protocol, and Discount- Roth, Alvin E. 1995. “Bargaining Experiments.” ing in Legislative Bargaining.” Journal of Public In The Handbook of Experimental Economics, eds. Economics 89: 1497–517. John Kagel and Alvin Roth. Princeton, NJ: Frechette,´ Guillaume R., John H. Kagel, and Mas- Princeton University Press, 253–348. simo Morelli. 2005c. “Behavioral Identifica- Roth, Alvin E., Vesna Prasnikar, Masahiro Okuno- tion in Coalitional Bargaining: An Experimen- Fujiwara, and Shmuel Zamir. 1991.“Bargain- tal Analysis of Demand Bargaining and Alter- ing and Market Behavior in Jerusalem, Ljubl- nating Offers.” Econometrica 73: 893–939. jana, Pittsburgh, and Tokyo: An Experimen- Gamson, William A. 1961. “A Theory of Coali- tal Study.” American Economic Review 81: 1068– tion Formation.” American Sociological Review 95. 26: 373–82. Schofield, Norman, and Michael Laver. 1985. Gamson, William A. 1964. “Experimental Studies “Bargaining Theory and Portfolio Payoffs of Coalition Formation.” In Advances in Exper- in European Coalition Governments, 1945– imental Social Psychology. vol. 1, ed. Leonard 1983.” British Journal of Political Science 15: 143– Berkowitz. San Diego: Academic Press. 64. Guth,¨ Werner, Rolf Schmittberger, and Bernd Swaab, Roderick I., Mary C. Kern, Daniel Dier- Schwarze. 1982. “An Experimental Analysis meier, and Victoria Medvec. 2009. “Who Says of Ultimatum Bargaining.” Journal of Economic What to Whom? The Impact of Communica- Behavior and Organization 75: 367–88. tion Setting and Channel on Exclusion from Huffaker, David, Roderick I. Swaab, and Daniel Multiparty Negotiation Agreements.” Social Diermeier. in press. “The Language of Coali- Cognition 27: 385–401. tion Formation in Online Multiparty Negotia- Swaab, Roderick I., Tom Postmes, Peter Neijens, tions.” Journal of Language and Social Psychology. Marius H. Kiers, and Adrie C. M. Dumay. Kagel, John H., and Katherine Willey Wolfe. 2002. “Multiparty Negotiations Support: The 2001. “Test of Fairness Models Based on Role of Visualization’s Influence on the Devel- Equity Considerations in a Three-Person Ulti- opment of Shared Mental Models.” Journal of matum Game.” Experimental Economics 4: 203– Management Information Systems 19: 129–50. 20. Taylor, Paul J., and Sally Thomas. 2005. “Linguis- Laver, Michael, and Norman Schofield. 1990. tic Style Matching and Negotiation Outcome.” Multiparty Government: The Politics of Coali- Paper presented at the International Associa- tion in Europe. Oxford: Oxford University tion for Conflict Management (IACM) 18th Press. Annual Conference, Seville, Spain. Retrieved Martin, Lanny W., and Randolph T. Stevenson. from http://papers.ssrn.com/sol3/papers.cfm? 2001. “Government Formation in Parliamen- abstract id=736244 (November 16, 2010). tary Democracies.” American Journal of Political Van Beest, Ilja, Gerben A. Van Kleef, and Eric Science 45: 33–50. Van Dijk. 2008. “Get Angry, Get Out: The 412 Daniel Diermeier

Interpersonal Effects of Anger Communication of Payoffs in Coalition Governments.” British in Multiparty Negotiation.” Journal of Experi- Journal of Political Science 31: 627–49. mental Social Psychology 44: 993–1002. Wason, P. C. 1966. “Reasoning.” In New Hori- Warwick, Paul V., and James N. Druckman. 2001. zons in Psychology, ed. Brian Malzard Foss. Har- “Portfolio Salience and the Proportionality mondsworth: Penguin Press, 135–51. CHAPTER 29

Negotiation and Mediation

Daniel Druckman

Knowledge about negotiation and media- States and the Soviet Union over the 1948– tion comes primarily from laboratory exper- 49 blockade of Berlin, between Kennedy iments. The question asked in this chap- and Khrushchev in 1962 over Soviet missile ter is what value is added by experiments bases in Cuba, and between Carter and Kho- for understanding processes of elite bargain- meini concerning American hostages in Iran ing? This question is addressed in the fol- in 1979–80. Leaders and their representatives lowing sections. After describing the inter- also confront each other face to face to dis- national negotiation context, I provide a cuss their interests over security, monetary brief overview of the experimental approach. and trade, or environmental issues. These Then, key studies on distributive and integra- meetings may take the form of summits, such tive bargaining are reviewed, as well as exam- as the 1986 meeting between Reagan and ples of experiments that capture complexity Gorbachev in Reykjavik, or more protracted without forfeiting the advantages of experi- meetings, such as the long series of talks mental control. The chapter concludes with a between their countries’ representatives over discussion of the value added by experiments. arms control, beginning with the Strategic Arms Limitations Talks and winding up with the Strategic Arms Reduction Talks. The Context Many negotiations occur among more Negotiating in the international context takes than two nations. They may occur between several forms. It occurs from a distance and blocs, such as the North Atlantic Treaty face to face; deals with multiple complex Organization–Warsaw Pact discussions in the issues; and includes bilateral, multilateral, and 1970s over mutual and balanced force reduc- global participation. National leaders often tions. They may take the form of three- make demands or exchange proposals from or four-party discussions at which simulta- a distance. Well-known examples include neous bilateral negotiations take place. One the bilateral exchanges between the United example is the discussion among Iceland,

413 414 Daniel Druckman

Norway, Russia, and the Faroe Islands over 1. The Experimental Method fishing rights in the North Atlantic. Although Icelandic negotiators rejected the Russo- Most elite bargainers are career profes- Norwegian offer, they reached an agreement sionals.1 They differ in many ways from with the Faroes; the Norwegians protested the subjects who serve as role players in this agreement. Other examples of simultane- negotiation experiments. Among the differ- ous bilateral talks come from the area of free ences are experience, stakes, issue-area exper- trade such as the North American Free Trade tise, actors in bureaucratic politics, imple- Agreement (Canada, the United States, and mentation challenges, and accountability to Mexico) and between Singapore, Australia, government agencies or to international and the United States. From the area of secu- organizations. However, there are some sim- rity comes the example of the 1962–63 par- ilarities: similar bargaining choice dilemmas, tial nuclear test ban talks between the United decision-making processes, tactical options, States, Great Britain, and the Soviet Union. and intrateam or coalition dynamics. A ques- Negotiations also occur in multilateral set- tion is whether we emphasize the differences tings, where representatives from many or the similarities. The case for differences is nations gather for discussions of regional, made by Singer and Ray (1966), who pointed continental, and global issues. Notable exam- out several “critical” dimensions of differ- ples are the Uruguay Round of the General ence that exist between the small group lab- Agreements on Tariffs and Trade, the negoti- oratory where decision-making experiments ations establishing the European Community are conducted and the more complex bureau- (the Single European Act), the ongoing dis- cracies in which policy-making decisions are cussions among members of the Organization made. The argument for similarities was for Security and Cooperation in Europe and made by Bobrow (1972): “We should move among members of the UN Security Coun- rapidly toward treating phenomena that cross cil, the talks that led to the Montreal Protocol national lines as instances of phenomena that on ozone depletion, and the discussions that occur in several types of social units. Accord- resulted in the Rio Declaration on Environ- ingly, alliances become coalitions; negotia- ment and Development. tions between nations become bargaining; These examples share a number of fea- foreign policy choices become decision mak- tures, including high-stakes and high-drama, ing” (55). Both arguments have merit. An multilevel bargaining at the intersection emphasis on differences is reflected in the case between intra- and international actors; the study tradition of research. Similarities are need to manage complexity; accountability assumed when experimentalists argue for rel- to national constituencies; implications for evance of their findings to the settings being national foreign policies; experienced nego- simulated. In this chapter, I discuss implica- tiators; ratification (for treaties); and con- tions of experimental research to elite bar- cern for proper implementation of agree- gaining in the international setting. The value ments. Many of these features are captured added by this research reinforces the “sim- by detailed case studies of particular negoti- ilarities” perspective. It does not, however, ations. They are difficult to study in exper- diminish the importance of the differences iments, even when attempts are made to listed previously. I have argued elsewhere simulate real-world settings. What then can for striking a balance between the respective experiments offer? This question is addressed strengths of case-oriented and experimental by showing that experiments provide added value to the contributions made by case stud- 1 The career professional designation would apply to ies. Knowledge gained from experiments is civil servants or foreign service officers, but not to presented following a discussion of the rel- political appointees. The latter are usually appointed for relatively brief stints as special envoys or ambas- evance of the experimental method to the sadors. Their term in office typically ends when study of elite bargaining. administrations change. Negotiation and Mediation 415 research on negotiation, and I return to this 2. Distributive Bargaining idea in later sections. Two of the more vigorous proponents Early experiments on negotiation focused of the experimental method on bargaining primarily on distributive bargaining. This argued that “abstraction and model build- refers to situations in which the interests ing are necessary to reduce the problem to of the bargainers are in conflict and where manageable proportions. The experimental each attempts to obtain the largest share of method can contribute to the process of iden- whatever is being contested. These contests tifying critical variables and the nature of often conclude with agreements on outcomes their roles in conflict situations” (Fouraker somewhere between the bargainers’ open- and Siegel 1963, 207). It is this heuristic ing positions. Bargaining researchers have function of experiments that may be most been concerned with factors that influence valuable. It tells us where to look – which 1) whether an agreement will be reached, 2) variable or cluster of variables accounts for the amount of time needed to reach an agree- negotiation behavior? By the early 1970s, ment, 3) the type of agreement reached (as we had already accumulated a storehouse of equal or unequal concessions), and 4) the bar- knowledge about bargaining from the labo- gainers’ satisfaction with the agreement and ratory (see Rubin and Brown 1975). Spurred their willingness to implement it. on by the early accomplishments, bargaining researchers have added additional store- Offer Strategies houses to the “property.” A steady increase of publications, and the founding of sev- A large number of experiments were con- eral journals and professional associations ducted in the 1960s, spurred by Siegel and dedicated to the topic, has resulted in a Fouraker’s (1960) findings about levels of cross-disciplinary epistemic community of aspirations or goals. They found that “the researchers. The list of variables explored bargainer who 1) opens negotiations with a has expanded considerably, frameworks and high request, 2) has a small rate of concession, models abound, and innovative methodolo- 3) has a high minimum level of expectation, gies have emerged. An attempt is made to and 4) is very perceptive and quite unyield- capture these developments without losing ing, will fare better than his opponent who sight of the challenge of relevance to elite provides the base upon which these relative bargaining. evaluations were made” (93). These findings The experimental literature is organized suggest that toughness pays. A question raised into two parts. One, referred to as distribu- is do these results apply to a wide range of tive bargaining, reviews the findings from a bargaining situations? The question of gen- large number of studies conducted primar- erality was evaluated by a flurry of experi- ily from 1960 to 1980. Another, referred to ments conducted during the 1960s and 1970s. as integrative bargaining or problem solving, Many of those experiments examined a bar- discusses a smaller number of experiments gainer’s change in offers made in response to conducted more recently. This distinction, the other’s concession strategy. suggested originally by Walton and McKer- The bargaining studies did not support sie (1965), has resonated as well with pro- the generality of the Siegel-Fouraker (1960) cesses of elite bargaining in the international conclusion. They showed that a hard offer context (Hopmann 1995). Both sections trace strategy works only under certain conditions: the development of research from the earli- when the bargainer does not have infor- est experiments, which provided a spark for mation about the opponent’s payoffs and later studies. Relevance to elite bargaining when there is substantial time pressure. The is demonstrated with results obtained from chances of a settlement increased when the analyses of distributive and integrative bar- opponent used a soft or intermediate offer gaining processes in situ. rather than a hard offer strategy. The best 416 Daniel Druckman overall strategy for obtaining agreement is that “toughness pays.” More recently, Lytle matching: it resulted in greater bargainer and Kopelman (2005) showed that better dis- cooperation than unconditional cooperation, tributive outcomes occurred when bargain- unconditional competition, or partial cooper- ers’ threats (hard strategy) were combined ation (for a review of the findings, see Hamner with friendly overtures (softer strategy). and Yukl 1977). Another example of convergence between These findings support Osgood’s (1962) findings obtained in the laboratory and from well-known argument that cooperation will a real-world case is provided by Druckman be reciprocated rather than exploited. and Bonoma (1976) and Druckman (1986). Referred to as graduated reciprocation in tension The former study was conducted with chil- reduction (GRIT), Osgood reasoned that uni- dren bargaining as buyers and sellers. The lateral concessions would remove the main results showed that disappointed expectations obstacle to an opponent’s concession mak- for cooperation led bargainers to adjust their ing, which is distrust. The initial conces- concessions, leading to a deadlock. The lat- sion would set in motion a cycle of recip- ter study was conducted with documentation rocated or matched concessions. Support for from a military base rights case and analyzed this hypothesis was obtained by Pilisuk and with the BPA coding categories. The results Skolnick (1968): they found that the best also showed that negotiators adjusted their strategy is one that uses conciliatory moves in offer strategy when expectations for coop- the beginning and then switches to matching. eration were disappointed: the time-series Referred to also as “tit for tat,” the match- analysis revealed a pattern of switching from ing strategy has been effective in produc- soft to hard moves when the discrepancy ing agreements over the long term (Axelrod between one’s own and others’ cooperation 1980). It has been demonstrated by Crow increased. The resulting mutual toughness (1963) in an internation simulation (without led to an impasse that often produced a control groups) and in partial nuclear test turning point in the talks. Referred to as ban talks, referred to as the “Kennedy exper- “threshold adjustment,” this pattern has been iment” (Etzioni 1967). The test ban case was demonstrated in eight cases of international also used as a setting for hypothesis testing. negotiation (Druckman and Harris 1990). The distinction between hard and soft bar- The similar findings obtained from a labo- gaining strategies has informed analyses of ratory study with children and from a case simulated and actual international negotia- study with professional negotiators bolster tions. Using a coding system referred to as the argument for generality of negotiation “bargaining process analysis” (BPA), Hop- processes. Taken together, the Hopmann- mann and Walcott (1977) showed convergent Wolcott and Druckman studies underscore findings from a simulation and case study of the relevance of laboratory research for the 1962–63 Eighteen-Nation Disarmament understanding real-world elite bargaining. Conference leading to the agreement on the Partial Nuclear Test Ban Treaty. They found The Bargaining Environment that increased tensions in the external environment increased 1) the amount of hostility The early bargaining experiments focused in mutual perceptions, 2) the proportion of primarily on the other bargainer’s conces- hard relative to soft bargaining strategies, 3) sion behavior and such features of the setting the employment of commitments, 4)theratio as time pressure and atmosphere. Bargain- of negative to positive affect, and 5)theratio ing moves or concessions and outcomes were of disagreements to agreements in substantive the key dependent variables. Other inde- issues under negotiation. The increase in hos- pendent variables studied during this period tile attitudes and the toughening of positions were group representation, prenegotiation detracted from arriving at agreements. These experience, and orientation. Blake and Mou- results provide additional refuting evidence ton’s (1961, 1962) Human Relations Training for the Siegel-Fouraker (1960) conclusion Laboratory served as a venue for experiments Negotiation and Mediation 417 on the impact of representing groups on These results were explained in terms of resolving intergroup disputes. They con- the interdependent structure of the tasks and cluded that the commitments triggered by aspirations for cooperation or solidarity. Sim- representation are a strong source of inflex- ilar results were obtained by Druckman and ibility in negotiation. Although fraught with Albin (2010) for outcomes of peace agree- problems of inadequate controls, their work ments. In their comparative study, equality stimulated a fruitful line of investigation. mediated the relationship between the con- These studies showed that representation flict environment and the durability of the effects on flexibility are contingent on the agreements. Thus, again, convergent findings stakes: high stakes in the form of payoffs or were obtained between laboratory and case reputations produce stronger effects than low analyses. stakes. However, the pressures can be offset by the setting and by salient outcomes: pri- Summary vate negotiations (Organ 1971) and fair solutions (Benton and Druckman 1973) serve to The discussion in this section reveals contri- increase a representative’s flexibility, leading butions made by experiments to our under- to agreements rather than impasses. standing of distributive bargaining processes. But it is also the case that other variables Three contributions are highlighted. One have stronger impacts on bargaining behavior concerns the effects of different bargaining than representation. One of these variables strategies: the best strategy is likely to con- is a bargainer’s orientation as competitive or sist of generous opening moves combined cooperative. Particularly strong effects were with matching or reciprocating concessions. obtained when the orientation was manipu- Another deals with the impact of various fea- lated in instructions (Summers 1968; Organ tures of the bargaining situation: bargain- 1971). Another was prenegotiation experi- ing orientation and prenegotiation have the ence. The key distinction is between study- strongest impact on compromising behav- ing issues (more flexibility) and strategizing ior. A third contribution is to more complex (less flexibility) prior to negotiation – whether negotiation settings: simulations of interna- this activity was done unilaterally (own team tional negotiations have demonstrated the members only) or bilaterally (both teams) deleterious effects of stress, the impasse- made little difference (Bass 1966; Druck- producing impact of asymmetric power struc- man 1968). Of the ten independent variables tures, and opportunities provided by impasses compared in a meta-analysis on compromis- for progress. Relevance of experiments is bol- ing behavior, these produced the strongest stered further by convergent results obtained effect sizes (Druckman 1994). Representation from case studies, including the systematic and accountability ranked sixth and seventh, analysis of single cases (Etzioni 1967; Hop- respectively.2 mann and Walcott 1977; Beriker and Druck- Many of the experimental findings call man 1996) and comparative analyses of a rel- attention to the importance of reciprocity in atively large number of cases (Druckman and bargaining. Regarded as a norm (Gouldner Harris 1990; Druckman 2001; Druckman and 1960), reciprocal moves reflect a principle Albin 2010). The convergence between find- distributive principle of equality. Strong sup- ings obtained on equality in the laboratory port for this principle is found in Deutsch’s and from analyses of peace agreements is par- (1985) experiments on distributive justice. ticularly striking. His laboratory subjects showed a strong preference for equal distributions with little variation across subject populations or tasks. 3. Integrative Bargaining

Another perspective on negotiation emerged 2 Other variables in the analysis included time pressure, initial position distance, opponent’s strategy, large and influenced experimentation beginning in versus small issues, framing, and visibility. the 1970s. This perspective, referred to as 418 Daniel Druckman integrative bargaining, describes a situation are shown to be effective. One is referred to where parties attempt to jointly enlarge the as heuristic trial and error (HTE): each bar- benefits available to both (or all) so that gainer seeks the other’s reactions to a variety they may gain a larger value than attained of proposals and options, known also as trial through compromise. The focus is on positive balloons. Another is information exchange: sum rather than nonzero sum (mixed motive) each bargainer asks for and provides informa- outcomes. Conceived of initially by Follett tion about needs and values. Both strategies (1940) and developed further by Walton convey flexibility; they contrast to the dis- and McKersie (1965) and Rapoport (1960), tributive strategies that convey rigidity in the the approach gained momentum with the process of seeking favorable outcomes. Their popular writing of Fisher and Ury (1981), effectiveness depends, however, on maintain- the decision-theory approach taken by Raiffa ing a problem-solving orientation through- (1982), and Zartman and Berman’s (1982) out the bargaining process. They also depend diagnosis-formula-detail perspective. These on mutual resolve in maintaining high aspi- theoretical contributions have been comple- rations, referred to as rigidity with respect to mented by experiments designed to provide goals. empirical foundations for the concept. Each strategy also has limitations. Fur- The most compelling argument for inte- ther experiments revealed the challenges. grative bargaining comes from experimental The effectiveness of HTE depends on findings. Results obtained across many exper- either knowing or constructing the inte- iments conducted by Pruitt and his colleagues grative options from available information. show that the average correlation between When these options are not known, bargain- joint profits and distributive (integrative) bar- ers must reconceptualize the issues or try gaining behavior is inverse (direct) and sta- new approaches. This requires some form of tistically significant. Yet, despite these find- information exchange. Discussing values and ings, bargainers tend to prefer distributive priorities can provide insight into the joint approaches. Why does this occur? An answer reward structure, but it can also backfire when to this question is provided by Pruitt and the information reveals other incompatibili- Lewis (1977). Parties tend to imitate each ties. Thus, the new information can either other’s distributive behavior: threats elicit move the process forward or embroil the par- counterthreats, and bargainers are less will- ties in a continuing impasse. The former con- ing to make concessions to the extent that sequence is more likely to occur when both the other’s demands are viewed as being bargainers commit to a problem-solving ori- excessive. Thus, bargaining may “gravitate entation. However, that orientation, which toward a distributive approach because it also requires mutual resolve, can result in requires only one party to move the interac- impasses as well. What then can bargainers do tion in that direction, while the firm resolve to encourage the positive and discourage the of both parties is needed to avoid such move- negative impacts of these strategies? Insights ment” (170). Their experiments explored come from the results of more recent experi- the strategies that encourage this mutual ments. “resolve.” The systematic construction of alternative offer packages has been shown to be an effective HTE strategy. Referred to as multiple Integrative Strategies equivalent simultaneous offers (MESOs), this The key finding is that integrative bargaining strategy consists of presenting the other bar- (and high joint outcomes) depends on flexible gainer with alternative packages of roughly rigidity. This approach consists of remaining equal perceived value. It has been more ben- relatively rigid with respect to goals but flexi- eficial than single package offers. Experimen- ble with regard to the strategies used to attain tal results showed that 1) more offers were these goals. The research reveals how this accepted; 2) more satisfaction was expressed approach may be achieved. Two strategies with the accepted offers; 3) the presenting Negotiation and Mediation 419 bargainer was viewed as being more flexible; pave the way for eventual integrative out- and 4) when both bargainers used MESOs, comes. they were more likely to reach an efficient An interesting chicken-and-egg problem outcome (Medvec et al. 2005). The multiple emerges from these experimental findings: and simultaneous features of MESOs provide does reduced conflict depend on achiev- an opportunity to compare alternative pack- ing integrative agreements, or do integrative ages and then choose one from the menu. agreements depend on a relaxation of ten- This is likely to enhance the perceived value sions? A way around the dilemma of causality of the choice made by an opposing bar- is to assume that the problem is circular, that 3 gainer. The equivalent feature assures that is, that context and process are intertwined. each choice provides the same value to the Hopmann and Walcott’s (1977) simulation presenting bargainer. This is especially the findings show that more agreements occur case when a well-defined scoring system is when tensions are reduced. Thus, context used. And, because the priorities of both bar- influences outcomes. Integrative processes gainers must be understood prior to con- and outcomes have improved relationships in structing the packages, the chosen offer is case studies on the durability of peace agree- likely to be an integrative outcome (Medvec ments (Druckman and Albin 2010), computer and Galinsky 2005). simulations that model distributive and inte- The process of constructing MESOs grative negotiations (Bartos 1995), and exper- includes developing an understanding of both imental simulations that compare facilitation one’s own and others’ priorities. Differ- with fractionation approaches to negotia- ent priorities are a basis for trades, known tion (Druckman, Broome, and Korper 1988). as “logrolling.” This is also an element in Thus, processes and outcomes influence con- the information exchange process. However, text. These findings suggest that negotiation information exchange goes further. It encour- processes are embedded in contexts. Inte- ages bargainers to explore each other’s under- grative bargaining is facilitated by amiable lying interests, values, and needs, which may relationships between the bargainers. It is be regarded as root causes of the conflict. also encouraged when negotiators maintain a The sensitivities involved in such deep probes problem-solving orientation throughout the can escalate the conflict, as Johnson (1967) bargaining process. discovered in his hypothetical court case experiment and Muney and Deutsch (1968) Problem-Solving Orientation reported in their social issues simulation. The information received by bargainers in a role- Recall that a problem-solving orientation reversing condition revealed incompatibili- increases the effectiveness of integrative ties that led to impasses. However, the infor- strategies and the chances of obtaining favor- mation did produce greater understanding of able joint outcomes (Pruitt and Lewis 1977). the other’s positions: more attitude and cog- A key question is how to sustain this ori- nitive change occurred in the role reversing entation. Experiments have provided some than in the self-presentation conditions of clues, including priming, vigorous cognitive their experiments (see also Hammond et al. activity, and mediation. Results from a meta- 1966). These findings suggest that short-term analysis of bargaining experiments showed bargaining failures may not impede long- that primed orientations produced stronger term efforts at resolving conflicts; that is, effects on outcomes than unprimed (or the insights achieved during the information selected) orientations. The strongest effects exchange process may be valuable in diagnos- were produced by constituent or supervisor ing the other’s intentions (Van Kleef et al. communications to adopt either cooperative 2008). They may also contribute to future or competitive strategies (e.g., Organ 1971). workshops designed to reduce hostility and 2000 negative stereotypes (Rouhana ). The 3 For a discussion of these issues in the area of arms diagnoses and reduced hostility may, in turn, control, see Druckman and Hopmann (1989). 420 Daniel Druckman

The weakest effects occurred when bargain- a mediator’s activities can be phased with ers were selected on prenegotiation attitudes early suggestions geared toward compromise toward cooperation (negotiation as a prob- and later advice oriented toward agreements lem to be solved) or competition (negotiation that provide more joint benefits than a com- as a win–lose contest) (e.g., Lindskold, Wal- promise outcome. Thus, a trusted mediator ters, and Koutsourais 1983). Thus, explicit can effectively encourage bargainers to sus- communications or instructions help sustain tain a problem-solving orientation. a problem-solving or competitive bargaining The research has provided an answer to the strategy. question about the conditions for sustained A field study conducted by Kressel et al. problem solving. They combine strong com- (1994) compared mediators who used either a munications from constituents or principals problem-solving style (PSS) or a settlement- with mediator activities that enhance credi- oriented style (SOS) in child custody cases. bility and identify a solution that maximizes The former (PSS) approach emphasizes the joint benefits. However, another question value of searching for information that can be remains: do the laboratory findings corre- used to reach an integrative agreement. The spond to results obtained from studies of real- latter (SOS) emphasizes the value of efficient world negotiations? This section concludes compromise solutions. Although SOS was with a discussion of research that addresses preferred by most mediators, PSS produced this question. better outcomes. It resulted in more frequent durable settlements as well as a generally Problem Solving in Situ more favorable attitude toward the mediation experience. A key difference between In her analyses of thirteen cases of histori- the approaches is effort. To be effective, cal negotiations involving the United States, PSS requires vigorous cognitive activity that Wagner (2008) found that the sustained use includes three linked parts: persistent ques- of problem-solving behaviors was strongly tion asking, an analysis of sources of conflict, correlated with integrative outcomes. This and a plan for achieving joint benefits. Thus, finding corresponds to experimental results a structured and vigorous approach by nego- showing higher joint profits for bargainers tiators or mediators is needed to sustain and who use problem-solving strategies. But the reap the benefits from problem solving. case data also provided an opportunity to Progress toward integrative outcomes also refine this result in two ways. By divid- depends on the perceived credibility of the ing her cases into six stages, Wagner could mediator. Suggestions made by mediators are examine trends in problem-solving behav- more likely to be taken seriously when the ior. Although sustained problem solving was implications for who gives up what are clear needed for integrative outcomes, the best and do not favor one bargainer over the other. outcomes occurred for cases where these An experiment by Conlon, Carnevale, and behaviors were frequent during the first two Ross (1994) showed that mediators who sug- thirds of the talks, particularly in the fourth gest compromises (equal concessions by all stage. These outcomes were also facilitated bargainers) produced more agreements than when negotiators developed formulas during those who made suggestions that could result the early stages: for example, identifying the in either asymmetric (favoring one party terms of exchange to guide bargaining in a more than another) or integrative (favoring 1942 trilateral trade talk between the United both parties but complex) outcomes. The States, United Kingdom, and Switzerland, fair mediator is given latitude to encour- or identifying joint goals for each article in age bargainers to take risks, such as avoiding the 1951–52 United States–Japan Adminis- the temptation to agree on the compromise trative Agreement. These refinements extend outcome in favor of information exchange the experimental results to processes that are toward the more complex integrative agree- less likely to occur in relatively brief labora- ment. An implication of these findings is that tory simulations. Negotiation and Mediation 421

Other correspondences to laboratory native proposals. Effectiveness is increased results were obtained from Wagner’s (2008) when this process is done systematically in the analyses. The professional negotiators in her form of MESOs. Another strategy, referred cases bargained more than they problem to as information exchange, consists of ask- solved; that is, in only one case did the ing for and providing information about val- percentages of problem-solving statements ues and needs. Effectiveness depends on the exceed forty percent. This finding is echoed extent to which the new information facil- by Hopmann’s (1995) appraisal of interna- itates the search for integrative solutions; tional negotiators’ focus on relative gains it is reduced when the information reveals and competition, and it concurs with results additional incompatibilities between nego- from comparative cases analyses on negotia- tiators. The effectiveness of these strategies tions to resolve violent conflicts (Irmer and also depend on relationships between the Druckman 2009). It corresponds to Pruitt negotiators, their willingness to sustain a and Lewis’ (1977) observations about pref- problem-solving orientation throughout the erences for distributive bargaining among process, and the perceived credibility of medi- laboratory bargainers and to Kressel et al.’s ators. These findings come from experiments. (1994) tabulation of the relative frequencies They correspond to results obtained from of SOS (59% of the cases) to PSS (41%). Her analyses of complex, real-world negotiations. finding that negotiators track each other’s Those analyses also refine the experimen- behavior by responding in kind to the other’s tal results by capturing trends in problem- moves resonates with process findings from solving behavior through stages and calling the experiments analyzed in De Dreu, Wein- attention to the usefulness of formulae as gart, and Kwon’s (2000) meta-analysis. In guides to bargaining. both the experiments and cases, many negotiators reciprocated each other’s problem- solving behaviors.4 This sort of reciprocation 4. Capturing Complexity by amateurs and professionals was particu- in the Laboratory larly likely for negotiators who understood negotiation strategies. Thus, educating nego- The correspondences obtained between lab- tiators about strategies – particularly the dis- oratory and case findings on distributive tinction between distributive and integrative and integrative bargaining suggest that these bargaining – may increase their propensity to are general processes that occur in a vari- use approaches that are more likely to lead to ety of negotiating situations. An example better outcomes (see also Odell [2000]onthis is the importance of a sustained problem- point). solving orientation throughout the bargaining process: sustained problem solving led to integrative agreements. An advantage of the Summary laboratory is to provide a platform for causal The discussion in this section highlights con- analysis. These analyses do not, however, tributions made by experiments to our under- reveal details of processes that occur in partic- standing of integrative bargaining. A chal- ular real-world negotiations. An advantage of lenge for both negotiators and mediators is case studies is that they provide an opportu- to resist the temptation to engage in distribu- nity to record – often through the lens of con- tive bargaining. Early experiments showed tent analysis categories – the details. Conclu- that two strategies are likely to be effective. sions from these analyses may take the form of One, referred to as HTE, consists of seek- such statements because an increased preva- ing the other’s reaction to a variety of alter- lence of problem-solving behavior in the middle stages (as compared to early and late 4 This finding corresponds to the preference for dis- stages) occurred in cases that resulted in inte- tributive equality obtained in experiments by Deutsch (1985) and in case analyses by Druckman and Albin grative agreements. A challenge for analysts (2010). These studies were discussed previously. is to find a way of combining the advantages 422 Daniel Druckman of the experimental laboratory with those of tation of agreements (Sawyer and Guetzkow detailed case studies. The discussion in this 1965; Randolph 1966). The frameworks have section addresses that challenge. been useful for organizing literature reviews The challenge is met by incorporating (Druckman 1973), chapters in edited books complexity in laboratory environments with- on negotiation (Druckman 1977), case studies out forfeiting the key advantages of experi- (Ramberg 1978), scenario construction (Bon- mental design, namely, random assignment ham 1971), and teaching and training courses and controls. An attempt to address this issue (Druckman 1996, 2006), as well as guides was made by the early research on the inter- for web-based, computer-generated advice nation simulation (INS). This ambitious pro- on impasse resolution (Druckman, Harris, gram of research encompassed a wide vari- and Ramberg 2002). As organizing devices, ety of studies ranging from abstract models these frameworks are primarily synthetic or (e.g., Chadwick 1970) to simulation experi- integrative. The question of interest is how ments (e.g., Bonham 1971). Yet, despite this to bridge the gap between frameworks, which variety, researchers shared the goal of pro- capture complexity, and experiments, which ducing a valid corpus of knowledge about investigate causal relations among a few vari- international relations. Their collective suc- ables. This question is addressed by research cess was documented by ratings of correspon- on the situational levers of negotiating dence among INS findings and other sources flexibility. of data, including anecdotal reports, experiments, and field studies: The results were 5 Situational Levers mixed (Guetzkow and Valadez 1981). More relevant perhaps for this chapter were the This project was an attempt to reproduce efforts made by the INS researchers to design the dynamics of actual cases in a random- complex laboratory environments that per- ized experimental design. Key variables from mitted detailed data collections and analy- the Sawyer-Guetzkow framework – sixteen ses. These environments are examples of how in all – were incorporated in each of four complexity can be incorporated in laboratory stages (prenegotiation planning, setting the settings. They served as models for the sys- stage, the give and take, and the endgame) tematic comparisons performed with negoti- of a conference referred to as “Cooperative ation simulations. Measures to Reduce the Depletion of the Ozone Layer.”6 Drawing on previous studies, hypotheses were developed about the timing Frameworks and effects of each variable on negotiating Frameworks have been constructed to orga- behavior: for example, issue positions were nize the various influences and processes of linked or not linked to political ideologies international negotiation. These include pre- in the prenegotiation stage, and a deadline conditions, issues, background factors, con- did or did not exist in the endgame. Three ditions, processes, outcomes, and implemen- experimental conditions were compared: 1) all variables in each stage were geared toward 5 Among the strongest correspondences was the hypothesized flexibility (issues not linked to arousal of identification with the fictitious nations. ideologies, deadline), 2) all variables geared Role players identified with their laboratory groups toward inflexibility (issues linked to ideolo- in a manner similar to decisions makers in the system 3 being simulated. These findings bolster the case for gies, no deadline), and ) a mixed condition external validity of laboratory studies. They also arbi- proceeding from hypothesized inflexibility trated between alternative theories of ethnocentrism. in the early stages (issues linked to ideolo- However, it is also the case that these results may be due to the role players’ theories about how they may gies) to hypothesized flexibility in later stages be expected to behave. Referred to as demand characteristics, the role expectations of simulation participants is an alternative explanation for the corre- 6 This simulation was modeled on the 1992 global envi- spondences obtained between simulation and field ronmental declaration on environment and develop- findings (Janda in press). ment negotiated in Rio de Janeiro. Negotiation and Mediation 423

(a deadline). This was a 3 (flexibility condi- gies rather than studying the issues. It was tion) × 4 (stages) experimental design. The likely in later stages when wide media cov- simulation was replicated with two samples: erage occurred and when attractive alterna- environmental scientists at a Vienna-based tives were available (see also Druckman and international organization and diplomats at Druckman 1996). the Vienna Academy of Diplomacy (Druck- Additional experiments provided insights man 1993). By bringing elite bargainers into into the timing of moves and the role of medi- the laboratory, the relevance of the findings ation (Druckman 1995). Negotiators reached for international negotiation is increased. agreement more often when their oppo- The analytical challenge presented by this nent showed flexibility following a period project was to unpack the set of variables of intransigence. This finding adds the vari- in each stage. By situating a negotiation able of timing to the idea of firm but flexi- process in a complex setting where many ble behavior (Pruitt and Lewis 1977). Early variables operate simultaneously, it is diffi- firmness followed by later flexibility worked cult to distinguish among them in terms of best. Suggestions made by mediators had their relative impact on negotiating behav- less impact on flexibility than other factors ior. In technical terms, manipulated variables designed into the situation (e.g., media cov- within the stages are not orthogonal to each erage, alternatives). Thus, a mediator’s advice other. The design is suited to evaluate the may be a weaker lever than other aspects 7 main effects of alternative types of packages of the designed situation. It may, however, and stages, including the interaction between be the case that advice has more impact them. Thus, it was necessary to use another when combined with diagnosis and analysis, analysis strategy. Turning to an earlier liter- as shown in the next section. ature on psychological scaling, the method of pair comparisons, discussed by Guilford Electronic Mediation (1950), was appropriate. This method was adapted to the task of comparing pairs of vari- A three-part model of mediation was evalu- ables in each negotiating stage with regard ated in the context of electronic mediation. to their impact on flexibility. The judgment Referred to as negotiator assistant (NA), the took the form of does having an ideology web-based mediator implements three func- make you more or less flexible than being tions – diagnosing the negotiating situation, your nation’s primary representative? A set analyzing causes of impasses, and provid- of computations results in weights for the set ing advice to resolve the impasse (Druck- of variables in each stage and experimental man et al. 2002). It was used in conjunction condition. The weighted variables are then with a simulated negotiation that captured arranged in trajectories, showing the key fac- the issues leading to the 2003 war in Iraq. tors that operated in each negotiating stage Student role players negotiated seven issues and experimental condition by sample (scien- involving weapons inspection, border troops, tists or diplomats). and terrorism. Three experimental compar- Similar results were obtained for the sci- isons were performed to assess the impact entist and diplomat samples. They suggest of NA: compared to no mediation (experi- the conditions likely to produce flexibility or ment 1), advice only (experiment 2), and a live intransigence. Flexibility is more likely dur- mediator (experiment 3). Results showed that ing the early stages when negotiators are in the role of delegate advisors rather than 7 Stress may, however, play a larger role in real-world primary representatives for the delegation. negotiation. Results obtained from a random design field experiment conducted at the Washington, DC, They are likely to be flexible in later stages small claims court showed that contesting parties did when the talks are not exposed to media not respond to such manipulated aspects of the sit- attention and when they have unattractive uation as the configuration of furniture or orientation instructions. These findings were interpreted in alternatives. Intransigence was more likely in terms of the overwhelming effects of emotions on the early stages when they prepared strate- decisions. 424 Daniel Druckman significantly more agreements were obtained Zechmeister 1973). Various experimental in each experiment when negotiators had simulations (political decision making, prison access to NA between rounds (Druckman, reform, internal conflict resembling Cyprus, Druckman, and Arai 2004). The e-mediator ecumenical councils) were used to evalu- produced more agreements than a scripted ate some of the propositions: namely, con- live mediator despite an expressed prefer- cerning the link between values and inter- ence for the latter. These results demonstrate ests (Druckman et al. 1988) and divisions the value of electronic tools for supporting on values within negotiating teams (Jacob- complex negotiations. The study also demon- son 1981). These propositions described strates the value of embedding experiments in static relationships between variables. Other complex simulations that resemble the types propositions captured process dynamics and of real-world cases described in the opening were demonstrated with a case of failed section of this chapter. Whether these tools negotiation in the Philippines: namely, help resolve impasses in those cases remains converging and diverging values through to be evaluated.8 time (Druckman and Green 1995). The case study complemented the experiments; together, the methods provided a compre- 5. Comparing Simulations with Cases hensive assessment of the theory-derived propositions. Explicit comparisons of data obtained from More recent research on turning points simulation and cases were performed by illustrates the difference between retrospec- Hopmann and Walcott (1977) and Beriker tive and prospective analysis. A set of cases and Druckman (1996). Results obtained in was used to trace processes leading toward the former study showed that stress pro- and away from critical departures in each of duced similar dysfunctional effects in a lab- thirty-four completed negotiations on secu- oratory simulation of the partial nuclear rity, trade, and the environment (Druck- test ban talks and in the actual negotiation. man 2001). A key finding is that crises trig- Real word simulation correspondences were ger turning points. This and other findings also obtained in the latter study on power provided insight into the way that turning asymmetries in the Lausanne Peace negoti- points emerged in past cases of elite bargain- ations (1922–23). Content analyses of pro- ing. The findings were less informative with cesses recorded in transcripts and gener- regard to predicting their occurrence. Thus, ated by simulation role players showed both we designed two experiments to learn about similarities and some dissimilarities. Both the conditions for producing turning points. studies support the relevance of laboratory Both experiments showed that the social cli- experiments for understanding real-world mate (perceptions of trust and power) of the negotiations. Further support comes from negotiation moderated the effects of pre- two research streams on other negotiation cipitating factors on outcomes. The impact processes. of crises on turning points depends on the Research on the interplay between inter- climate surrounding the negotiation. The ests and values illustrates complementary experiments identified an important contin- strengths of experiments and case stud- gency in the emergence of turning points ies. The studies were intended to evalu- (Druckman, Olekalns, and Smith 2009). ate propositions derived from the literature These lines of research demonstrate the on the sociology of conflict (Druckman and value of multimethods. They highlight complementarities between experiments and case 8 Evidence for convergent validity of the NA diagnostic studies. Used together, the methods pro- function was provided by comparisons of predicted vide the dual advantages of hypothesis testing with actual outcomes obtained in nine cases. Com- and contextual interpretation, as well as the puted diagnosed outcomes corresponded to actual outcomes in eight of the nine international nego- strengths of both prospective causal analysis tiations (Druckman et al. 2002). and retrospective comparisons. Negotiation and Mediation 425

observed in laboratory experiments (Pruitt Simulations and Cases in Training and Lewis 1977), field studies of mediated Another contribution made by experiments is child custody cases (Kressel et al. 1994), to skills training for elite negotiators, includ- and both historic (Wagner 2008) and more ing diplomats and foreign service officers. recent (Hopmann 1995) cases of international The training procedures emphasize a con- negotiation. Interestingly, it also occurred in nection between research and practice. This cross-cultural bargaining experiments with is done 1) by presenting the research-based children, even when higher payoffs would knowledge in the form of narratives, and be obtained from cooperative strategies than 2) by conducting a sequence of exercises maximizing differences strategies (McClin- that are linked to the knowledge. The nar- tock and Nuttin 1969). An important ques- ratives are summaries of findings on each tion is how to change preferences from less to of sixteen themes (e.g., emotions, culture, more optimal bargaining strategies. Answers experience, flexibility). Key insights are high- are provided from experimental findings, but lighted, with special attention paid to coun- they are also relevant for negotiating in real- terintuitive findings and prescriptions for world settings. practice. For example, quick agreements are Two approaches, based on the idea of often suboptimal; thus, discourage rapid con- flexible rigidity, have been evaluated. One, cession exchanges, particularly in negotia- referred to as HTE, consists of gauging the tions between friends. other’s reactions to a variety of proposals and The exercises represent each of four nego- options. When this is done systematically, tiating roles: analyst, strategist, performer, in the form of MESOs, it is often effective. and designer. In their roles of analyst and Another, referred to as information exchange, strategist, trainees apply relevant narratives consists of asking for and providing informa- to such real work cases as Panama Canal tion about needs and values. When guided by and the Korean Joggers. In their role as per- a credible mediator, the exchange process is former, they participate in the security issues often effective, particularly when the infor- simulation described previously in the section mation revealed helps direct the talks toward on e-mediation. As designers, they construct integrative outcomes. It is also the case, how- 9 their own scenarios for training exercises. ever, that the effectiveness of both approaches The training has been conducted across four depends on maintaining a problem-solving continents and may have subtly infused exper- orientation throughout the negotiation. Sus- imental knowledge into professional negoti- tained problem solving has been important in 10 ating practices. the laboratory and in situ. Convergent findings about problem solving attest to the value of experiments as 6. Conclusion: Experiments as platforms for producing generalized knowl- Value-Added Knowledge edge. They do not, however, attest to their value in capturing context-specific knowl- A salient finding obtained across negotiating edge. Case analyses provided additional domains is that bargainers prefer to com- information about the frequency of problem pete for relative gains rather than problem solving during different stages and about the solve for joint gains. This preference was value of formulae. A question of interest is whether this sort of contextual detail would be 9 Recent findings show that designers learn more about discovered in more complex laboratory simu- negotiation concepts than classmates who role-play lations. An answer is found in the research on those designs (Druckman and Ebner 2008). Learning advantages occur as well for students exposed to the situational levers of flexibility and electronic original journal articles used for the narrative sum- mediation. maries (Druckman and Robinson 1998). The complex environmental negotiation 10 The complete training package with evaluation results is presented in Druckman and Robinson used to study situational levers provided (1998)andDruckman(2006). more specific results on staged processes than 426 Daniel Druckman other, less complex, experimental platforms. Bobrow, Davis B. 1972. “Transfer of Meaning The security issues simulation used to study across National Boundaries.” In Communication electronic mediation allowed role players to in International Politics, ed. Richard J. Merritt. experience electronic and live mediator func- Urbana: University of Illinois Press. 1971 tions. Both studies show that a balance can be Bonham, Matthew. . “Simulating Interna- struck between rigor and relevance. Further- tional Disarmament Negotiations.” Journal of Conflict Resolution 15: 299–315. more, complementary advantages of experi- Chadwick, Richard W. 1970. “A Partial Model of ments and cases were evident in the work on National Political-Economic Systems.” Journal turning points, where both retrospective and of Peace Research 7: 121–32. prospective analyses were performed, and in Conlon, Donald E., Peter Carnevale, and William the work on values and interests, where both H. Ross. 1994. “The Influence of Third hypothesis testing and holistic approaches Party Power and Suggestions on Negotia- were used. Thus, experimental knowledge tion: The Surface Value of a Compromise.” adds to our understanding of the case exam- Journal of Applied Social Psychology 24: 1084– 113 ples described at the beginning of the chap- . 1963 ter. More compelling perhaps are training Crow, Wayman J. . “A Study of Strategic applications. The gap between experiments Doctrines Using the Inter-Nation Simulation.” Journal of Conflict Resolution 7: 580–89. with students and cases with professionals is De Dreu, Carsten K. W., Laurie R. Weingart, and bridged by the use of experimental knowl- Seungwoo Kwon. 2000. “Influence of Social edge in diplomatic training programs. To the Motives on Integrative Negotiation: A Meta- extent that these programs influence the way Analytic Review and Test of Two Theories.” that diplomats negotiate, experimental find- Journal of Personality and Social Psychology 78: ings contribute directly to elite bargaining. 889–905. Deutsch, Morton. 1985. Distributive Justice: A Social-Psychological Perspective. New Haven, CT: References Yale University Press. Druckman, Daniel. 1968. “Prenegotiation Expe- Axelrod, Robert. 1980. “More Effective Choice rience and Dyadic Conflict Resolution in a in the Prisoner’s Dilemma.” Journal of Conflict Bargaining Situation.” Journal of Experimental Resolution 24: 379–403. Social Psychology 4: 367–83. Bartos, Otto J. 1995. “Modeling Distributive and Druckman, Daniel. 1973. Human Factors in Inter- Integrative Negotiations.” Annals of the Amer- national Negotiations: Social-Psychological Aspects ican Academy of Political and Social Science 542: of International Conflict. Sage Professional Paper 48–60. in International Studies, vol. 2,no.02–020. Bass, Bernard M. 1966. “Effects on the Subsequent Beverly Hills and London: Sage. Performance of Negotiators of Studying Issues Druckman, Daniel, ed. 1977. Negotiations: Social- or Planning Strategies Alone or in Groups.” Psychological Perspectives. Beverly Hills, CA: Psychological Monographs 80: 1–31. Sage. Benton, Alan, and Daniel Druckman. 1973. Druckman, Daniel. 1986. “Stages, Turning Points, “Salient Solutions and the Bargaining Behavior and Crises: Negotiating Military Base Rights: of Representatives and Non-Representatives.” Spain and the United States.” Journal of Conflict International Journal of Group Tensions 3: 28–39. Resolution 30: 327–60. Beriker, Nimet, and Daniel Druckman. 1996. Druckman, Daniel. 1993. “The Situational Levers “Simulating the Lausanne Peace Negotiations of Negotiating Flexibility.” Journal of Conflict 1922–1923: Power Asymmetries in Bargain- Resolution 37: 236–76. ing.” Simulation & Gaming 27: 162–83. Druckman, Daniel. 1994. “Determinants of Com- Blake, Robert R., and Jane S. Mouton. 1961. promising Behavior in Negotiation: A Meta- “Loyalty of Representatives to Ingroup Posi- Analysis.” Journal of Conflict Resolution 38: 507– tions during Intergroup Competition.” Sociom- 56. etry 24: 177–83. Druckman, Daniel. 1995. “Situational Levers Blake, Robert R., and Jane S. Mouton. 1962. of Position Change: Further Explorations.” “Comprehension of Points of Communality in Annals of the American Academy of Political and Competing Solutions.” Sociometry 25: 56–63. Social Science 542: 61–80. Negotiation and Mediation 427

Druckman, Daniel. 1996. “Bridging the Gap Mutual Security.” In Behavior, Society, and between Negotiating Experience and Analy- Nuclear War, Volume One, eds. Philip E. Tet- sis.” Negotiation Journal 12: 371–83. lock, Jo L. Husbands, Robert Jervis, Paul C. Druckman, Daniel. 2001. “Turning Points in Stern, and Charles Tilly. New York: Oxford International Negotiation: A Comparative University Press, 85–173. Analysis.” Journal of Conflict Resolution 45: 519– Druckman, Daniel, Mara Olekalns, and Philip L. 44. Smith. 2009. “Interpretive Filters: Social Cog- Druckman, Daniel. 2006. “Uses of a Marathon nition and the Impact of Turning Points in Exercise.” In The Negotiator’s Fieldbook: The Negotiation.” Negotiation Journal 25: 13–40. Desk Reference for the Experienced Negotiator, eds. Druckman, Daniel, and Victor Robinson. 1998. Andrea Kupfer Schneider and Christoper Hon- “From Research to Application: Utilizing eyman. Washington, DC: American Bar Asso- Research Findings in Negotiation Train- ciation, 645–56 ing Programs.” International Negotiation 3: 7– Druckman, Daniel, and Cecilia Albin. 2010.“Dis- 38. tributive Justice and the Durability of Peace Druckman, Daniel, and Kathleen Zechmeister. Agreements.” Review of International Studies. 1973. “Conflict of Interest and Value Dis- DOI:10.1017/S0260210510000549. sensus: Propositions in the Sociology of Con- Druckman, Daniel, and Thomas V. Bonoma. flict.” Human Relations 26: 449–66. 1976. “Determinants of Bargaining Behavior Etzioni, Amitai. 1967. “The Kennedy Experi- in a Bilateral Monopoly Situation II: Oppo- ment.” Western Political Quarterly 20: 361–80. nent’s Concession Rate and Similarity.” Behav- Fisher, Roger, and William Ury. 1981. Getting to ioral Science 21: 252–62. Yes: Negotiating Agreement without Giving In. Druckman, Daniel, Benjamin J. Broome, and Boston: Houghton Mifflin. Susan H. Korper. 1988. “Value Differences and Follett, Mary Parker. 1940. “Constructive Con- Conflict Resolution: Facilitation or Delink- flict.” In Dynamic Administration: The Collected ing?” Journal of Conflict Resolution 32: 489–510. Papers of Mary Parker Follett,eds.HenryC. Druckman, Daniel, and James N. Druckman. Metcalf and L. Urwick. New York: Harper. 1996. “Visibility and Negotiating Flexibility.” Fouraker, Lawrence E., and Sidney Siegel. 1963. Journal of Social Psychology 136: 117–20. Bargaining Behavior. New York: McGraw-Hill. Druckman, Daniel, James N. Druckman, and Tat- Gouldner, Alvin W. 1960. “The Norm of Reci- sushi Arai. 2004. “e-Mediation: Evaluating the procity: A Preliminary Statement.” American Impacts of an Electronic Mediator on Negoti- Sociological Review 25: 161–78. ating Behavior.” Group Decision and Negotiation Guetzkow, Harold, and Joseph J. Valadez. 1981. 13: 481–511. Simulated International Processes: Theories and Druckman, Daniel, and Noam Ebner. 2008. Research in Global Modeling. Beverly Hills, CA: “Onstage or Behind the Scenes? Relative Sage. Learning Benefits of Simulation Role-Play and Guilford, J. P. 1950. Psychometric Methods.New Design.” Simulation & Gaming 39: 465–97. York: McGraw-Hill. Druckman, Daniel, and Justin Green 1995.“Play- Hammond, Kenneth R., Frederick J. Todd, Mar- ing Two Games: Internal Negotiations in the ilyn Wilkins, and Thomas O. Mitchell. 1966. Philippines.” In Elusive Peace: Negotiating an “Cognitive Conflict between Persons: Applica- End to Civil Wars, ed. I. William Zartman. tions of the ‘Lens Model’ Paradigm.” Journal of Washington, DC: Brookings Institution Press, Experimental Social Psychology 2: 343–60. 299–331. Hamner, W. Clay, and Gary A. Yukl. 1977. Druckman, Daniel, and Richard Harris. 1990. “The Effectiveness of Different Offer Strate- “Alternative Models of Responsiveness in gies in Bargaining.” In Negotiations: Social- International Negotiation.” Journal of Conflict Psychological Perspectives, ed. Daniel Druckman. Resolution 34: 234–51. Beverly Hills, CA: Sage, 137–60. Druckman, Daniel, Richard Harris, and Bennett Hopmann, P. Terrence. 1995. “Two Paradigms Ramberg. 2002. “Computer-Assisted Interna- of Negotiation: Bargaining and Problem Solv- tional Negotiation: A Tool for Research and ing.” Annals of the American Academy of Political Practice.” Group Decision and Negotiation 11: and Social Science 542: 24–47. 231–56. Hopmann, P. Terrence, and Charles Walcott. Druckman, Daniel, and P. Terrence Hopmann. 1977. “The Impact of External Stresses and 1989. “Behavioral Aspects of Negotiations on Tensions on Negotiations.” In Negotiations: 428 Daniel Druckman

Social-Psychological Perspectives, ed. Daniel Osgood, Charles E. 1962. An Alternative to War Druckman. Beverly Hills, CA: Sage, 301–23. or Surrender. Urbana: University of Illinois Irmer, Cynthia, and Daniel Druckman. 2009. Press. “Explaining Negotiation Outcomes: Process or Pilisuk, Marc, and Paul Skolnick. 1968. “Inducing Context?” Negotiation and Conflict Management Trust: A Test of the Osgood Proposal.” Jour- Research 2: 209–35. nal of Personality and Social Psychology 8: 121– Jacobson, Dan. 1981. “Intraparty Dissensus and 33. Interparty Conflict Resolution.” Journal of Con- Pruitt, Dean G., and Steven A. Lewis. 1977. flict Resolution 25: 471–94. “The Psychology of Integrative Bargaining.” Janda, Kenneth. in press. “A Scholar and a Sim- In Negotiations: Social-Psychological Perspectives, ulation Ahead of Their Time.” Simulation & ed. Daniel Druckman. Beverly Hills, CA: Sage, Gaming. 161–92. Johnson, David W. 1967. “The Use of Role Rever- Raiffa, Howard. 1982. The Art and Science of Nego- sal in Intergroup Competition.” Journal of Per- tiation. Cambridge, MA: Harvard University sonality and Social Psychology 7: 135–42. Press. Kressel, Kenneth, Edward A. Frontera, Samuel Ramberg, Bennett. 1978. The Seabed Arms Control Forlenza, Frances Butler, and L. Fish. 1994. Negotiation: A Study of Multilateral Arms Con- “The Settlement Orientation versus the trol Conference Diplomacy. Denver: University of Problem-Solving Style in Custody Mediation.” Denver Press. Journal of Social Issues 50: 67–84. Randolph, Lillian. 1966. “A Suggested Model of Lindskold, Svenn, Pamela S. Walters, and Helen International Negotiation.” Journal of Conflict Koutsourais. 1983. “Cooperators, Competi- Resolution 10: 344–53. tors, and Response to GRIT.” Journal of Conflict Rapoport, Anatol. 1960. Fights, Games, and Debates. Resolution 27: 521–32. Ann Arbor: University of Michigan Press. Lytle, Anne L., and Shirli Kopelman. 2005. Rouhana, Nadim N. 2000. “Interactive Conflict “Friendly Threats? The Linking of Threats and Resolution: Issues in Theory, Methodology, Promises in Negotiation.” Paper presented at and Evaluation.” In International Conflict Reso- the annual meeting of the International Associ- lution after the Cold War, eds. Paul C. Stern and ation for Conflict Management, Seville, Spain. Daniel Druckman. Washington, DC: National McClintock, Charles, and J. Nuttin. 1969. “Devel- Academy Press, 294–337 opment of Competitive Behavior in Children Rubin, Jeffrey Z., and Bert R. Brown. 1975. The across Two Cultures.” Journal of Experimental Social Psychology of Bargaining and Negotiation. Social Psychology 5: 203–18. New York: Academic Press. Medvec, Victoria Husted, and Adam D. Galin- Sawyer, Jack, and Harold Guetzkow. 1965.“Bar- sky. 2005. “Putting More on the Table: How gaining and Negotiation in International Rela- Making Multiple Offers Can Increase the Final tions.” In International Behavior: A Social- Value of the Deal.” Negotiation 4: 3–5. Psychological Analysis, ed. Herbert C. Kelman. Medvec, Victoria Husted, Geoffrey L. New York: Holt. Leonardelli, Adam D. Galinsky, and Aletha Siegel, Sidney, and Lawrence E. Fouraker. 1960. Claussen-Schulz. 2005. “Choice and Achieve- Bargaining and Group Decision Making.New ment at the Bargaining Table: The Distribu- York: McGraw-Hill. tive, Integrative, and Interpersonal Advantages Singer, J. David, and Paul Ray. 1966. “Decision- of Making Multiple Equivalent Simultaneous Making in Conflict: From Interpersonal to Offers.” Paper presented at the International International Relations.” Menninger Clinic Bul- Association for Conflict Management (IACM) letin 30: 300–12. 18th Annual Conference, Seville, Spain. Summers, David A. 1968. “Conflict, Compro- Muney, Barbara F., and Morton Deutsch. 1968. mise, and Belief Change in a Decision-Making “The Effects of Role-Reversal during the Dis- Task.” Journal of Conflict Resolution 12: 215– cussion of Opposing Viewpoints.” Journal of 21. Conflict Resolution 12: 345–56. Van Kleef, Gerben A., Eric van Dijk, Wolf- Odell, John. 2000. Negotiating the World Economy. gang Steinel, Fieke Harinck, and Ilja Van Ithaca, NY: Cornell University Press. Beest. 2008. “Anger in Social Conflict: Cross- Organ, Dennis. 1971. “Some Variables Affecting Situational Comparisons and Suggestions for Boundary Role Behavior.” Sociometry 34: 524– the Future.” Group Decision and Negotiation 17: 37. 13–30. Negotiation and Mediation 429

Wagner, Lynn M. 2008. Problem-Solving and Bar- An Analysis of a Social Interaction System.New gaining in International Negotiations. Leiden, York: McGraw-Hill. The Netherlands: Martinus Nijhoff. Zartman, I. William, and Maureen R. Berman. Walton, Richard E., and Robert B. McKersie. 1982. The Practical Negotiator. New Haven, CT: 1965. A Behavioral Theory of Labor Negotiations: Yale University Press. CHAPTER 30

The Experiment and Foreign Policy Decision Making

Margaret G. Hermann and Binnur Ozkececi-Taner

In an inﬂuential monograph, Snyder, Bruck, involved, the decision-making process, and and Sapin (1954; see also Snyder et al. 2002) the decisions that are made. In effect, exper- argued that people and process matter in iments provide us with access to the tempo- international affairs and launched the study ral sequence that occurs during the decision- of foreign policy decision making. They con- making process and help us study how the tended that it is policy makers who perceive preferences policy makers bring to the pro- and interpret events and whose preferences cess shape what happens both in terms of become aggregated in the decision-making the nature of that process and the result- process that shape what governments and ing decisions. The experiment also allows institutions do in the foreign policy arena. us to compare what happens when a partic- People affect the way that foreign policy ular problem, process, or type of leader is problems are framed, the options that are absent as well as present, with there often considered, the choices that are made, and being few records of instances in govern- what gets implemented. To bolster their ment foreign policy making that provide the claims, Snyder and his associates brought controlled environment that an experiment research from cognitive, social, and organiza- does. tional psychology to the attention of scholars This tool became even more relevant to interested in world politics. They then intro- the study of foreign policy making as a result duced the experiment as a potential method- of Simon’s (1982, 1985) experiments, indi- ological tool. cating that decision making was not nec- Because it remains difﬁcult to gain access essarily rational – that the nature of peo- to policy makers and the policy-making pro- ple’s preferences matter and that rationality cess in real time, the experiment has become is bounded by how the people involved pro- a tool for simulating “history” and for doing cess information, what they want, the ways so under controlled conditions. It allows in which they represent the problem, their us to explore the causal relationships that experiences, and their beliefs. In effect, deci- occur between the nature of the people sion makers “do not have unlimited time,

430 The Experiment and Foreign Policy Decision Making 431 resources, and information” to make choices try to simulate the time constraints and real- that maximize their movement toward their life pressures under which such decision mak- goals (Chollet and Goldgeier 2002, 157). ers work. Second, the experimenters have They “satisfice,” settling for the first accept- taken seriously checking that their manipu- able option rather than pushing for ever lations work and that the simulated experi- more information and an even more optimal ence has fully engaged the so-called policy choice. “People are, at best, rational in terms makers. Some of the unexpected insights that of what they are aware of, [but] they can be these experiments have afforded us have come aware of only tiny, disjointed facets of real- because those participating as subjects have ity” (Simon 1985, 302). In effect, it becomes become caught up in the experience. important to learn about foreign policy makers’ “views of reality” as their preferences, so defined, shape their actions. Indeed, these 1. Predispositions, Preferences, early experiments showed that 1) beliefs are and Decisions “like possessions” guiding behavior and are only reluctantly relinquished (Abelson 1986); Schafer (1997) was interested in whether, in 2) the more complex and important (the more conflict situations, policy makers’ images of lifelike and ill structured) a problem, the less their own and the so-called enemy influenced decision makers act like Bayesian information their preferences regarding the other country processors (Alker and Hermann 1971); and 3) and their choice of strategy as well as their prior knowledge about a problem appears to resulting foreign policy choices. Do policy shape cognition and focus decision making makers’ worldviews matter? By choosing to (Sylvan and Voss 1998). How decision mak- use the experiment to explore this question, ers define and represent problems may or may Schafer was able to ensure that policy makers’ not match how an outside, objective observer images temporally preceded his assessment of views them. policy preferences and resulting decisions – In the rest of this chapter, we examine that he could talk about causation and not in some detail a set of experiments that just correlation. have built on what Snyder and his colleagues Schafer (1997) used a 2 × 2 factorial de- and Simon discovered. We focus on three sign. His two independent variables were 1) important questions in which the experi- the historical relationship between the coun- ment as a methodology is and has been par- tries involved in the conflict – friendly/ ticularly useful in helping us gain insights cooperative versus unfriendly/conflictual, into foreign policy decision making: 1) how and 2) the cultural similarity between the do policy makers’ predispositions shape pol- countries – similar versus different. The “policy preferences and behavior? 2) how does icy makers” in the experiment were seventy- the way in which a problem is framed six college students enrolled in an interna- influence which options are viewed as rele- tional relations course who were randomly vant? and 3) how do individual preferences assigned to the four treatment groups. Each become aggregated in the decisions of gov- was given briefing material concerning an ernments – that is, whose positions count and international conflict between their fictitious why? country and another country. Both countries It is important that the reader keep two had similar military resources, and the con- caveats in mind as we describe experiments flict held the possibility for dire consequences that address these questions. First, to achieve for each. The only differences among the some semblance of experimental realism in briefing materials were the alternative views these experiments, the problems and scenar- of the other country – whether their histor- ios that are used are patterned after historical relationship had been generally friendly ical events with an attempt to have subjects or unfriendly and whether their cultures were face similar situations to those of actual for- similar or different. Participants were told eign policy makers. Some researchers even that they were valued advisors to the president 432 Margaret G. Hermann and Binnur Ozkececi-Taner of their country and that their recommenda- problem; can it predispose them to make dif- tions were important to the decision-making ferent decisions than when such experience is process. After reading the briefing materi- lacking? Consider, for example, that the last als, they were asked to indicate their current three American presidents – Clinton, Bush, attitudes toward the other country; the gen- and Obama – have had little experience in eral strategy that they believed their coun- dealing with foreign policy before coming try should follow toward that other country; to office. They have had less background on and the diplomatic, economic, and military which to draw in the decision-making process responses their country should take given the when compared to presidents such as Eisen- situation. hower or George H. W. Bush, who spent Checks on the manipulation to make sure their careers leading up to the White House that those involved picked up on the two dif- dealing with foreign policy problems. Mintz ferent image dimensions that differentiated (2004) sought to explore this issue and to do their country from the other one revealed so with persons actually involved in the for- that they did. The results – using analysis of eign policy arena, that is, military officers in variance – indicated that both differences in the U.S. Air Force. By using such individu- historical relationship and in culture led to als as participants in his experiment, he could parallel types of attitudes toward the “other.” be assured that they had some prior experi- Those with the supposed negative historical ence in making policy decisions and that his relationship tended to view the other country experiment worked toward achieving experi- as an enemy and were predisposed toward mental realism and external as well as internal hostile and conflictual strategies in dealing validity. with the other. Similarly, those where the Seventy-two military officers who were opposing country was different in culture part of the command and instructional staff of from them also viewed the other as an enemy the U.S. Air Force Academy were randomly and believed that conflict was the best strategy assigned to deal with either a familiar or unfa- to choose. When it came to the participants’ miliar problem. Those in the experimental decisions, there was a significant interaction condition meant to involve a familiar problem between historical relationship and similarity were presented with a scenario asking them of culture. Those in the condition in which to deal with a military dispute that erupted participants had both a negative history and between two small countries over control of a a different culture from the other country large uranium field, leading one to invade the with which they were in conflict made other and hold foreign citizens hostage. Four decisions that were much more conflictual alternatives were posed to those in this condi- than were made in any other condition – the tion; they could use force, engage in contain- conflict was exacerbated. In effect, cultural ment, impose sanctions, or do nothing. For differences mattered more in the policy the unfamiliar problem, participants were choices made between perceived enemies faced with choosing the site for a new naval than between perceived friends. Only, appar- base in the Pacific; four islands unknown to ently, when the relationship with another those involved were nominated for such a country is perceived to be both historically site, and the four islands became the alterna- and currently unfriendly and negative do tives under consideration. The officers made cultural differences become important in their decisions using a computerized decision decision making – the other country has to board that provided information regarding be viewed as both an enemy and “not like the various alternatives as well as recorded us” for attitudes to shape the choices that are the nature of their search for this information made. Schafer’s (1997) experiment suggests and the strategies they used. The information that images matter under certain circum- that was presented indicated the alternative’s stances. likely political, economic, military, and What if policy makers have experience or diplomatic impacts – positive or negative – if expertise in dealing with a particular type of selected. The Experiment and Foreign Policy Decision Making 433

Manipulation checks indicated that the to decide to do given their opponent’s inva- officers dealing with the unfamiliar problem sion of the islands; they chose among fifteen searched for considerably more information different alternatives ranging from a peace- than those faced with the familiar problem. ful attempt to resolve the crisis to an all- Indeed, the officers involved with the familiar out military retaliation. This decision was problem latched onto a preferred alternative then followed by another regarding what each almost immediately and explored its impli- thought the country representing Argentina cations before checking out any of the other would do given “Britain’s” response. alternatives. They took a shortcut based on Beer et al. (2004) report a significant inter- their experience. Those officers dealing with action effect between the conflict-focused the unfamiliar problem, however, kept seek- prime (introductory material) and the per- ing information regarding the implications of sonality of the subjects with regard to the type all four alternatives on political, economic, of decision that was made. Participants high military, and diplomatic efforts, comparing in dominance were more likely to choose to and contrasting the effects of each alternative engage in conflict when primed to think about as they considered each domain. The offi- crisis and war, regardless of whether the focus cers used different decision-making strate- was costs or appeasement; when not primed, gies in dealing with familiar and unfamiliar there was no difference between those high problems, their predispositions having more and low in dominance. The priming moti- impact on preferences and choices with the vated the more dominant participants to want familiar problem. to take charge. In the Schafer (1997) and Mintz (2004) As can occur in experiments, Beer and experiments just described, the predisposi- his colleagues (2004) also discovered an tions of the policy makers were primed by the unexpected result that helps us link the experimental conditions. Is there any way to Schafer (1997) and Mintz (2004) experiments examine predispositions that do, indeed, pre- described previously and provides an impor- cede participation in the experiment? Beer, tant insight regarding the possible inter- Healy, and Bourne (2004) were interested action between predispositions, preferences, in exploring this question and used a pre- and choices in foreign policy decision mak- test to assess the levels of dominance and ing. Forty-five percent of the participants cor- submissiveness of their experimental subjects rectly recognized the scenario when asked at using scales from the sixteen personality fac- the end of the experiment if it reminded them tor questionnaire (Institute for Personality of any recent actual event. There was little and Ability Testing 1979). The subjects, who difference between the decisions of subjects were students in an introduction to psychol- high and low in dominance for those who ogy course, were then randomly assigned to correctly identified the event in the scenario – one of three experimental conditions where they based their selection of an alternative on they viewed 1) material reminding them of their perceptions of the historical scenario. the terrible costs of war through statistics But those high in dominance chose signif- and graphs from World War I, 2) material icantly more conflictual actions when they concerning the way in which the appease- did not recognize the event. Prior knowledge ment of aggression can lead to war with was used when the situation seemed famil- a focus on the Munich conference and its iar, whereas personality shaped decision mak- aftermath, or 3) no introductory material. ing without such prior knowledge. In other Following the priming conditions, all three words, personal predispositions are impor- groups read a common scenario with fic- tant in decision making unless the policy tionalized countries modeled after the Falk- maker has prior experience or expertise – lands/Malvinas Islands crisis between Britain in such circumstances, experience seems to and Argentina in 1982. After reading the sce- take precedence in shaping preferences as nario, subjects were asked to indicate what well as the generation of alternatives and the country representing Britain was likely choices. 434 Margaret G. Hermann and Binnur Ozkececi-Taner

2. Frames and Expectations player coming through defection. Of interest was how the Wall Street and Commu- An important part of the decision-making nity Organization frames would shape expec- process is the identification and framing of tations regarding underlying norms and rules problems. Indeed, this phase is often where in the early rounds of play. people play a key role in shaping what will Sure enough, the frames had a significant occur. Consider, for example, how the same effect on play. Two thirds of those believ- event – September 11, 2001 –wasiden- ing that they were playing the Community tified and framed differently by leaders in Organization game cooperated in the first Britain and in the United States. Tony Blair round of play, whereas only one third of those announced at the Labour Party Conference perceiving themselves to be involved in the just hours after the Twin Towers collapsed Wall Street game cooperated. Indeed, this that we had just experienced a “crime against striking difference only increased over time civilization” – police and the courts were as the Community Organization game facil- the appropriate instruments for dealing with itated more cooperation and the Wall Street what had happened and justice was the value game more defections. In effect, the frames at stake. George W. Bush framed the event as overrode the students’ ideas about their own “an attack against America” and pronounced predispositions to cooperate or to defect in a a war on terror engaging the military and call- game such as the prisoner’s dilemma. As the ing forth nationalism. These frames focused authors note, who gets to frame a problem the attention of the respective policy-making gains influence over the policy-making pro- communities and shaped expectations regard- cess. ing who would be involved and the nature of Building on the large body of nonexperi- the options that were available. Experiments mental research suggesting that “democracies have been usefully used to demonstrate the do not fight one another” (for an overview, importance of frames in defining foreign pol- see Chan 1997), Mintz and Geva (1993) icy problems as well as their influence on designed an experiment to explore the effects shaping the options considered and result- of how a country is framed – as a democracy ing decisions. Three illustrate the effects of or a nondemocracy – on decisions regarding frames. the use of force. Does being told that a coun- The first is an experiment intended to try is one or the other almost automatically show how relatively easy it is for frames to shape responses where the use of force is con- change the way that a policy maker looks at cerned? To enhance external validity, Mintz a situation. Ross and Ward (1995) explored and Geva ran the experiment three times – the influence of a frame on students who once with U.S. college students, once with were recruited to play the prisoner’s dilemma U.S. adults who were not college students game – urged on by their dorm leaders who but members of community organizations, selected them because of the dorm leaders’ and once with Israeli college students who beliefs that these students represented good had been or were members of their coun- examples of either “defectors” or “coopera- try’s military. Participants were randomly tors.” The recruits were randomly assigned assigned to one of two conditions – one in to play either a Wall Street or a Community which the adversary in a hypothetical cri- Organization game. Those playing each game sis was described as democratic and one in were given instructions indicating the nature which the adversary was described as non- of the game – either that they were members democratic. All read a scenario that was mod- of a Wall Street trading firm or that they were eled after what happened in the first Persian part of a community not-for-profit organiza- Gulf War and involved one nation (the adver- tion. Subjects then made decisions based on a sary) invading another as a result of a con- prisoner’s dilemma matrix of payoffs with the flict over uranium and, in the process, tak- largest payoff for both players resulting from ing hostages. In the course of the scenario, cooperation, but the largest payoff for a single the adversary was framed as a democracy The Experiment and Foreign Policy Decision Making 435 or a nondemocracy. Participants were faced ing gains or losses. Decision makers appear with three alternatives regarding the nature to be more sensitive to discrepancies that of their country’s response: use force, set up are closer to their reference point than to a blockade, or do nothing militarily. those further away and, perhaps more impor- In all three runs of the experiment, par- tant, to view that “losing hurts more than a ticipants picked up on the frame for the con- comparable gain pleases” (McDermott 2001, dition to which they were assigned. Indeed, 29). As Foyle (1999) observes about American they viewed the adversary as more dissimilar presidents, their foreign policy choices are from their own country when it was the non- much more affected by constituents and con- democracy than the democracy. Moreover, text when they fear losing the public’s sup- in each of the runs the participants signif- port for their policies or their administration. icantly approved the use of force against a In a similar vein, Tomz (2009) talks about nondemocracy more than against a democ- this phenomenon in terms of audience costs racy. This significant difference only held, and has found that British members of parlia- however, for the use of force; for the block- ment who are involved in the foreign policy- ade and “do nothing” alternatives, the frame making process provide evidence of anticipat- was not determinative. To complement their ing such costs as part of their decision making. hesitation regarding the use of force against a Generally, the experiments conducted to democracy, the participants perceived such a document prospect theory and the effects of use of force as a foreign policy failure. The framing involve posing a choice dilemma to frame brought with it limitations on who subjects with the options under consideration could become a target of military force as framed either, for example, in terms of how well as how choosing to use force would be many people will die or in terms of how many evaluated. will live. This difference in framing leads to Probably the most influential experiments taking the risky option when the focus is on on framing in the literature on foreign pol- how many people will die and the more con- icy decision making are those conducted servative option when the frame focuses on by Kahneman and Tversky, which have the people who will live. Kowert and Her- evolved into what is known as prospect theory mann (1997) wondered how generalizable the (Tversky and Kahneman 1992; Kahneman effects of loss and gain frames were across a and Tversky 2000; see also Farnham 1994; variety of different types of problems. What McDermott 2001). In essence, their findings happens, as is often the case in foreign pol- indicate that how individuals frame a situa- icy decision making, when problems cross tion shapes the nature of the decision they domains? In an experimental setting, students are likely to make. If policy makers perceive in introductory political science courses were themselves in a domain of gains (things are faced with choice dilemmas focused around going well), then they are likely to be risk economic issues, political problems, or health averse. However, if their frame puts them in concerns. the domain of losses (things are going poorly), Interestingly, the results indicated that the then they are likely to be more risk prone or framing effect postulated by prospect theory risk seeking. Decisions depend on how the held for political and health problems but not policy maker frames the problem. Critical for for economic issues. Indeed, three fourths of determining whether a decision maker finds the subjects involved were risk averse when him- or herself in the domain of losses or political and health problems were framed gains is the individual’s reference point or as a gain, and two thirds were risk acceptant definition of the status quo. Problems arise for similar types of concerns when they were when decision makers face situations where framed in terms of loss. However, roughly there is a discrepancy between what is occur- two thirds of the subjects were risk averse in ring and their reference point. The direction the economic domain regardless of the frame. of the discrepancy indicates whether the deci- They tended to seek out the certain option, sion maker interprets the situation as involv- even though a little less so when the problem 436 Margaret G. Hermann and Binnur Ozkececi-Taner was framed in the domain of losses as opposed leader – to that of a group or a coalition. to gains (sixty-three percent to seventy-eight Here, too, the experiment has proven to be percent, respectively). Given that the stu- a useful tool in helping us understand the dents participating in the study were enrolled foreign policy-making process. It has pro- in a public university, many working full- vided us with insights, but with one caveat. time jobs to afford an education, economic The laboratory experiments to be reviewed choice dilemmas may have been more real here have generally focused on ad hoc groups, to them than those dealing with politics or bringing participants together at one point in health. This result suggests the importance time, rather than on ongoing groups. More of determining the reference point in con- often than not, aggregations of individuals in sidering the nature of the frame – for these the foreign policy-making process expect to particular students, their reference point for interact across time. economic concerns could have immediately One set of experiments on the rules of put them in the domain of gains – “I have aggregation operating in groups has exam- things under control” and “let’s not rock the ined the roles that the members of the group boat.” Of course, it could also be the case play – in this case, whether those involved in that prospect theory has more of an effect for the group are leaders and able to make deci- some kinds of problems than for others. sions in the group itself, such as occurs at summits, or delegates who must check back with those they represent in the decision- 3. Aggregating Individual making process – and the types of decisions Preferences into Government and decision process that are likely to result Decisions (e.g., Hermann and Kogan 1968; Myers and Lamm 1976; Semmel 1982; Isenberg 1986; The studies we just described focused on indi- Brauer and Judd 1996). To reinforce the viduals and how they make decisions. But assigned roles in these experiments, students governments are not single individuals, nor are generally randomly divided into groups do they act as a unit. Indeed, consider the depending on status either defined by some wide array of entities that are responsible difference or stipulated in the instructions. for making foreign policy, for example, party For example, in one case, upper classmen standing committees, military juntas, leader were designated as the leaders and underclass- and advisors, cabinets, interagency groups, men the delegates, and each experienced the parliamentary committees, and loosely struc- other’s status in an initial meeting where the tured revolutionary coalitions. The individ- leader’s position could prevail if there was dis- uals comprising these entities do not always agreement. In these experiments, leaders and agree about what should happen with regard delegates usually met in pairs to start with, to foreign policy. In fact, an examination of coming to some agreement as to what their a range of foreign policy decisions (Beasley joint positions were with regard to either a et al. 2001) indicated that around seventy- scenario or a set of dilemmas. The leaders five percent of the time, those involved dis- and delegates were then formed into sepa- agree about the nature of the problem, the rate groups composed of either all leaders or options that are feasible, or what should hap- all delegates to again wrestle with the prob- pen. As Eulau (1968) argued so long ago, lems. Following this interaction, each par- “although concrete political action is invari- ticipant was asked to indicate whether the ably the behavior of individual human actors, deliberations with the others playing a simi- the politically significant units of action are lar role had any effect on their original dyadic groups, associations, organizations, institu- decisions. tions, coalitions, and other types of collec- Having been told that they would be tivities” (209). Of interest are the rules of checking back with their leaders later, dele- aggregation for moving from decision mak- gates were more likely to focus on achieving a ing at the individual level – say, that of a compromise so that all could gain something The Experiment and Foreign Policy Decision Making 437 and not lose everything. Leaders without the who was interested in understanding social same need to report back were more likely to change rather than maintenance of the status engage in extensive argument and debate, to quo, sought to understand when minorities choose one of their members’ positions, and could have an influence (see also Wood 2000; to show the greatest change in their positions Kaarbo 2008). He became even more intent from where they started. Indeed, the delegate on his enterprise when he found that eight groups were more satisfied generally with the percent of the majorities in his experimental intergroup process when they compromised. groups went along with the minority when he For the leader groups, decision making with had a confederate minority declare that the other leaders took precedence over choices blue color slide that everyone was seeing was made in their dyadic sessions with their dele- actually green – and, as he noted, they went gates. But for the delegates, the dyadic deci- along even though the majority was twice sion making remained important when they the size of the minority. Moscovici’s (1976) met with other delegates. experimental research, and that inspired by The changes in position found among both him, has led to the postulation of the “dual the leader and delegate groups were at first process” model of social influence that indi- believed to indicate the diffusion of responsi- cates there is a difference in the influence bility that making decisions in a group affords process when engaged in by a majority or a – change is feasible because accountability for minority. Specifically, when the majority is the decision can be spread and shared with doing the influencing, the conflict is social little loss of face (e.g., Kogan and Wallach and the minority only changes its position 1967; Vertzberger 1997). Such behavior has publicly, whereas when the minority is doing since been viewed as the result of persuasion; the influencing, the conflict is cognitive, with group members “with more radically polar- the members of the majority “trying to com- ized judgments and preferences invest more prehend the deviant position” and in the pro- resources in attempts to exert influence and cess of such reconsideration changing their lead others” (284), and, as a result, the discus- original position, even though their conversion includes “persuasive arguments that the sions are generally delayed and not done pub- typical subject has not previously considered” licly (De Vries and De Dreu 2001, 3). (Myers and Lamm 1976, 611). Both explana- Minorities in groups who are committed tions seem appropriate to the findings with to their positions and resolved that their plan regard to leaders and delegates. The diffusion or option is best generally cause the group of responsibility notion, in effect, provides to revisit the problem of concern and, if con- cover for the groups composed of delegates, sistent and persistent, can influence the posi- whereas persuasion appears to be a more tion of the majority. Minorities can also use appropriate explanation for the behavior of procedural manipulations to facilitate making the groups comprised of leaders. Indeed, the the group more conducive to their positions delegate groups chose one party’s position – say, by pushing for a change from majority only when one of the delegates was so highly rule to consensus or unanimity, by support- committed to his or her position that the ing incremental adoption of policies, or by group risked deadlock and, as a result, return- reframing the problem (e.g., De Dreu and ing to the leader with no decision at all. Beersma 2001; Kaarbo 2008). Consider an Generally, those experimenting with experiment by Kameda and Sugimori (1993). groups have viewed influence among mem- These experimenters were interested in bers as going one way – usually the major- studying how the decision rule guiding dis- ity overpowering any minority. Groupthink cussion and policy making in groups affected (Janis 1982) is one example of such influence, the nature of the process and choice, as well with stress leading to concurrence seeking as the impact of group composition – when because members want to remain part of the opinion was split versus not split – on decision group (see also Brown 2000; Hermann et al. making. Increasingly more often in foreign 2001; Garrison 2003). But Moscovici (1976), policy making, decisions are being made with 438 Margaret G. Hermann and Binnur Ozkececi-Taner a consensus orientation or unanimity deci- groups in the experiment. And members of sion rule. Arguments are made that this rule these groups were the least likely to view the facilitates reaching the most feasible solution alternative options the groups had in a posi- possible to a problem at the moment, and tive light – it was almost as if members said one with which those involved are likely to “the other options we have to consider are no comply and implement (Hagan et al. 2001; better than what we are doing and may even Hermann et al. 2001). By way of contrast, a lead to worse outcomes.” But, interestingly, majority decision rule is viewed as likely to the members in the minority evaluated the create a minority ready and willing to con- situation more negatively when they were in tinue to push the majority to revisit any deci- the majority rule condition than in the una- sion if it starts to go bad (Beasley et al. 2001). nimity condition. Being unable to overturn a Kameda and Sugimori (1993) involved 159 vote and being in the minority when decisions Japanese students in a 2 × 2 between-subjects can be decided by majority rule led those in factorial design (majority vs. unanimity deci- such positions to dislike the experience and sion rule; opinion in group split vs. not to be dissatisfied with the decision that was split). Subjects were presented with a simu- made. lated decision task and asked to make an ini- Groups governed by a unanimity decision tial decision indicating their preferences for rule appeared to move to maintain the sta- what should be done. They were randomly tus quo more than those with a majority rule. assigned to groups so that one set of three- Moreover, such groups were even more con- person groups had no differences among their strained when they had majority and minority initial preferences, whereas the other set of views represented among the members of the three-person groups had two persons favor- group. The easiest way to reach consensus ing one option and a third person with a dif- was to stick to the original policy. The way to ferent opinion. The groups were then asked move to have policy makers consider chang- to discuss the problem and reach a decision ing policy appears to be through constituting based on the decision rule governing their groups with a majority decision rule and some process. Each group was told that they had split in the opinion among group members, sole responsibility for the decision. The deci- the latter promoting discussion and breaking sion in the simulated task focused on whether immediate consensus. But when the minor- the groups should continue to support a pol- ity in such groups is overruled, they are often icy – or, in this case, a person – that was dissatisfied with what has occurred and can not performing as desired. Would the groups force the group to revisit the issue again if the focus on maintaining the original consensus new choice is not effective (for an application or move to change policy? of these ideas to a set of foreign policy cases, The results indicated an interaction see Beasley et al. 2001). Decision rules appear between the nature of the decision rule and to differentially affect group process and, in cohesion around the choice of policy. The turn, the nature of the decisions made. groups that had a member with a minor- An important type of group involved in the ity position who were guided by a majority foreign policy-making process involves lead- rule were the most likely to move away from ers and their advisors. Indeed, all leaders have the previous consensus and change policy, advisors. We have discovered through case whereas that same type of group functioning studies that it appears leaders pay more atten- under a unanimity decision rule was the least tion to some of their advisors than to others, likely to change positions and, thus, the most often depending on how they have structured likely to continue the original consensus pol- the setting and their leadership style (e.g., icy regardless of negative feedback. Indeed, Preston 2001; Mitchell 2005). Redd (2002) the groups with a member with a minority sought to explore the influence of advisors position who were guided by a unanimity rule where he could have control over the rela- were the most likely to persist in the pres- tionship of the advisors to the leader and ence of negative feedback of all four types of chose to do so using an experiment. He was The Experiment and Foreign Policy Decision Making 439 interested in “how and in what manner lead- native” (Redd 2002, 355). Such was partic- ers obtain information from advisors” and the ularly the case when the political advisor’s effect on policy (343). To study this question, recommendation came early in the process. he manipulated the status of leaders’ advi- However, when the advisors were viewed as sors. What happens when one – in this case, equal in importance, the participants were the political advisor – has a higher status or forced to consider the recommendations of importance in the leader’s eyes than the oth- each advisor for each alternative. There was ers versus when all are of relatively equal sta- no shortcut or person to sensitize them for tus? Importance of advisors was manipulated or against a specific option, and, as a result, in two ways: 1) the participants acting as lead- they were more likely to choose the alterna- ers were randomly assigned to a condition in tive with the highest utility. Trusted advisors which the advisors were unequal in status ver- appear to have short-circuited their leaders’ sus one where they were equal, and 2)the search for information by shaping the way order in which the advisors’ positions were in which the problem was defined and the presented to the participants (political advi- options that were considered. A wider search sor information being presented first rather for information was necessary when advisors than last). were perceived as rather equal in expertise Subjects were undergraduates taking polit- and status. ical science courses who were presented with a foreign policy scenario involving “a military dispute between two small countries that 4. In Conclusion erupted over control of a large uranium field” and resulted in one of these nations invading The illustrative experiments described and the other and taking foreign citizens hostage discussed here represent how this method- (Redd 2002, 345). Participants used a decision ology is being and can be used to study for- board in making their decisions, facilitating eign policy decision making. The experiment tracing their decision-making process. The provides us with a venue to examine pro- decision board contained four advisors’ opin- cesses involved in the making of policy as ions on four options. The four types of advi- well as the linkages between these processes sors focused on political, economic, diplo- and the resulting choices. It enables us to matic, or military issues. The alternatives that explore the temporal sequence of “what leads the participants had to choose among were to to and shapes what” in a way that case studies do nothing, engage in containment, impose and observational studies do not. In effect, it international sanctions, or use force. In the provides a controlled environment in which condition in which advisors were not equal, to engage in process tracing – to follow the participants were told that their chief of staff way in which the decision process evolves. or political advisor gave them the best advice Moreover, the experiment allows us to com- and the most successful policy recommenda- pare and contrast the presence and absence of tions. In the condition where advisors were the phenomenon under examination and not equal, participants were told that their advi- just to focus on the situations where it was sors gave equally good advice and recommen- present, which is more generally the case in dations. the literature. In effect, without the experi- The results of the experiment showed that ment, we know little about what would have when the political advisor was viewed as giv- occurred if the phenomenon under study had ing the most relevant advice, participants not occurred. Is what actually happened any tended to focus on the nature of that advi- different, and, if so, how? sor’s position and to use it as a guide for what Yes, there are problems with experiments, other information was sought. And “decision as there are with any methodology. Most makers tended not to select the alternative of the experiments described above involved that the political advisor evaluated negatively, college students who are not currently policy regardless of the overall utility of that alter- makers – although a number of the studies 440 Margaret G. Hermann and Binnur Ozkececi-Taner did try to replicate or do their research with Because of the difficulty of gaining access subjects who were older than college stu- to the people and processes involved in for- dents or actually engaged in certain types eign policy making, even in the United States, of policy making (e.g., Air Force officers). studying the political aspects of such policy And the groups whose decision making was making has lagged institutional and structural examined were set up in the laboratory and analyses of why governments do what they do were not going to last longer than the exper- in international affairs. The experiment and iment, even though an important factor in these new variants of it offer those of us inter- ongoing policy-making groups is the “shadow ested in understanding foreign policy mak- of the future” phenomenon, where groups ing a way to explore how people and process have some sense that they will continue to matter. interact in the future and, thus, temper their behavior with this expectation in mind. But even here, there were attempts to bring some References flavor of the ongoing group into the laboratory, for instance, by having leader and dele- Abelson, Robert P. 1986. “Beliefs Are Like Posses- gate groups composed of upper- and lower- sions.” Journal for the Theory of Social Behavior level students. Moreover, there are always 16: 223–50. 1971 questions regarding experimental realism; for Alker, Henry A., and Margaret G. Hermann. . instance, policy makers face more than one “Are Bayesian Decisions Artificially Intelligent: problem at a time, they are often not respon- The Effect of Task and Personality on Conser- vatism in Processing Information.” Journal of sible for identifying or framing a problem, Personality and Social Psychology 19: 31–41. and they must delegate decisions to others to Beasley, Ryan, Juliet Kaarbo, Charles F. Hermann, implement. and Margaret G. Hermann. 2001. “People and Nevertheless, the three experiments exam- Processes in Foreign Policymaking.” Interna- ined in each section in this chapter focus on tional Studies Review 3: 217–50. questions that have been difficult to explore Beer, Francis A., Alice F. Healy, and Lyle E. other than through experiments. They have Bourne, Jr. 2004. “Dynamic Decisions: Exper- provided us with new insights into how for- imental Reactions to War, Peace, and Ter- eign policy decisions are made and with some rorism.” In Advances in Political Psychology,ed. Margaret G. Hermann. London: Elsevier, 139– intriguing ideas that can be explored fur- 67 ther in case and observational studies. They . Brauer, Markus, and Charles M. Judd. 1996. have also laid the groundwork for scholars “Group Polarization and Repeated Attitude in the field to work on introducing more of Expressions: A New Take on an Old Topic.” the experimental method into their case and European Review of Social Psychology 7: 173–207. observational studies. Consider, for exam- Brown, Rupert. 2000. Group Processes. Oxford: ple, experiments embedded in surveys where Blackwell. manipulations and control conditions are ran- Chan, Steve. 1997. “In Search of Democratic domly presented to those who are interviewed Peace: Problems and Promise.” Mershon Inter- (e.g., Herrmann, Tetlock, and Visser 1999; national Studies Review 41: 59–91. 2002 Tomz 2007). And with case studies, there Chollet, Derek H., and James M. Goldgeier. . is growing interest in the use of structured, “The Scholarship of Decision Making: Do focused comparison where the same ques- We Know How We Decide?” In Foreign Pol- icy Decision-Making Revisited, eds. Richard C. tions are systematically asked of all cases and Snyder, H. W. Bruck, Burton Sapin, Valerie the cases that are selected to study do and M. Hudson, Derek H. Chollet, and James M. do not exhibit the phenomenon being exam- Goldgeier. New York: Palgrave Macmillan, ined (e.g., Kaarbo and Beasley 1999; Mitchell 153–80. 2005). Such techniques are intended to bring De Dreu, Carsten K. W., and Bianca Beersma. some internal validity into the field while 2001. “Minority Influence in Organizations.” at the same time facilitating experimental In Group Consensus and Minority Influence: Impli- realism. cations for Innovation,eds.CarstenK.W.De The Experiment and Foreign Policy Decision Making 441

Dreu and Nanne K. De Vries. Oxford: Black- Kaarbo, Juliet, and Ryan K. Beasley. 1999.“A well, 258–83. Practical Guide to the Comparative Case Study De Vries, Nanne K., and Carsten K. W. De Method in Political Psychology.” Political Psy- Dreu. 2001. “Group Consensus and Minor- chology 20: 369–91. ity Influence: Introduction and Overview.” In Kahneman, Daniel, and Amos Tversky. 2000. Group Consensus and Minority Influence: Implica- Choices, Values, and Frames. Cambridge: Cam- tions for Innovation, eds. Carsten K. W. De Dreu bridge University Press. and Nanne K. De Vries. Oxford: Blackwell, Kameda, Tatsuya, and Shinkichi Sugimori. 1993. 1–14. “Psychological Entrapment in Group Deci- Eulau, Heinz. 1968. “Political Behavior.” In Inter- sion Making: An Assigned Decision Rule and national Encyclopedia of the Social Sciences. vol. a Groupthink Phenomenon.” Journal of Person- 12, ed. David L. Sills. New York: Macmillan, ality and Social Psychology 65: 282–92. 203–14. Kogan, Nathan, and Michael A. Wallach. 1967. Farnham, Barbara, ed. 1994. Avoiding Losses/ “Risk Taking as a Function of the Situation, Taking Risks: Prospect Theory in International the Person, and the Group.” In New Dimensions Politics. Ann Arbor: University of Michigan in Psychology III, eds. Michael A. Wallach and Press. Nathan Kogan. New York: Holt, Rinehart, and Foyle, Douglas C. 1999. Counting the Public In: Winston, 111–278. Presidents, Public Opinion, and Foreign Policy. Kowert, Paul A., and Margaret G. Hermann. 1997. New York: Columbia University Press. “Who Takes Risks? Daring and Caution in Garrison, Jean A. 2003. “Foreign Policymaking Foreign Policy Making.” Journal of Conflict Res- and Group Dynamics: Where We’ve Been olution 41: 611–37. and Where We’re Going.” International Studies McDermott, Rose. 2001. Risk-Taking in Interna- Review 6: 177–83. tional Politics: Prospect Theory in American For- Hagan, Joe D., Philip P. Everts, Haruhiro Fukui, eign Policy. Ann Arbor: University of Michigan andJohnD.Stempel.2001. “Foreign Pol- Press. icy by Coalition: Deadlock, Compromise, and Mintz, Alex. 2004. “Foreign Policy Decision Mak- Anarchy.” International Studies Review 3: 169– ing in Familiar and Unfamiliar Settings.” Jour- 216. nal of Conflict Resolution 48: 49–62. Hermann, Charles F., Janice Gross Stein, Bengt Mintz, Alex, and Nehemia Geva. 1993. “Why Sundelius, and Stephen G. Walker. 2001. Don’t Democracies Fight Each Other? An “Resolve, Accept, or Avoid: Effects of Group Experimental Study.” Journal of Conflict Reso- Conflict on Foreign Policy Decisions.” Inter- lution 37: 484–503. national Studies Review 3: 133–68. Mitchell, David. 2005. Making Foreign Policy: Pres- Hermann, Margaret G., and Nathan Kogan. 1968. idential Management of the Decision-Making Pro- “Negotiation in Leader and Delegate Groups.” cess. Burlington, VT: Ashgate. Journal of Conflict Resolution 12: 332–44. Moscovici, Serge. 1976. Social Influence and Social Herrmann, Richard K., Philip E. Tetlock, and Change. Trans. by Carol Sherrard and Greta Penny S. Visser. 1999. “Mass Public Deci- Heinz. London: Academic Press. sions to Go to War: A Cognitive-Interactionist Myers, David G., and Helmut Lamm. 1976.“The Framework.” American Political Science Review Group Polarization Phenomenon.” Psychologi- 93: 553–73. cal Bulletin 83: 602–27. Institute for Personality and Ability Testing. 1979. Preston, Thomas. 2001. The President and His Inner Administrative Manual for the 16PF. Cham- Circle: Leadership Style and the Advisory Process in paign, IL: Institute for Personality and Ability Foreign Affairs. New York: Columbia Univer- Testing. sity Press. Isenberg, Daniel J. 1986. “Group Polarization: A Redd, Steven B. 2002. “The Influence of Advisers Critical Review and Meta-Analysis.” Journal of on Foreign Policy Decision Making: An Exper- Personality and Social Psychology 50: 1141–51. imental Study.” Journal of Conflict Resolution 46: Janis, Irving L. 1982. Groupthink: Psychological 335–64. Studies of Policy Decisions and Fiascoes. 2nd ed. Ross, Lee, and Andrew Ward. 1995. “Psychologi- Boston: Houghton Mifflin. cal Barriers to Dispute Resolution.” In Advances Kaarbo, Juliet. 2008. “The Social Psychology of in Experimental Social Psychology. vol. 27,ed.M. Coalition Politics.” International Studies Review P. Zanna. San Diego: Academic Press, 255– 10: 57–86. 304. 442 Margaret G. Hermann and Binnur Ozkececi-Taner

Schafer, Mark. 1997. “Images and Policy Prefer- sion Making. Cambridge: Cambridge Univer- ences. Political Psychology 18: 813–29. sity Press. Semmel, Andrew K. 1982. “Small Group Dynam- Tomz, Michael. 2007. “Domestic Audience Costs ics in Foreign Policymaking.” In Biopolitics, in International Relations: An Experimental Political Psychology, and International Politics,ed. Approach.” International Organization 61: 821– Gerald Hopple. New York: St. Martin’s, 94– 40. 113. Tomz, Michael. 2009. “The Foundations of Simon, Herbert A. 1982. Model of Bounded Ratio- Domestic Audience Costs: Attitudes, Expec- nality. Cambridge, MA: MIT Press. tations, and Institutions.” In Expectations, Simon, Herbert A. 1985. “Human Nature in Poli- Institutions, and Global Society, eds. Masaru tics: The Dialogue of Psychology with Political Kohno and Aiji Tanaka. Tokyo: Keiso-Shobo. Science.” American Political Science Review 79: Retrieved from http://www.stanford.edu/∼ 293–304. tomz/pubs/Tomz-AudCost-Foundations- Snyder, Richard C., H. W. Bruck, and Bur- 2009-04-14b.pdf (December 5, 2010). ton Sapin. 1954. Decision Making as an Tversky, Amos, and Daniel Kahneman. 1992. Approach to the Study of International Poli- “Advances in Prospect Theory: Cumulative tics. Foreign Policy Analysis Project Series Representation of Uncertainty.” Journal of Risk No. 3. Princeton, NJ: Princeton University and Uncertainty 5: 297–323. Press. Vertzberger, Yaacov. 1997. “Collective Risk Tak- Snyder, Richard, H. W. Bruck, Burton Sapin, ing: The Decision-Making Group.” In Beyond Valerie M. Hudson, Derek H. Chollet, and Groupthink, eds. Paul ’t Hart, Eric K. Stern, James M. Goldgeier. 2002. Foreign Policy and Bengt Sundelius. Ann Arbor: University of Decision-Making Revisited. New York: Palgrave Michigan Press. Macmillan. Wood, Wendy. 2000. “Attitude Change: Persua- Sylvan, Donald A., and James F. Voss, eds. 1998. sion and Social Inﬂuence.” Annual Review of Problem Representation in Foreign Policy Deci- Psychology 51: 539–70. Part IX

ADVANCED EXPERIMENTAL METHODS

CHAPTER 31

Treatment Effects

Brian J. Gaines and James H. Kuklinski

Within the prevailing Fisher-Neyman-Rubin The rationale for substituting group aver- framework of causal inference, causal effects ages originates in the logic of the random are deﬁned as comparisons of potential out- assignment experiment: each unit has differ- comes under different treatments. In most ent potential outcomes; units are randomly contexts, it is impossible or impractical to assigned to one treatment or another; and, observe multiple outcomes (realizations of in expectation, control and treatment groups the variable of interest) for any given unit. should be identically distributed. To make Given this fundamental problem of causal- causal inferences in this manner requires ity (Holland 1986), experimentalists approx- that one unit’s outcomes not be affected by imate the hypothetical treatment effect by another unit’s treatment assignment. This comparing averages of groups or, sometimes, requirement has come to be known as the averages of differences of matched cases. stable unit treatment value assumption. Hence, they often use (Y |t = 1) − (Y |t = 0) Until recently, experimenters have repor- | = − | = to estimate E[(Yi t 1) (Yi t 0)], label- ted average treatment effects as a matter ing the former quantity the treatment effect or, of routine. Unfortunately, this difference of more accurately, the average treatment effect. averages often masks as much as it reveals. Most crucially, it ignores heterogeneity in Many individuals have provided invaluable advice since treatment effects, whereby the treatment we ﬁrst began working on this topic. Jake Bowers, Jamie affects (or would affect if it were actually expe- Druckman, Jude Hays, Tom Rudolph, Jasjeet Sekhon, and Paul Sniderman all offered thoughtful comments on rienced) some units differently from others. related papers, which have shaped this chapter and the This chapter critically reviews how re- project more generally. We are especially indebted to searchers measure, or fail to measure, het- Don Green, whose advice and encouragement at several steps along the way were indispensable. We presented erogeneous treatment effects in random papers related to this chapter at the American Political assignment experiments, and takes as its Science Association and Midwest Political Science Asso- integrating theme that these effects deserve ciation meetings, as well as at Indiana University, Loyola University (Psychology Department), Florida State Uni- more attention than scholars have given versity, and the University of Illinois. them. In multiple ways, such heterogeneity,

445 446 Brian J. Gaines and James H. Kuklinski when not addressed, reduces an experiment’s 4. Social Context: Sometimes an individual’s capacity to produce a meaningful answer to social milieu, which is determined by the initial research question. the choices of others, conditions a treat- The remainder of this chapter discusses ment’s impact on him or her. Situa- four varieties of heterogeneity that can com- tions of this sort, in which an individ- plicate and derail interpretations of estimated ual’s outcome depends on the treatment treatment effects: statuses of other individuals, encompass various institutional settings and violate 1. Noncompliance: Calculations intended to the stable unit treatment value assump- correct for measured and unmea- tion. They thereby raise potentially seri- sured failures to comply with control/ ous complications for the random assign- treatment assignment, especially in field ment experiment and observational experiments, imply possible variance in studies alike. the treatment’s impact on units and, equally, ignorance of how the treatment For any given study, heterogeneity of one would have affected untreated cases. or more of these types might be an issue. Other, more subtle forms of noncom- Sometimes, more than one variety of hetero- pliance, such as nonresponse in survey geneity will obtain; in other cases, none will. experiments, pose related but distinct These sources of heterogeneity can confound problems. efforts to identify an important real life phe- 2. Pre-Treatment: Acknowledging real- nomenon by means of estimating an experi- world pre-treatment of cases admits the mental treatment effect. Unfortunately, most possibility that a random assignment ex- are not directly or fully observable within the periment captures not the discrete experiment itself. effect of treatment, but the average Two crucial implications follow. First, an marginal effect of additional treatment, experimenter should avoid thinking only in conditional on an unmeasured level of terms of a single average treatment effect. real-world pretreatment. At one ex- One might ultimately choose to compute and treme, a universal, powerful, and endur- report only an average treatment effect, but ing real-world effect can ensure that an this decision should be made consciously, experimental treatment has no impact, and the rationale stated explicitly. Second, leading to an erroneous conclusion about prior to designing an experiment, a researcher real-world cause and effect. should seriously consider the plausibility 3. Selection: Random assignment generates of heterogeneous effects. The temptation, an estimate of the average treatment in this regard, is to call for good theory. effect over the whole population. In the Although we welcome strong theory, devel- real world, individuals often self-select oping thoughtful expectations about the phe- into or out of the treatment. Hence, nomenon under study would be a welcomed the experimentally estimated average first step. treatment effect can differ markedly from the real-world average treatment effect, depending on whether those who 1. Importance of Distributions to the selected out would have, in the event Study of Treatment Heterogeneity of exposure, reacted differently to the treatment from those who selected in. Social scientists routinely construe treatment Both average treatment effects, experi- effects solely in terms of the difference in mental and real-world, can be interest- means between treatment and control con- ing and important, but researchers often ditions. Because a treatment’s effect on each fail to discriminate between them, or to subject is generally unobservable, this differ- identify which is of primary substantive ence of group averages does not fully summa- interest. rize the information generated by a random Treatment Effects 447 assignment experiment. The distributions on ing beyond means, to distributions, is always the dependent variable(s), taken by all ran- prudent. domly constituted groups, can be especially useful when exploring the possibility of heterogeneous treatment effects. 2. Measured and Unmeasured Suppose, for instance, that an experi- Noncompliance and Treatment menter seeks to determine whether requiring Effects people to state and justify their policy positions produces more moderate positions, Because field experimenters conduct their overall, than requiring them only to state research directly in the real world, they typ- their positions, without rationale. The ically lose some control over delivery of the experimenter randomly assigns subjects to treatment, thus failing to implement fully the the treatment (state-and-justify) and control assignment of units to treatment and control (state-only) conditions. He or she measures conditions. For this reason, they have devoted policy positions on seven-point scales and, more attention to heterogeneous treatment on each item, finds no difference in means effects than have users of other experimental across conditions. types. Nevertheless, treatment effect hetero- It would be premature to conclude that geneity stemming from noncompliance can the treatment lacks effect. Suppose that in pervade all experiments, sometimes in not so the control condition, where people are not obvious ways. required to justify, policy preferences are To begin, let T designate treated and uniformly distributed. Moreover, those indi- C designate controlled, by which we mean viduals in the treatment condition who are untreated. Using ∗ to designate assignment, inclined to take relatively centrist positions in we can partition units into two mutually the absence of justification moderate further exclusive, exhaustive groups according to ∗ ∗ when asked to defend these positions. Those their assignment status, T and C . We can inclined to take relatively extreme positions, also partition units into another two mutu- in contrast, become even more extreme when ally exclusive and exhaustive groups, accord- they both state and defend their positions. ing to their actual experience, T and C. Members of the control group would thus To characterize both assignment and actual exhibit approximately uniform preferences, experience, we designate the four exhaus- whereas members of the treatment group tive, mutually exclusive, and not necessarily would pile up in the middle and at the end observable statuses by listing assignment sta- ∗ ∗ points. The means of the two groups would be tus first and experience second: T T, T C, ∗ ∗ identical, or nearly so, leading the researcher C T, C C. Units in the first and last type to miss the substantively interesting dynam- can be described as compliers, and those in ics associated with the treatment.1 Generally, the middle two types as noncompliers.Only ∗ ∗ a comparison of means is inadequate if treat- the two groups T and C are randomly con- ment produces multiple and offsetting effects stituted, and they provide analytic leverage (Fisher 1935). via their equivalent-in-expectation composi- Comparing distributions is more difficult tions. Full heterogeneity in parameters across and less familiar than comparing means (or the four types produces nonidentification, so conditional means), but equating “treatment scholars impose assumptions. effect” with differences in means is plainly Political scientists have focused primarily too limiting. Throughout the remainder of on one instance of the problem, the observ- this chapter, we compare means (or propor- able, unintended nontreatment of some cases ∗ tions) in the interest of simplicity. But look- assigned to treatment (i.e., the T C cases) (Gerber and Green 2000; Gerber, Green, and Larimer 2008; Hansen and Bowers 2009). In a 1 A careful researcher might note differences in vari- canonical mobilization experiment, for exam- ances and proceed accordingly (Braumoeller 2006). ple, a researcher randomly selects a subset 448 Brian J. Gaines and James H. Kuklinski of individuals from a public list of registered Hence, tE can be estimated by taking the ∗ ∗ voters and exposes them to a mobilization difference in turnout for the T and C message, such as a postcard containing some groups and dividing by αˆ , the estimate of ∗ information about the election or an argu- the proportion type Es, derived from the T ment in favor of voting. A naive estimate of group. the treatment effect would be the eventually Note that, under these assumptions, no observable difference in actual, real-world estimate of tH is available, and so only one turnout between those mobilized in this man- of the two postulated treatment effects is ner (the Ts) and those not (the Cs), ignoring evaluated. The premises of the calculation assignments. Alternatively, one might com- are that there could be heterogeneity in the ∗ pare the TstotheC s. effects of the treatment, and that the exper- Recognizing that neither of those differ- iment generates estimates of only some of ences adjusts for the possibility of systematic the potentially interesting parameters. There differences between compliers and noncom- is, just the same, a tendency among scholars pliers, field experimenters typically divide the to describe the estimate as “the” treatment measured treatment effect by the proportion effect. Downplaying the unmeasured hypoth- of assigned-to-treatment cases that acciden- esized effect is perhaps natural, but it is also tally went untreated by virtue of nondeliv- imprecise, if not perverse.3 ery of the cards.2 This remedy assumes, often Treatment is not, of course, delivery of a implicitly, that the delivery process measures message to a mailbox, but, rather, exposure an otherwise unobservable quality in voters, of the individual to that message. Accord- which might be thought of as being hard to ingly, actual treatment in a field experiment reach. The companion hypothesis is that the on mobilization is sometimes unobservable. easy to reach (E) and hard to reach (H) could Some cards will be discarded as junk without differ both in their baseline probabilities of having been read or processed, some cards voting (bE and bH) and in their responses to will be read, but not by the intended individ- mobilization (tE and tH). uals, and so on. Unmeasured nontreatment of ∗ The goal is to posit an explicit model with T cases leads the researcher to overestimate α sufficient restrictions to permit estimates of and underestimate tE. The resulting bias, interesting parameters. Assuming only two therefore, is conservative. ∗ types is a start. Taking the partition of T s In a departure from the usual approaches, to be a valid measure of type means that, by Hansen and Bowers (2009) use sampling assumption, the empirical estimate of α,the theory to generate confidence intervals for proportion of the public that is type E,isthe treatment effects given noncompliance. In ∗ number of T T cases divided by the number the terminology introduced previously, they ∗ of T cases. Further assuming no accidental compute a confidence interval for the mean treatment of those assigned to control (say, baseline rate, with minimal assumptions by spillover within households, as when about compliance types or a noncompliance multiple people read the postcard) means model, and then use simple arithmetic to ∗ no C T cases. Finally, random assignment translate the observed treatment effect on creates identical expected mixtures of E and those reached (tE) into a confidence interval. ∗ ∗ H types in the T and C groups. So, if Agnostic about types, the method emphasizes Y is an indicator for turning out to vote, average effects, and Hansen and Bowers make | ∗ = α + − α then E(Y C ) (b E ) (1 )(b H ) and a compelling case for its widespread use. | ∗ = α + + − α . E(Y T ) (b E tE ) (1 )(b H ) There are other approaches to the noncompliance problem. Albertson and Law- rence (2009), studying the effects of expo- 2 Equivalently, one can estimate a two-stage least sure to particular television programs, had squares linear probability model of voting, with 3 an assignment-to-treatment indicator as an instru- The baseline rates, bE and bH, are jointly determined, ment for actually being treated (Angrist, Imbens, and so one could report bounds given the estimate of tE. Rubin 1996). Most analyses, however, ignore them. Treatment Effects 449 to cope with multiple forms of noncompli- message or a placebo environmental mes- ∗ ∗ ance, both T C cases and C T cases (as mea- sage successfully contacted subjects between sured by unverified self-reports of behavior). one third and one half of the time. By com- They, too, introduce a host of assumptions to paring turnout rates of 1) the two groups simplify the problem to the point of tractabil- of contacted subjects and 2) the two groups ity. Positing three types – 1) always T, 2) com- of housemates, Nickerson generates esti- pliers who behave as assigned, and 3) never mates of direct and indirect mobilization. T – they next assume a common treatment But in an otherwise careful presentation, he effect (unobserved, of course, for type 3 sub- says little about the unmeasured, unknown jects) but differing baseline rates (in their case, effects treatment would have had on those probabilities of correctly answering factual not contacted. He does briefly puzzle over questions). As a result, they obtain an esti- the unexpected lack of difference between mate of the treatment effect from another both preexperimental and postexperimen- ratio of a difference and a proportion, the tal turnout rates of the placebo and con- latter estimable on the assumptions that only trol (those assigned to treatment or placebo, ∗ ∗ type 1sareC T and only type 3sareT C: but not successfully contacted) groups. But, having noted this surprising similarity in ∗ E(Y |T ) = α1(b1 + t1) + α2(b2 + t2) baseline rates, he offers no additional qualifi- +(1 − α1 − α2)(b3) cations about possible heterogeneity in treat- | ∗ = α + + α ment effects. The experiment reveals noth- E(Y C ) 1(b1 t1) 2(b2) ing about the susceptibility of the control +(1 − α1 − α2)(b3) subjects to the political message. The study, N T ∗ T N C∗ T although persuasive and precise, nevertheless α2 = − ∗ ∗ exhibits the usual inattention to the prospect N T N C ∗ of significant treatment effect heterogene- N C C = (α2 + α1) − α1 = ity. Because it is highly innovative, we single N ∗ C it out. N T ∗3 − = (α2 + α3) − α3 Barring mistakes of implementation, labo- N T ∗ ratory and survey experiments treat all those ∗ ∗ Y |T − Y |C chosen for treatment, and no one else.4 ∴ tˆ2 = . αˆ 2 Generally, therefore, basic compliance with assignment is not an issue. However, when The approach is clever, although the subjects do not understand the treatment or typology would not work with multiple itera- do not take it seriously, these experimenters tions wherein some initial noncompliers shift face problems akin to noncompliance. Labo- to complying even with the same assign- ratory experiments also sometimes lose sub- ment. To achieve identification the authors jects who depart midway through a study, made a strong assumption of homogenous whereas surveys usually suffer nonresponse treatment effects across the posited types, on some items, so that the behavior of inter- notwithstanding their potentially disparate est (the dependent variable) can be missing. untreated states. Common practice is to quietly drop missing Nickerson (2005) reviews several alter- data cases from analysis, often without even native assignment strategies, including the reporting the number of lost observations. In placebo as control approach, which reduces such instances, noncompliance is masked, not the need to impose assumptions by creat- overcome. ing multiple instances of the contact process. He later demonstrates this design in a study investigating contagion/spillover effects on 4 We assume that experimental treatments success- the housemates of those treated with mobi- fully simulate the phenomena of interest. Myriad 2008 well-known issues concerning validity and strength lizing messages (Nickerson ). In two of treatment arise in experiments, but these are not cities, canvassers with either a mobilization our primary concern here. 450 Brian J. Gaines and James H. Kuklinski

Suppose that nonresponse is systematic, How to interpret experimental treatment such that it wrecks the initial balance that effects depends critically on one’s assump- random assignment aims to achieve. A sim- tions about the relationship between the two ple fix consists of introducing covariates to contexts, real-world and experimental sim- account for different rates of nonresponse ulation thereof. Validity concerns abound, in the treatment and control conditions. If, of course. In laboratory studies of negative however, nonresponders differ from respon- advertisements, for example, scholars strive ders with respect to baseline rates or treat- to mimic the look and tone of real ads and ment effects, then this simple fix will not disguise their purposes by embedding the ads necessarily suffice. In the event that the in news broadcasts and (temporarily) mislead- treatment itself induces nonresponse, this ing subjects about the study’s goal. When fea- unintended effect thwarts measurement of sible, they even create a homelike setting. the expected treatment effect, as captured Less discussed is how real-world pre- in differences between means. Fortunately, treatment might complicate the interpreta- the data sometimes contain evidence of the tion of any difference in behavior between problem. But solutions are not easy; impu- the experimentally treated and untreated. tation, for instance, will generally require Researchers use random assignment to strong assumptions. Bounds for the effects of ensure that the treated will be identical to nonresponse on estimates can be computed, the untreated, in expectation, apart from the but they need not be narrow (Manski 1995, experimental treatment itself. Why, then, 22–30). is nonexperimental pre-treatment different Although laboratory experimenters suc- from any other trait that might be pertinent cessfully avoid overt noncompliance most of to the behavior of interest but should be bal- the time, they must be sensitive to less obvi- anced, in expectation, across groups? As long ous forms of noncompliance with the treat- as there is some possibility that experimental ment, such as a lack of engagement. Sur- subjects arrive already having been exposed vey researchers might view nonresponses as to the actual treatment being simulated, the a form of noncompliance and tackle the phe- experiment estimates not the average treat- nomenon head on. Because the various forms ment effect, but, rather, the average marginal of noncompliance have design implications, effect of additional treatment (Gaines, Kuk- users of all types of experiments would benefit linski, and Quirk 2007). from identifying the extent of noncompliance To be concrete, suppose that viewing and making appropriate adjustments. Even negative advertisements decreases everyone’s = when solutions are costly, addressing the probability√ of voting such that p(votei) problem can only improve the end product. bi(1 – 0.1 ki), where k is the number of ads seen in the last j days and b is a baseline, untreated probability, both varying across 3. Pre-Treatment and Treatment individuals. The estimated treatment effects Effects from an experiment would then depend on the distributions of b and k, and on the chance Much of the time, political scientists use variation in these parameters across treat- experiments to understand ongoing, real- ment and control groups. But the critical world causal relationships. They use survey point, for now, is that experiments conducted experiments to reveal how the strategic fram- in a relatively pristine population, unex- ing of issues in live debates shapes citizens’ posed to advertisements, will find much larger views and preferences. They conduct voter effects (i.e., differences between reported vote mobilization field experiments to determine intentions of treatment and control groups) the real-world impacts of “get out the vote” than those performed on subjects who were campaigns. They experimentally study the inundated with advertisements before they effects of negative advertisements in the midst were ever recruited into the study. If every- of nasty campaigns. one’s baseline probability of voting is 0.7, Treatment Effects 451 and no one has seen any ads, the experiment it because random assignment, by design, will estimate a seven percent demobilization entirely eliminates self-selection into or effect from seeing one negative ad (a treat- out of treatment. To achieve this objective, ment that might imprecisely be described as the random assignment experiment exposes being exposed to negative ads). If, instead, some individuals to a treatment that they subjects arrive at the laboratory having seen, would never receive in the real world and on average, four ads that have already affected fails to expose some to a treatment they their voting plans, the estimated demobiliza- would receive. Whenever the experimenter’s tion effect will be less than two percent. purpose is to estimate real-world treatment The foregoing is a bit fast and loose effects that are shaped by selection pro- because it rests on a fuzzy assumption of long cesses, therefore, randomization might not duration treatment effects. The experimenter be appropriate. The experimenter begins might offer an ironic rebuttal: pre-treatment with one question – what is the (average) is a negligible concern because the real-world real-world treatment effect? – but unwit- effect will be short term, so that subjects will tingly answers another – what would be the enter the experiment as if they had never been (average) treatment effect if everyone were treated. However, this argument implies that exposed? the researcher is studying fleeting and, hence, Consider, again, the ongoing debate about possibly unimportant forces. If political scien- the effects of exposure to negative campaign tists view negative advertisements as impor- ads on voting turnout. Observational studies tant because they affect people’s inclinations have generally found negative ads to have no to vote, then they must surely believe that an effect or a small positive effect (Lau et al. experiment wherein they expose some sub- 1999). In contrast, a highly visible study jects to an ad or two and others to none is found that exposure to such ads decreased generally susceptible to pre-treatment effects, turnout by about five percentage points unless, perhaps, the experiment is conducted (Ansolabehere et al. 1994). The authors, outside election season. who explicitly take prior observational stud- At minimum, experimenters should con- ies as their point of departure, use an exper- sciously consider whether pre-treatment has iment to generate their primary findings, occurred. Whenever the answer is affirma- and later analyze aggregated observational tive, they will want to enumerate the implica- data to confirm their experimental results tions fully. They should ponder whether the (Ansolabehere, Iyengar, and Simon 1999). existence of a real-world pre-treatment biases Putting aside the measurement problems the experimental estimate downward. Even a that beset the observational studies, the simple yes or no, without any estimated mag- observational and experimental results should nitude of the bias, represents an improvement not be the same unless everyone in the real over failure to pose the question at all.5 world is exposed to campaign ads, or there is no difference in the effects of exposure to these ads between those who do and those 4. Self-Selection as a Moderator who do not experience them in real life. The of Treatment Effects experiments conducted by Ansolabehere et al. (1999), in other words, almost certainly esti- Despite efforts by Tobin (1958), Heckman mate the potential, not the actual, treatment (1976), and others, selection continues to effect. challenge observational studies. Experimen- Is there any way, then, that experimenters talists, in contrast, have been able to ignore can estimate the treatment effects that often seem to be their primary interest, that is, 5 Transue, Lee, and Aldrich (2009)demonstrate effects moderated by self-selection? To find spillover effects occurring across survey experiments such an estimation procedure would not only within a single survey. Sniderman’s chapter in this volume offers a thoughtful response to the pre- increase the scope of questions an exper- treatment argument. imenter could address, it would facilitate 452 Brian J. Gaines and James H. Kuklinski communication between users of observa- (t) and the baseline (untreated) probability of tional and experimental data such that they exhibiting the behavior (b). could begin to address genuinely common The group randomly assigned to control questions. status (RC) does not get the treatment and One simple and promising approach consists of a mixture of the two types with an entails incorporating self-selection into ran- expected proportion of dom assignment experiments. Subjects are = α + − α . randomly assigned to treatment, control, or E(YRC ) b0 (1 )bn self-selection conditions, and those assigned to the self-selection condition then choose Those randomly assigned to treatment (RT) to receive or not receive the treatment.6 also comprise a mixture (the same mixture, in The experimental simulation of the treatment expectation), but because they get the treat- must be highly valid, with the selection mech- ment, they have an expected proportion of anism resembling real-world selection. Analysis of the data created by such an = α + + − α + . experiment could proceed roughly along the E(YRT ) (b0 t0) (1 )(bn tn) lines of adjusting for noncompliance, as discussed previously. Assume a dichotomous The third group, randomly assigned to the outcome variable, Y, so that the analysis condition wherein they select whether to be focuses on the difference in proportions (i.e., treated (RS), has an expected proportion that means of the indicator Y) between control is a weighted average of the baseline rate for and treatment groups. There are only two type ns and the treatment-adjusted rate for behavioral options and two observable types: type os (provided, again, that the experimen- those who opt into the treatment given the tal treatment selection process perfectly sim- chance (type o) and those who opt out (type ulates real-world selection): 7 n). We let α be the proportion of the former = α + + − α . type within the population and assume that E YRS (b0 t0) (1 )(bn) types o and n differ in both treatment effect Under these assumptions, the observed rates 6 Hovland (1959) first proposed this approach. Since generate estimates of both postulated treat- we first wrote on the topic of incorporating self- ment effects. A little arithmetic confirms that selection into random assignment experiments, a related article has appeared (Shadish, Clark, and estimators for the treatment effects on selec- Steiner 2008). It proposes a vaguely similar design, tors and nonselectors are, respectively, although for a different purpose: to explore whether nonrandomized experiments can generate data that, − − suitably analyzed, replicate results from their ran- YRS YRC YRT YRS tˆ0 = and tˆ = . domized counterparts. Unlike us, the authors regard αˆ n 1 − αˆ the results of the random assignment experiment as a gold standard, always preferable, not always feasible. In our view, random assignment can deliver fool’s Just as one can accommodate unintended gold if the aim is to characterize the real-world phe- nontreatment in field experiments, creating nomenon under study. Commenting on the Shadish a simulation of realistic self-selection allows et al. article, Little, Long, and Lin (2008) come closer to our view when they argue that “the real scientific exploitation of otherwise unobservable vari- value of hybrid designs is in assessing not the effects ance. The preceding design reveals every- of selection bias, but rather their potential to com- thing that the random assignment experi- bine the advantages of randomization and choice to yield insights about the role of treatment preference ment reveals, and more. Most crucially, it in a naturalistic setting” (1344). adds information useful for studying a phe- 7 Heterogeneity of types need not match observable nomenon that seems likely to feature sub- behavior perfectly, of course. If, say, a treatment has distinct effects on type u people, who usually opt to stantial selection effects in the real world. be treated (but not always), and type r people, who When trying to understand how a treatment rarely select treatment (but not never), the analysis works in a world where, first, some units are proceeds similarly, but with more distracting algebra. Assuming homogeneity within observable groups is exposed to it and others are not, and, second, merely an analytic convenience. the exposed react differently to the treatment Treatment Effects 453 r than the unexposed would have reacted, a Available vouchers are thus given to a ran- self-selection module can prove helpful and, domly chosen subset of the applicants. 8 r arguably, necessary. All voucher recipients enroll in private schools. r All unsuccessful voucher applicants remain 5. Ecological Heterogeneity in in public schools. r Treatment Effects Analysts compare subsequent performance by the voucher recipients to subsequent People self-select into or out of treatments. performance by the unsuccessful voucher Sometimes these decisions do no more than applicants in order to make inferences determine whether the isolated individual will about the effect of being educated in a pri- or will not receive the treatment. At other vate school. times, however, the individual’s choice of treatment also determines with whom he or she will interact. If, in turn, such interac- By studying applicants only, the researcher tions shape the behavior of interest, then controls for a range of unobservable fac- the effect of the treatment will be condi- tors that are likely associated with parental tional on the social ecology within which ambition and the child’s performance. Study- an individual is embedded. This additional ing applicants only also restricts the domain source of potential heterogeneity, when not about which one can draw inferences and thus taken into account, will weaken the validity changes the research question. Still, the use of a random assignment experiment. Exam- of randomization by the program adminis- ples abound, but for purposes of illustration, trators is felicitous, ensuring that accepted consider schools. and rejected pupils will be similar in moti- School voucher programs have been at the vation and potential – identical, in expecta- center of intense controversies for decades, tion, other than their differences induced by with analyses purporting to show very dif- the treatment (attendance at private school or ferent results on whether they help or hin- not). Can this seemingly foolproof design go der educational achievement. The literature wrong? is vast: confining attention to one arbitrary For simplicity, suppose further that there year, one can find review essays skeptical are only two kinds of pupils, those with low that vouchers work (e.g., Ladd 2002), anal- potential (because of cognitive limitations, yses of randomized field experiments report- poor work ethic, lack of family support, etc.) ing strong positive effects (e.g., Angrist et al. and those with high potential. There are also 2002;Howelletal.2002), analyses report- only two schools, one public and one pri- ing mixed results (Caucutt 2002), and essays vate, and they are not necessarily the same complaining that all studies have been too size. Achievement is measured by a test that narrowly focused (Levin 2002). is accurate except for random error, and indi- Several studies of vouchers have exploited vidual students’ measured achievements are a natural experiment wherein an actual generated as follows: program implements random selection for choosing awardees. By way of illustrative Y = β0 + β1 H + β2 P + β3 (HP) example, suppose that +β + ε, 4 ZH r A school voucher program has more applicants than spots. where H and P are indicators for high potential pupils and pupils attending private 8 In a very different context, Orbell and Dawes (1991) schools, respectively; ZH is the proportion of alter a prior experimental design to permit players students in the given school who are high of prisoners’ dilemmas to self-select continuation of ε play and thereby uncover theoretically important and potential types; and is a standard stochastic otherwise invisible heterogeneity. disturbance term. 454 Brian J. Gaines and James H. Kuklinski

Assume that potential is impossible to potential pupils’ scores, but the private school measure and thus acts as a lurking variable.9 puts more weight on the comparatively more Of primary interest is the accuracy of the numerous high potential students, thereby estimated effect of attending private school mixing some β1 into what was meant to be an generated by the (quasi)experimental analysis estimate of β2. A successful selection model described previously, which limits attention might reduce this bias if there are good (mea- to voucher applicants and exploits random sured) predictors of the unmeasured pupil assignment. For contrast, we also consider a potential. naive estimator, the public–private mean dif- For the experimental analysis, the differ- ference. ence between the treatment group (successful In the absence of interaction effects (β3 voucher applicants, placed in private schools) = 0 and β4 = 0), test scores would be ran- and control group (unsuccessful applicants, domly distributed around means as follows: remaining in public schools) does not depend E(Y|H = P = 0) = β0; E(Y|H = 0, P = 1) on the mix of low and high potential stu- = β0 + β2; E(Y|H = 1, P = 0) = β0 + β1; dents in the applicant pool. Random selec- and E(Y|H = P = 1) = β0 + β1 + β2. Con- tion ensures, in expectation, re-creation of the clusions depend on the signs of the betas. If, applicant pool’s ratio of high to low potential for instance, β2 is negative (private schools pupils in both treatment and control. Suppose perform more poorly than public schools for that α is the proportion of the applicants with pupils of both types) and β1 is positive (high high potential. The difference between treat- potential students perform better than oth- ment and control will be (1 – α)(β0 + β2 – β0) ers), yet high potential students are dispro- + α(β0 + β1 + β2 –(β0 + β1)) = β2. Hence, portionately found in the private school, then random assignment will have served its pur- the result could be an instance of Simpson’s pose and solved the problem of a lurking vari- paradox, wherein the public school’s overall able, which caused bias in the observational mean achievement score is lower than the pri- estimate. vate school’s mean score, even though stu- If β3 > 0, then test scores are randomly dents of both types perform better in the distributed around those same means, except public school. It is probably more plausible that E(Y|H = P = 1) = β0 + β1 + β2 + β3. that both β1 and β2 are positive, and that the An observational study subtracting the pub- ratio of high potential pupils to low poten- lic school mean from the private school mean tial pupils is higher in the private school than would again obtain an upwardly biased esti- in the public school. Letting λ and δ rep- mate of β2, the private school bonus, namely, resent the proportions of private and public β2 + (λ – δ)β1 + λβ3. There are now, school pupils of high potential, respectively, by assumption, two distinct private school we assume that the private school draws a dis- effects, but this estimate need not lie between proportionate share of high potential pupils, β2, the true effect for the low potential pupils, so that λ – δ>0. and β2 + β3, the true effect for the high A naive observational study that compares potential pupils. public to private mean achievements would The share of the voucher applicants who have an expected treatment effect (benefit were high potential is still α. In the experi- obtained by attending private school) of mental context, the difference between treatment and control is thus β2 + αβ3.For α 0 1 (1 − λ)(β0 − β2) + λ (β0 + β1 + β2) any between and (noninclusive), this value overestimates the achievement effect of − (1 − δ) β0 − δ (β0 + β1) = β2 + (λ − δ) β1. attending private school for pupils with low potential, but underestimates the effect for The estimate is biased upward because both pupils with high potential. In the extreme means are weighted averages of high and low case wherein only high potential pupils apply α = 1 9 Inability to observe pupil potential is a strong assump- for vouchers ( ), the difference between tion, employed to simplify the examples. treatment and control is an accurate estimate Treatment Effects 455 of the returns on private schooling for high formance attributable to the schools’ differ- potential pupils. Like the observational study, ent ecologies. He or she might or might not the experimental analysis would be suscepti- want to credit private schools with recruiting ble to the mistake of inferring that all pupils more capable student bodies when making could have their scores boosted by that same the comparison of types. However, that move amount by being educated in private schools. can lead to false inferences about the likely However, the estimate is usefully bounded by effects of broadening a voucher program. In the two real-world effects. any case, there is no guarantee that the ana- Most interesting is the situation wherein lyst who seeks to exploit randomization will there are both pupil–school interaction get a correct or unbiased estimate of private effects (β3 > 0) and pupil–pupil effects (β4 > school benefits if there are ecological effects 0). Now achievements are more complicated, that cannot easily be measured directly. depending not only on pupil and school types, Actual analyses of vouchers that employ but also on the concentration of high poten- this design often introduce more compli- tial pupils within each school. Again, estimat- cated models that include sensitivity to vari- ing a single private school effect is mislead- ous pupil traits (e.g., Rouse 1998), so we are ing because private schooling delivers distinct not advancing a complaint about the voucher benefits to pupils with low or high potential. literature in particular. We aim merely to Moreover, all pupils gain from contact with reinforce the point that random assignment high potential peers, and this effect can be is no panacea. When the treatment to which felt in the private or public school, depending subjects are randomly assigned has compli- on who applies for vouchers and on the out- cated effects that interact in some manner come of the lottery after the voucher scheme with other unobserved (or even unobservable) is oversubscribed. The estimated treatment factors, causal inferences can be biased, not effect for a quasiexperimental analysis that only in observational studies, but also in stud- exploits the randomization will depend on the ies that aim to exploit (partial) randomiza- actual realization of the random selections. tion of treatment assignment. Scholars ignore The difference between treatment and con- the issue of treatment effect heterogeneity at trol will contain not only the slight bias noted their own peril. previously, but also a term representing the differences in the schools’ distributions of low and high potential pupils. 6. Beyond Internal and External In short, there are reasons to think that Validity a given pupil’s performance might sometimes be affected by the assignment status of The two criteria that scholars traditionally other pupils. This represents a violation of apply when assessing the random assignment Rubin’s (1990a, 1990b) stable unit treatment experiment are internal and external validity. value assumption, in that individual i’s poten- Conventional wisdom says that it typically tial outcome, given a school choice, depends excels at achieving internal validity, while on unit j’s assignment status. Because the falling short of achieving external validity. public and private school populations were How, if at all, do the four varieties of treat- not formed entirely by random assignment, ment heterogeneity that we discuss challenge the analyst of the natural experiment cre- this conventional wisdom? ated by the voucher lottery inherits distinct Most fundamentally, the preceding dis- school ecologies (mixtures of student) that cussion underlines that assessing the random can plausibly condition the treatment effect assignment experiment only in terms of the (the impact on performance of attending pri- two types of validity is too limiting. Instead, vate school). we propose, when researchers use experi- It is not obvious whether a researcher ments to make inferences about a world that should include, in an estimate of “private exists outside the experimental context, they schooling effects,” these differences in per- should think in terms of a complex nexus 456 Brian J. Gaines and James H. Kuklinski consisting of research question, real-world chapter, assume that a team of researchers context, and experimental context. Often, the has been given the opportunity to use a research question arises because the exper- random national sample of individuals in imenter observed an event or phenomenon their experiments. The researchers accept the about which he or she would make causal offer, thinking that the sample will enable inferences. That very observation implies that them to make more encompassing state- subjects might enter the experiment already ments than they could have made otherwise. having been pre-treated, that some subjects In other words, they believe that they have received the pre-treatment while others did improved their claim to proper causal infer- not, or that social interactions generated the ence. They randomly assign their respon- event or phenomenon. In turn, each possibil- dents to conditions and measure average ity should signal the importance of revisiting treatment effects.10 the initial research question. Should the study This all sounds ideal, and in the absence of really focus on marginal rather than average heterogeneous treatment effects, it might be. treatment effects? Does the researcher really As the experimental study of politics contin- want to know the causal effect when every- ues to mature, however, we expect political one receives the treatment, or, given what he scientists to find the usually implicit assump- or she has already observed, is the treatment tion of no heterogeneity less and less tenable. effect only among those who likely receive Consequently, they will increasingly question it when carrying on their day-to-day lives of the value of random samples, strictly random more interest? Should the researcher’s ques- assignment, and average treatment effect esti- tion acknowledge the existence of social inter- mates. What meaning accrues to an experi- actions and their likely effects on the depen- mentally generated average treatment effect dent variable of interest? Similarly, given estimated on the basis of a national sample of hunches about processes and phenomena out- citizens? Does a random assignment exper- side the experimental context, exactly how iment suffice when self-selection pervades should the researcher design the experiment? life? Can experimentalists ignore social inter- Implicit in the preceding discussion is what actions when those interactions at least par- might be called a “paradox of social-scientific tially determine the magnitude of the treat- experimental research”: to know how to ment effect? These sorts of questions, and design an experiment properly for purposes the answers that political scientists give, will of making inferences about a larger world motivate and shape the next generation of requires knowledge of how the processes and experiments. phenomena of interest work in the real world. The strong possibility of heterogeneous Without that knowledge, the researcher can- treatment effects will increase the importance not know how to design the experiment or of knowing the experimental context. It will how to interpret the results. In strong form, also nudge researchers to use experiments this paradox implies that conducting experi- more strategically than they have to date. mental research for purposes of making causal Creative designs, such as Nickerson’s two- inferences about the real world is impossible. person household study, broaden investiga- We recommend, however, that social scien- tion of effects beyond the realm of the iso- tists not accept the paradox, but use it instead lated individual, in the spirit of the Columbia as a reminder that experimental research studies. It is probably true, more gener- requires considerable thought and reflection, ally, that a deliberate focus on experimenting much more than we all once supposed.

10 With the exception of some survey experiments, 7. Conclusion national samples rarely exist in practice. We believe that, if given their choice, most political scientists would use national random samples to conduct their By way of conclusion, and to underline var- experiments. In this regard, our discussion addresses ious observations we make throughout this current thinking, not necessarily current practice. Treatment Effects 457 within multiple homogenous populations can Angrist, Joshua D., Guido W. Imbens, and Don- be very helpful. ald B. Rubin. 1996. “Identification of Causal Consider, again, the effects of negative Effects Using Instrumental Variables.” Journal 91 444 55 campaign ads on voting turnout. Researchers of the American Statistical Association : – . Ansolabehere, Stephen D., Shanto Iyengar, and could deliberately select pools of subjects 1999 from distinct environments, chosen accord- Adam Simon. . “Replicating Experiments Using Aggregate and Survey Data: The Case of ing to expectations about important forms Negative Advertising and Turnout.” American of real-world pre-treatment. They might, for Political Science Review 93: 901–9. example, implement parallel survey or labo- Ansolabehere, Stephen D., Shanto Iyengar, Adam ratory experiments in states or districts with Simon, and Nicholas Valentino. 1994.“Does and without ongoing negative campaigns, or Attack Advertising Demobilize the Elec- with various levels of campaign negativity, to torate?” American Political Science Review 88: explore how real-world context affects exper- 829–38. imental results. Fortunately, the emergence Braumoeller, Bear G. 2006. “Explaining Variance; of Internet delivery as a means of conducting Or, Stuck in a Moment We Can’t Get Out of.” Political Analysis 14: 268–90. survey experiments has already increased the 2002 viability of such studies.11 Caucutt, Elizabeth M. . “Educational Vouch- ers When There Are Peer Group Effects – In many respects, we are elaborating a Size Matters.” International Economic Review 43: viewpoint that has a kinship with those 195–222. espoused by Bowers’ and Druckman and Fisher, Ronald Aylmer, Sir. 1935. The Design of Kam’s chapters in this volume. The fore- Experiments. Edinburgh: Oliver and Boyd. most goal of the experimenter must be to Gaines, Brian J., James H. Kuklinski, and Paul J. get the cause and effect story right, and Quirk. 2007. “The Logic of the Survey Exper- that goal poses more challenges than sim- iment Reexamined.” Political Analysis 15: 1–20. ple experimental logic suggests. It requires Gerber, Alan S., and Donald P. Green. 2000.“The focused attention on the question-context- Effects of Personal Canvassing, Telephone experiment nexus. Heterogeneity in treat- Calls, and Direct Mail on Voter Turnout: A ment effects is not ultimately a nuisance, Field Experiment.” American Political Science Review 49: 653–64. but rather an important hypothesis about the Gerber, Alan S., Donald P. Green, and Christo- world to be tested carefully. pher W. Larimer. 2008. “Social Pressure and Vote Turnout: Evidence from a Large-Scale Field Experiment.” American Political Science References Review 102: 33–48. Hansen, Ben B., and Jake Bowers. 2009. “Attribut- Albertson, Bethany, and Adria Lawrence. 2009. ing Effects to a Cluster-Randomized Get-Out- “After the Credits Roll: The Long-Term the-Vote Campaign.” Journal of the American Effects of Educational Television on Public Statistical Association 104: 873–85 Knowledge and Attitudes.” American Politics Heckman, James J. 1976. “The Common Struc- Research 32: 275–300. ture of Statistical Models of Truncation, Sam- Angrist, Joshua, Eric Bettinger, Erik Bloom, ple Selection and Limited Dependent Vari- Elizabeth King, and Michael Kremer. 2002. ables and a Simple Estimator for Such Models.” “Vouchers for Private Schooling in Colombia: Annals of Economic and Social Measurement 5: Evidence from a Randomized Natural Experi- 475–92. ment.” American Economic Review 92: 1535–58. Holland, Paul W. 1986. “Statistics and Causal Inference.” Journal of the American Statistical Association 81: 945–60. Hovland, Carl I. 1959. “Reconciling Conflicting 11 Taking a cue from field experiments, these Results Derived from Experimental and Survey researchers should also study not only self-reported vote intent, but also actual turnout, eventually Studies of Attitude Change.” American Psychol- recorded in official records. Doing so should help dis- ogist 14: 8–17. criminate between short-lived and long-lived effects Howell, William G., Patrick J. Wolf, David E. of various treatments. Again, felicitously, Internet Campbell, and Paul E. Peterson. 2002. “School panels make validation easier and less costly. 458 Brian J. Gaines and James H. Kuklinski

Vouchers and Academic Performance: Results Orbell, John, and Robyn M. Dawes. 1991.“A from Three Randomized Field Trials.” Jour- ‘Cognitive Miser’ Theory of Cooperators’ nal of Policy Analysis and Management 21: 191– Advantage.” American Political Science Review 217. 85: 515–28. Ladd, Helen F. 2002. “School Vouchers: A Critical Rouse, Cecilia Elena. 1998. “Private School View.” Journal of Economic Perspectives 16: 3– Vouchers and Student Achievement: An Eval- 24. uation of the Milwaukee Parental Choice Pro- Lau, Richard R., Lee Sigelman, Caroline Held- gram.” Quarterly Journal of Economics 113: 553– man, and Paul Babbitt. 1999. “The Effects 602. of Negative Political Advertisements: A Meta- Rubin, Donald B. 1990a. “Comment: Neyman Analytic Assessment.” American Political Science (1923) and Causal Inference in Experiments Review 93: 851–76. and Observational Studies.” Statistical Science 5: Levin, Henry M. 2002. “A Comprehensive Frame- 472–80. work for Evaluating Educational Vouchers.” Rubin, Donald B. 1990b. “Formal Modes of Sta- Educational Evaluation and Policy Analysis 24: tistical Inference for Causal Effects.” Journal of 159–74. Statistical Planning and Inference 25: 279–92. Little, Roderick J., Qi Long, and Xihong Lin. Shadish, William R., M. H. Clark, and Peter M. 2008. “Comment [on Shadish et al.].” Journal Steiner. 2008. “Can Nonrandomized Experi- of the American Statistical Association 13: 1344– ments Yield Accurate Answers? A Randomized 50. Experiment Comparing Random and Nonran- Manski, Charles F. 1995. Identiﬁcation Problems in dom Assignments.” Journal of the American Sta- the Social Sciences. Cambridge, MA: Harvard tistical Association 103: 1334–43. University Press. Tobin, James. 1958. “Estimation of Relationships Nickerson, David W. 2005. “Scalable Protocols for Limited Dependent Variables.” Economet- Offer Efﬁcient Design for Field Experiments.” rica 26: 24–36. Political Analysis 13: 233–52. Transue, John E., Daniel J. Lee, and John H. Nickerson, David W. 2008. “Is Voting Conta- Aldrich. 2009. “Treatment Spillover Effects gious? Evidence from Two Field Experiments.” across Survey Experiments.” Political Analysis American Political Science Review 102: 49–76. 17: 143–61. CHAPTER 32

Making Effects Manifest in Randomized Experiments

Jake Bowers

Experimentalists want precise estimates of covariates may, in principle, enable pursuit treatment effects and nearly always care about of precision and statistical adjustment, this how treatment effects may differ across sub- chapter presents two alternative approaches groups. After data collection, concern may to covariance adjustment: one using modern turn to random imbalance between treatment matching techniques and another using the groups on substantively important variables. linear model – both use randomization as the Pursuit of these three goals – enhanced preci- basis for statistical inference. sion, understanding treatment effect heterogeneity, and imbalance adjustment – requires background information about experimental 1. What Is a Manifest Effect? units. For example, one may group similar observations on the basis of such variables A manifest effect is one we can distinguish and then assign treatment within those from zero. Of course, we cannot talk for- blocks. Use of covariates after data have mally about the effects of an experimental been collected raises extra concerns and treatment as manifest without referring to requires special justiﬁcation. For example, probability: a scientist asks, “Could this result standard regression tables only approximate have occurred merely through chance?” or the statistical inference that experimentalists “If the true effect were zero, what is the desire. The standard linear model may also chance that we’d observe an effect as large mislead via extrapolation. After provid- as this?” More formally, for a frequentist, ing some general background about how saying that a treatment effect is manifest means that the statistic we observe casts Many thanks to Jamie Druckman, Kevin Esterling, a great deal of doubt on a hypothesis of Don Green, Ben Hansen, Don Kinder, Jim Kuklinski, no effects. We are most likely to say that Thomas Leeper, Costas Panagopoulos, and Cara Wong. some observed effect casts doubt on the Parts of this work were funded by National Science Foundation (NSF) Grant Nos. SES-0753168 and SES- null hypothesis of no effect when we have 0753164. a large sample and/or when noise in the

459 460 Jake Bowers outcome that might otherwise drown out the ments either before treatment is assigned (by signal in our study has been well controlled. creating subgroups of units within which Fisher (1935) reminds us that although ran- treatment will be randomized during the domization alone is sufficient for a valid test recruitment and sample design of a study) of the null hypothesis of no effect, specific and/or after treatment has been assigned and features of a given design allow equally valid administered and outcomes measured (by tests to differ in their ability to make a treat- creating subgroups within which outcomes ment effect manifest: ought to be homogeneous or adjusting for covariates using linear models). With respect to the refinements of technique [uses of covariates in the planning of an experiment], we have seen above that these con- Common Uses (and Potential Abuses) tribute nothing to the validity of the experi- of Covariates in the Workflow ment, and of the test of significance by which of Randomized Experimentation we determine its result. They may, however, be important, and even essential, in permit- Every textbook on the design of experiments ting the phenomenon under test to manifest is, in essence, a book about the use of covari- itself. (24) ates in the design and analysis of experiments. This chapter does not substitute for such Of course, one would prefer a narrow con- sources. For the newcomer to experiments, fidence interval to a wide confidence inter- I summarize in broad strokes, and with min- val, even if both excluded the hypothesis of imal citations, the uses to which covariates no effects. As a general rule, more infor- may be put in the design and analysis of ran- mation yields more precision of estimation. domized experiments. After this summary, I One may increase information in a design by offer a perspective on the use of covariates gathering more observations and/or gather- in randomized experimentation that, in fun- ing more data about each observation. This damental ways, is the same as that found in paper considers covariates as a refinement of Fisher (1925, 1935), Cochran and Cox (1957), technique to make treatment effects manifest Cox (1958), and Cox and Reid (2000). I dif- in randomized studies. I focus first on sim- fer from these previous scholars in hewing ple uses of covariates in design and then offer more closely and explicitly to 1) the now some ideas about their use in post-treatment well-known potential outcomes framework adjustment. for causal inference (Rubin 1974, 1990; Nay- man 1990; Brady 2008; Sekhon 2008) and 2) randomization as the basis for statistical What Are Covariates? How Should We Use inference. Them in Experiments? A covariate is a piece of background informa- covariates allow precision tion about an experimental unit – a variable enhancement unchanged and unchangeable by the exper- Blocking on background variables before imental manipulation. Such variables might treatment assignment allows the experi- record the groups across which treatment menter to create subexperiments within effects ought to differ according to the theory which the units are particularly similar in motivating and addressed by the experimen- their outcomes, and adjustment using covari- tal design (e.g., men and women ought to ates after data have been collected may react differently to the treatment) or might also reduce nontreatment-related variation provide information about the outcomes of in outcomes. In both cases, covariates can the experiment (e.g., men and women might reduce noise that might otherwise obscure the be expected to have somewhat different out- effects of the treatment. comes even if reactions to treatment are Of course, such precision enhancements expected to be the same across both groups). arrive with some costs: implementing a block- Covariates may be used profitably in experi- ing plan may be difficult if background Making Effects Manifest in Randomized Experiments 461 information on experimental units is not dom imbalance. One may attempt to remove available before recruitment/arrival at the the effects of such covariates from the treat- lab (but see Nickerson 2005); care must be ment effect by some form of adjustment. For taken to reflect the blocking in the estima- example, one may use the linear regression tion of treatment effects to avoid bias and model as a way to adjust for covariates, or to take advantage of the precision enhance- one may simply group together observations ments offered by the design; and, in principle, on the basis of the imbalanced covariates. analysts can mislead themselves by perform- Adjustment, however, raises concerns that ing many differently adjusted hypothesis tests estimates of treatment effects may come to until they reject the null of no effects, even depend more on the details of the adjustment when the treatment has no effect. method rather than on the randomization and design of the study. Thus, the quandary: covariates enable subgroup analyses either risk known confusions of comparisons When theory implies differences in treatment or bear the burden of defending and assess- effects across subgroups, subgroup member- ing an adjustment method. An obvious strat- ship must be recorded, and, if possible, the egy to counter concerns about cherry-picking experiment ought to be designed to enhance results or modeling artifacts is to present both the ability of the analyst to distinguish group adjusted and unadjusted results and to specify differences in treatment effects. Covariates adjustment strategies before randomization on subgroups may also be quite useful for post occurs. hoc exploratory analyses designed not to cast doubt on common knowledge, but to suggest Randomization Is the Primary Basis further avenues for theory. for Statistical Inference in Experiments covariates allow adjustments A discussion of manifest effects is also a for random imbalance discussion of statistical inference: statisti- All experiments may display random imbal- cal tests quantify doubt against hypotheses, ance. Such baseline differences can arise even and a manifest effect is evidence that casts if the randomization itself is not suspect: great doubt on the null of no effects. On recall that one of twenty unbiased hypothesis what basis can we justify statistical tests for tests will reject the null of no difference at the experiments? predetermined error rate of α = .05 merely In national surveys, we draw random sam- due to chance. An omnibus balance assess- ples. Statistical theory tells us that the mean ment such as that proposed by Hansen and in the sample is an unbiased estimator of Bowers (2008) is immune from this problem, the mean in the population as long as we but any unbiased one-by-one balance assess- correctly account for the process by which ment will show imbalance in 100α percent of we drew the sample in our estimation. That the covariates tested. Thus, I call this prob- is, in a national survey, our target of infer- lem “random imbalance” to emphasize that ence is often (but not always) the population the imbalance could easily be due to chance from which the sample was drawn, and we and need not cast doubt on the randomiza- are justified in so inferring by the sampling tion or administration of a study (although design. discovery of extensive imbalance might sug- In other studies, we may not know how gest that scrutiny of the randomization and a sample was drawn (either we have no administration is warranted). well-defined population or no knowledge of Random imbalance in a well-randomized the sampling process or both). But we may study on substantively important covariates know how our observed outcome was gen- may still confuse the reader. In the presence erated: say we know that at the microlevel of random imbalance, comparisons of treated our outcome was created from discrete events to controls will contain both the effect of the occurring independently of each other in treatment and the differences due to the ran- time. In that case, we would be justified 462 Jake Bowers in claiming that our population was cre- ences than it does using the linear regression ated via a Poisson process: in essence, we model-based approximation.2 have a population-generating machine, data- Recently, Freedman (2008a, 2008b, generating process, or model of outcomes as 2008c) reminded us that the target of in- the target of our inference. ference in randomized experiments was the Now, what about randomized experi- counterfactual, and he noted that linear ments? Although we might want to infer to a regression and logistic regression were not model, or even to a population, the strength theoretically justified on this basis. Green of experiments is inference to a counterfac- (2009) and Schochet (2009) remind us that tual. The primary targets of inference in a linear regression can often be an excellent randomized experiment are the experimen- approximation to the randomization-based tal treatment groups: we infer from one to difference of means. The need for this ex- another. Randomization makes this inference change is obvious, even if it is echoed in the meaningful. But, randomization can also jus- early textbooks. Those early authors moved tify the statistical inference as well. The mean quickly to the technicalities of the approx- in the treatment group is a good estimator for imations rather than dwelling on the then what we would expect to observe if all of the uncomputable but theoretically justifiable experimental units were treated. The treat- procedures. As experimentation explodes as ment group in a randomized study is a ran- a methodology in political science, we are dom sample from the finite “population” of seeing many more small experiments, designs the experimental pool.1 where randomization is merely a (possibly All standard textbooks note this fact, but weak) instrument, and theories implying very they also point out that estimating causal heterogeneous treatment effects. I expect we effects using randomization-based theory can will find more of these along with many more be mathematically inconvenient or compu- large studies with fairly normal-looking out- tationally intensive and that, thus, using comes where treatment plausibly just shifts the large-sample sampling theory (and/or the normal curve of the treated away from normal distribution models) turns out to the normal curve of the controls. Rather than provide very good approximations to the hope that the linear model approximation randomization-based results most of the time. works well, this chapter presents analyses Since the 1930s, the computational appa- that do not require that approximation and ratus of randomization-based inference has thereby offers others the ability to check the expanded, as has its theoretical basis and approximation. applied reach. In this chapter, the statistical inference I present is randomization-based, even if most of it also uses large-sample the- 2. Strategies for Enhancing Precision ory. For example, it takes no more time to before Assignment execute a randomization-based test of the null hypothesis of no effect using mean differ- [W]e consider some ways of reducing the effect of uncontrolled variations on the error of the treatment comparisons. The general idea is 1 See Rosenbaum (2002b,ch.2) and Bowers and the common sense one of grouping the units Panagopoulos (2009) for accessible introductions to into sets, all the units in a set being as alike randomization inference; a mode of inference developed in different yet compatible ways by Fisher 2 This chapter is written in the mixture of R and A (1935,ch.2) (as a method of testing) and Neyman L TEX known as Sweave (Leisch 2002, 2005)and, (1990) (as a method of estimating mean differences). as such, the code required to produce the results, In this paper, I follow the Fisher-style approach tables, and figures (as well as additional analyses not in which causal effects are inferred from testing reported) are available for learning and exploration at hypotheses rather than estimated as points. Both http://jakebowers.org. Thus, I spend relatively little methods (producing a plausible interval for causal time discussing the details of the different methods, effects using a point ± a range or testing a sequence assuming that those interested in learning more will of hypotheses) often produce identical confidence download the source code of the chapter and adapt it intervals. for their own purposes. Making Effects Manifest in Randomized Experiments 463

as possible, and then assigning the treatments which sums across all units within control and so that each occurs once in each set. All com- treatment conditions. These two quantities parisons are then made within sets of similar are the same, even if one is written as an aver- units. The success of the method in reducing age over B pairs, because pairs are blocks of error depends on using general knowledge of equal size and, therefore, each block-specific the experimental material to make an appro- quantity ought to contribute equally to the priate grouping of the units into sets. (Cox sum. 1958, 23) The theory of simple random sampling We have long known that covariates enhance without replacement suggests that the vari- the precision of estimation to the extent that ances of these statistics differ. First, they predict outcomes. This section aims to n − 2 make this intuition more concrete in the con- = n (xi x¯) Var(dno pairs) − − 1 text of a randomized experiment. m(n m) =1 n i n 4 1 2 = (x − x¯) . An Example by Simulation n n − 1 i i=1 Imagine that we desire to calculate a differ- (3) ence of means. In this instance, we are using a fixed covariate x, and the purpose of this dif- Second, ference in means is to execute a placebo test or 1 2 B a balance test, not to assess the causal effects nb Var(dpairs) = B m (n − m ) of a treatment. Imagine two scenarios, one in b=1 b b b which a binary treatment Z = 1 is assigned ib nb − 2 = (xib x¯b ) to mb 1 subject within each of B pairs × = 1 = 1 ≤ = n − 1 b ,..., B; i ,..., nb, B n; n i=1 b B = 2 = 2 b=1 nb (for pairs n B and, thus, nb 2 B 2 = − 2. ), and another in which a binary treatment (xib x¯b ) (4) = 1 = = /2 B2 Zi is assigned to m n–m (n ), b=1 i=1 subjects i = 1,..., n without any blocking. Consider the test statistics If pairs were created on the basis of similarity > on x, then Var(dno pairs) Var(dpairs) because B nb n 2 B 2 2 1 =1 (x − x¯) > =1 =1 (x − x¯ ) . Any = / i i b i ib b dpairs Zibxib mb given xi will be farther from the overall B =1 =1 b i mean (x¯) than it would be from the mean nb of its pair (x¯ ). Note also that the con- − (1 − Z )x /(n − m ) b ib ib b b stants multiplying the sums are (4/n(n−1)) =1 i in the unpaired case and 8/n2 (because B = B 2 1 (n/2)) in the paired case. As long as n > = (Z x − (1 − Z )x ) , 2 2 B ib ib ib ib 2,(4/(n – n)) < (8/n ), and this differ- b=1 i=1 ence diminishes as n increases. Thus, ben- (1) efits of pairing can diminish as the size of the experiment increases as long as within- which reduces to the difference in means of pair homogeneity does not depend on sample x between treated and control units within size. pairs summed across pairs and To dramatize the benefits possible from blocking, I include a simple simulation study n = / based roughly on real data from a field dno pairs Zi xi m =1 experiment of newspaper advertisements and i 2006 n turnout in U.S. cities (Panagopoulos ). − − / − , The cities in this study differed in baseline (1 Zi )xi (n m) (2) i=1 turnout (with baseline turnout ranging from 464 Jake Bowers ) pairs d ( var ) 30 40 50 60 no pairs d ( 0 var 10 12

8 City 32 City 160 City Dataset Figure 32.1. Efficiency of Paired and Unpaired Designs in Simulated Turnout Data roughly ten percent to roughly fifty per- mal distribution with mean equal to the base- cent). Panagopoulos paired the cities before line outcome of its control group and standard randomly assigning treatment, with baseline deviation equal to the standard deviation of turnout differences within pair ranging from the pair. For the original dataset, the within one to seven percentage points in absolute pair differences on baseline turnout of 1, 4, value. The simulation presented here takes and 7 translate into standard deviations of roughly 0.7, 3, and 5. The “true” difference his original 8-city dataset and makes two 32 is required to be zero within every pair, but fake versions, one with cities and another each pair may have a different baseline level of 160 with cities. The expanded datasets were turnout and a different amount of variation – created by adding small amounts of uni- mirroring the actual field experiment. form random noise to copies of the original dataset. These new versions of the original Estimate variance of treatment effects dataset maintain the same general relation- under null Apply Equations (3) and (4).3 ships between treatment and outcomes and 32 1 covariates and pairs as the original, but allow Figure . shows that estimates of us to examine the effects of increasing sam- Var(dno pairs) were always larger than the , ple size. In this simulation, the covariate val- estimates for Var(dpairs) although the ratio ues are created so that the average difference Var(dno pairs)/ Var(dpairs) diminished as the of means within pair is zero. For each of the size of the experiment increased. The stan- 1,000 iterations of the simulation and for each dard deviation across the simulations for the dataset (n = 8, 32, 160), the procedure was as paired 8-city example (1.8) is close to the follows: 3 The actual computations used the xBalance command found in the RItools library for R (Bow- Create fake data based on original data ers, Fredrickson, and Hansen 2009), as described in Each pair receives a random draw from a nor- Hansen and Bowers (2008). Making Effects Manifest in Randomized Experiments 465 average standard deviation for the nonpaired make one or more observed (or unobserved) tests in the 160-city example (2.3). That is, covariates imbalanced merely by chance. If these simulations show that a paired exper- the imbalanced covariates seem particularly iment of 8 units could be more precise relevant to substantive interpretations of the than an unpaired experiment of 160 units outcome (as would be the case with outcomes (although, of course, a paired experiment of measured before the treatment was assigned), 160 units would be yet more precise [the then we would not want such differences to average null SD in that case is 0.5]). Notice confuse treatment-versus-control differences that, in this case, the advantages of the paired of post-treatment outcomes. design diminish as the size of the experiment increases but do not disappear. One Graphical Mode of Assessment Pairing provides the largest opportunity for enhancement of precision in the compar- Figure 32.2 provides both some reassurance ison of two treatments – after all, it is hard to and some worry about balance on the baseline imagine a set more homogeneous than two outcome in the thirty-two-city data exam- subjects nearly identical on baseline covari- ple. The distributions of baseline turnout are ates (Imai, King, and Nall 2009). And, even nearly the same (by eye) in groups 2 and 3, if it is possible for an unpaired design to pro- but differ (again, by eye) in groups 1 and 4. duce differences of means with lower variance Should these differences cause concern? than a paired design, it is improbable given The top row of Table 32.1 provides one common political science measuring instru- answer to the question about worries about ments. random imbalance on baseline outcome. We What do we do once the experiment has would not be surprised to see a mean differ- =− run? One can enhance precision using covari- ence of dblocks 1.2 if there were no real dif- ates either via post-stratification (grouping ference given this design (p = .2). Of course, units with similar values of covariates) or questions about balance of a study are not covariance adjustment (fitting linear models answerable by looking at only one variable. with covariates predicting outcomes). In the Table 32.1 thus shows a set of randomization- first case, analyses proceeds as if prestratified: based balance tests to assess the null of no estimates of treatment effects are made con- difference in means between treated and conditional on the chosen grouping. In the sec- trol groups on covariates one by one and ond case, linear models “adjust for” covari- also all together using an omnibus balance ates – increased precision can result from test. We easily reject the null of balance on their inclusion in linear models under the the linear combination of these variables in same logic as that taught in any introduction this case (p = .00003), although whether we to statistics classes. worry about the evident differences on percent black or median income of the cities may depend somewhat on the extent to which 3. Balance Assessment: Graphs we fear that such factors matter for our out- and Tests come – or whether these observed imbalances suggest more substantively worrisome imbal- We expect that the bias reduction operating ances in variables that we do not observe. characteristics of random assignment would Does this mean the experiment is broken? make the baseline outcomes in the control Not necessarily.4 group comparable to the baseline outcomes in the treatment group. If the distribution of 4 It would be strange to see such imbalances in a a covariate is similar between treated and con- study run with thirty-two observations rather than trol groups, then we say that this covariate is an eight-observation study expanded to thirty-two “balanced” or that the experiment is balanced observations artificially as done here. These imbalances, however, dramatize and clarify benefits and with respect to that covariate. Yet, it is always dangers of post-treatment adjustment as explained possible that any given randomization might throughout this chapter. 466 Jake Bowers

Table 32.1: One-by-One Balance Tests for Covariates Adjusted for Blocking in the Blocked Thirty-Two-City Study x x¯Ctrl x¯Trt dblocks Var(dblocks)Std.dblocks zp

Baseline outcome 27.125.8 −1.20.9 −0.1 −1.4 .2 Percent black 17.32.6 −14.74.2 −1.2 −3.5 .0 Median HH income 30.646.315.73.22.64.9 .0 Number of candidates 13.210.3 −2.92.0 −0.5 −1.5 .1

Notes: Two-sided p values are reported in the p column referring z to a std. normal distribution, approx- imating the distribution of the mean difference under null of no effects. An omnibus balance test casts doubt on the null hypothesis of balance on linear combinations of these covariates at p = .00003.Test statistics (dblocks) are generalizations of Equations (1) and (4) developed in Hansen and Bowers (2008) and implemented in Bowers et al. (2009). Statistical inference (z and p values) is randomization based but uses large-sample normal approximations for convenience. Other difference-of-means tests without the large-sample approximation, and other tests such as Kolmogorov-Smirnov and Wilcoxon rank sum tests, provide the same qualitative interpretations. For example, the p values on the tests of balance for baseline outcome (row 1) ranged from p = .16 and p = .17 for the simulated and asymptotic difference of means tests, respectively, to p = .33 and p = .72 for exact and simulated Wilcoxon rank sum and Kolmogorov-Smirnov tests. 50 40 30 Baseline Turnout (%) 20

Ctrl.1 Trt.1 Ctrl.2 Trt.2 Ctrl.3 Trt.3 Ctrl.4 Trt.4 Treatment Assignment by Blocks Figure 32.2. Graphical Assessment of Balance on Distributions of Baseline Turnout for the Thirty-Two-City Experiment Data Notes: Open circles are observed values. Means are long, thick black lines. Boxplots in gray. Making Effects Manifest in Randomized Experiments 467

Table 32.2: One-by-One Balance Tests for Covariates in the Blocked Eight-City Study x x¯Ctrl x¯Trt dblocks Var(dblocks) Std.dblocks zp

Baseline outcome 26.024.8 −1.22.0 −0.1 −0.6 .5 Percent black 16.82.4 −14.411.0 −1.1 −1.3 .2 Median HH income 29.244.515.48.12.51.9 .1 Number of candidates 13.010.0 −3.05.1 −0.5 −0.6 .6

Notes: An omnibus balance test casts doubt on the null hypothesis of balance on linear combinations of these covariates at p = .41. Test statistics (dblocks) are generalizations of Equations (1)and(4) developed in Hansen and Bowers (2008) and implemented in Bowers et al. (2009). Statistical inference (z and p values) is randomization based, but uses large-sample normal approximations for convenience.

Understanding p Values in Balance Tests of balance, whereas in the eight-city example this difference casts less doubt. Now, eval- Say we did not observe thirty-two observa- uating the difference between controls and tions in sets of four but only eight observa- treatment on actual outcome in the eight- tions in pairs. What would our balance tests city case gives d = 1.5 and, under the report then? Table 32.2 shows the results pairs null of no effects, gives Var(d ) = 2.6 – for such tests, analogous to those shown in pairs and inverting this test leads to an approxi- Table 32.1. mate eighty-eight percent confidence inter- Notice that our p values now quantify val of roughly [−7,7]: the width of the confi- less doubt about the null. Is there something dence interval itself is about fourteen points wrong with randomization-based tests if, by of turnout. Even if percent black were a per- reducing the size of our experiment, we would fect predictor of turnout (which it is not, with change our judgment about the operation of a pair adjusted linear relationship of 0.07 in the randomization procedure? The answer 5 the eight-city case), the p value of .2 indicates here is no. that the relationship with treatment assign- The p values reported from balance tests ment is weak enough, and the confidence used here summarize the extent to which ran- intervals on the treatment effect itself would dom imbalance is worrisome. With a sam- be wide enough, to make any random imbal- ple size of eight, the confidence interval for ance from percent black a small concern. That our treatment effect will be large – taking is, the p values reported in Table 32.2 tell the pairing into account, a ninety-five per- us that random imbalances of the sizes seen cent interval will be on the order of ±3.5 (as here will be small relative to the size of the roughly calculated on the baseline outcomes confidence interval calculated on the treat- in Section 2). For example, both Tables 32.1 ment effect. With a large sample, a small and 32.2 show the block-adjusted mean dif- random imbalance is proportionately more ference in percent black between treated and worrisome because it is large relative to the control groups to be roughly 14.5 percentage standard error of the estimated treatment points. In the thirty-two-city example, this effect. Given the large confidence intervals on difference cast great doubt against the null a treatment effect estimated on eight units, 5 Echoing Senn (1994), among others, in the clinical the random imbalances shown here are less trials world, Imai, King, and Stuart (2008)provide worrisome – and the p values encode this some arguments against hypothesis tests for balance in observational studies. Hansen (2008) answers these worry just as they encode the plausibility of criticisms with a formal account of the intuition pro- the null. vided here, 1) pointing out that randomization-based Thus, even though our eyes suggested tests do not suffer the problems highlighted by those authors, and 2) highlighting the general role that p that we worry about the random imbalance values play in randomization-based inference. on baseline turnout we saw in Figure 32.2, 468 Jake Bowers that amount of imbalance on baseline out- linear model or a post-stratification); and 2) come is to be expected in both the thirty-two whether a specification search was conducted and eight-city cases – it is an amount of bias with only the largest adjusted treatment effect that would have little effect on the treatment reported, representing a particularly rare or effect were we to adjust for it. Of course, strange configuration of types of units. Such the omnibus test for imbalance on all four worries do not arise in the absence of adjust- covariates simultaneously reported in Table ment. Yet, if the analyst declines to adjust, 32.1 does cast doubt on the null of balance – then he or she knows that part of the treat- and the tests using the d statistics in Table ment effect in his or her political participa- 32.1 suggest that the problem is with per- tion study is due to differences in education, cent black and median household income thereby muddying the interpretation of his or rather than baseline outcomes or number of her study. candidates.6 One may answer such concerns by announcing in advance the variables for which random imbalance would be particu- 4. Covariates Allow Adjustment larly worrisome and also provide a proposed for Random Imbalance adjustment and assessment plan a priori. Also, if one could separate adjustment from estima- A well-randomized experiment aiming to tion of treatment effects, one may also avoid the problem of data snooping. For example, explain something about political participa- 2009 tion showing manifest imbalance on educa- Bowers and Panagopoulos ( ) propose tion poses a quandry. If the analyst decides a power analysis–based method of choosing to adjust, then he or she may fall under sus- covariance adjustment specifications that can picion: even given a true treatment effect of be executed independently of treatment effect zero, one adjustment out of many tried will estimation, and it is well known that one provide a p value casting doubt on the null may poststratify and/or match without ever of no effect merely through chance. With- inspecting outcomes. Post-stratification may out adjustment, we know how to interpret p also relieve worries about whether compar- values as expressions of doubt about a given isons adjusted using linear models are artifacts of the functional form (Gelman and Hill hypothesis: low p values cast more doubt than 2007 9 high p values. Adjustment in and of itself does , ch. ). not invalidate this interpretation: a p value There are two broad categories of sta- is still a p value. Concerns center rather on tistical adjustment for random imbalance: 1) whether an “adjusted treatment effect” is adjustment by stratification and adjustment substantively meaningful and how it relates using models of outcomes. In both cases, to different types of units experiencing the adjustment amounts to choice of weights, treatment in different ways – that is, the con- and adjustment may be executed entirely to cerns center on the meaning of “adjustment” enhance precision even if there is no appre- in the context of the adjustment method (a ciable evidence of random imbalance. Notice that the “unadjusted” estimate may already be an appropriately weighted combination of block-specific treatment effects – and that 6 If we had twenty covariates and rejected the null of to fail to weight (or “adjust”) for block- balance with p < .05, we would expect to falsely specific probabilities of treatment assignment see evidence of imbalance in one of twenty covari- will confound estimates of average treat- ates. Thus, Hansen and Bowers (2008)urgetheuse of an omnibus test – a test that assessess balance ment effects (if the probabilities of assign- across all linear combinations of the covariates in the ment differ across blocks) and decrease preci- table. Yet, the variable-by-variable display is useful sion (if the variation in the outcomes is much in the same way that graphs such as Figure 32.2 are useful – in suggesting (not proving) the sources of more homogeneous within blocks than across imbalance. blocks). Making Effects Manifest in Randomized Experiments 469

Post-Stratification Enables Adjustment but Did the post-stratification help with the Must Respect Blocking and Design balance problems with the census variables evident in Table 32.1? Table 32.3 shows Say, for example, that within blocks defined the strata-adjusted mean differences and p by town, the treated group on average con- value for balance tests now adjusting for the tained too many men (and that although post-stratification in addition to the blocking. gender was important in the study, the Balance on baseline turnout and number of researcher either could not or forgot to block candidates does improve somewhat with the on it within existing blocks). An obvious matching, but balance on percent black and method of preventing “male” from unduly median household income does not appre- confusing estimates of treatment effects is ciably improve. Notice a benefit of post- to only compare men to men, within block. stratification here: the postadjustment bal- Analysis then proceeds using the new set ance test shows us that we have two covariates of blocks (which represent both the pre- that we could not balance. treatment blocks and the new post-treatment Discussion of the advantages of blocking in strata within them) as before. Section 2 is, in essence, about how to analyze One may also use modern algorithmic blocked (pre- or poststratified) experimental matching techniques to construct strata. data. The rest of the chapter is devoted to Keele, McConnaughy, and White (2008) understanding what it is that we mean by argue in favor of matching over linear models “covariance adjusted treatment effects.” for covariance adjustment and show simulations suggesting that such post-stratification can increase precision. Notice that matching Linear Models Enable Adjustment to adjust experiments is different from match- but May Mislead the Unwary ing in observational studies: matching here must be done without replacement in order to Even with carefully designed experiments respect the assignment process of the experi- there may be a need in the analysis to make ment itself, and matching must be full. That some adjustment for bias. In some situations is, although common practice in matching where randomization has been used, there in observational studies is to exclude certain may be some suggestion from that data that observations as unmatchable or perhaps to either by accident effective balance of impor- reuse certain excellent control units, in a ran- tant features has not been achieved or that domized experiment every observation must possibly the implementation of the random- be retained and matched only once. This lim- ization has been ineffective. (Cox and Reid its the precision enhancing features of match- 2000, 29) ing (at least in theory) because homogene- Cox and Reid’s discussion in their sec- ity will be bounded first by the blocking tion 2.3 entitled “Retrospective Adjustment structure before random assignment and then for Bias” echoes Cox (1958, 51–52) and again by requiring that all observations be Fisher (1925). What they call “bias,” I think matched. might more accurately be called “random Figure 32.3 shows confidence intervals imbalance.” resulting from a variety of post-stratification Although Fisher developed the analysis of adjustments made to the thirty-two-city covariance using an asymptotic F test that turnout data. In this particular experiment, approximated the randomized-based results, the within-set homogeneity increase result- others have since noted that the standard ing from post-stratification did not outweigh sampling-based infinite population or normal the decrease in degrees of freedom occur- model theory of linear models does not jus- ring from the need to account for strata: tify their use in randomized experiments. For the shortest confidence interval was for the example, in discussing regression standard unadjusted data (shown at the bottom of the errors, Cox and McCullagh (1982) note, “It plot). 470 Jake Bowers

Percent black Strata Structure: [2:1(1),1:1(13),1:2(1)]

Baseline turnout w/ propensity penalty caliper Strata Structure: [1:1(16)] Mahalanobis distance: baseline turnout and number of candidates Strata Structure: [2:1(1),1:1(13),1:2(1)] Propensity score w/ propensity penalty caliper Strata Structure: [1:1(16)]

Median household income Strata Structure: [1:1(16)]

Propensity score: baseline turnout and number of candidates Strata Structure: [2:1(1),1:1(13),1:2(1)]

Baseline turnout Strata Structure: [2:1(1),1:1(13),1:2(1)]

None Strata Structure: [4:4(4)] −1012345 95% and 66% Confidence Intervals for Treatment Effect on Turnout (Percentage Points) Figure 32.3. Post-Stratification Adjusted Confidence Intervals for the Difference in Tumout between Treated and Control Cities in the Thirty-Two-City Turnout Experiment Notes: Each line represents the interval for the treatment effect after different post-stratification adjustments have been applied within the existing four blocks of eight cities. Thin lines show the ninety-five percent intervals. Thick lines show the sixty-six percent intervals. The propensity score was calculated from a logistic regression of treatment assignment on baseline turnout, number of candidates running for office, percent black, and median household income (both from the 2000 Census), and block indicators. All post-stratification and interval estimation was done with full, optimal matching using the optmatch (Hansen and Fredrickson 2010)andRItools (Bowers et al. 2009) packages for R. Numbers below the post-stratification label show the structure of the stratification: for example, without any post-stratification the experiment had four blocks, each with four treated and four control cities. The matching on absolute distance on the propensity score with a propensity caliper penalty produced sixteen pairs of treated and control cities (1:1(16)). The match on absolute distance on baseline turnout produced one set with two treated and one control (2:1(1)), thirteen paired sets (1:1(13)), and one set with one treated and two controls (1:2(1)). Because no observation could be excluded, calipers implied penalties for the matching algorithm rather than excluding observations from the matching (Rosenbaum 2010,ch.8). is known . . . that ‘exact’ second-order prop- as those used for balance assessment and erties of analysis of covariance for precision placebo tests. This method avoids most of improvement do not follow from random- the criticisms by Freedman (2008a, 2008b, ization theory” (547). In this section, I pro- 2008c), and thus suggests itself as useful in vide an overview of a randomization-based circumstances where the linear model as an method for covariance adjustment that can approximation may cause concern or perhaps use normal approximations in the same way as a check on such approximations. Making Effects Manifest in Randomized Experiments 471

Table 32.3: One-by-One Balance Tests Adjusted for Covariates, by Post-Stratiﬁcation in the Blocked Thirty-Two-City Study

Post Hoc Full Matching on: Blocks Baseline Turnout Propensity Score xdstrata pdstrata pdstrata p

Baseline outcome −1.2 .2 −1.2 .3 −1.2 .3 Percent black −14.7 .0 −14.7 .0 −14.7 .0 Median HH income 15.7 .015.8 .015.7 .0 Number of candidates −2.9 .1 −2.6 .3 −2.9 .3

Notes: Two-sided p values assess evidence against the null of no effects. An omnibus balance test casts doubt on the null hypothesis of balance on linear combinations of these covariates adjusted for blocks, and two kinds of post hoc full matching within blocks at p = .00003,.003,and.004. Strata adjusted mean differences (dstrata) are generalizations of Equation (1) developed in Hansen and Bowers (2008)and implemented in the RItools package for R (Bowers et al. 2009). Statistical inference (p values) is randomization based, but uses large-sample normal approximations for convenience. Post hoc stratiﬁcation results from optimal, full matching (Hansen 2004) on either absolute distance on baseline turnout or absolute distance on a propensity score with propensity caliper penalty shown in Figure 32.3.

First, though, let us get clear on what it lines). As ought to be clear here, parallel means to “adjust for” random imbalance on a effects is a required assumption for covariance covariate. adjustment to be meaningful. In this case, a regression allowing different slopes and intercepts between the treatment groups shows What Does It Mean to Say That an the treatment slope of 0.25 and the control Estimate has been “Adjusted” for the group slope of 0.23; thus, the assumption is “Covariance” of Other Variables? warranted. Let us look first at how covariance adjust- What about with blocked data? Figure ment might work in the absence of blocking 32.5 shows the covariance adjustment for by looking only at the first block of eight units three different covariates: 1) baseline out- in the thirty-two-city dataset. Figure 32.4 comes (on the left), 2) percent black (in the is inspired by similar figures in Cox (1958, middle), and 3) median household income (in ch. 4) with dark gray showing treated units $1,000s, on the right). In each case, the data and black showing control units. The unad- are aligned within each block by subtracting justed difference of means is 6.6 (the verti- the block mean from the observed outcome cal distance between the open squares that (i.e., block centered). A linear regression of are not vertically aligned on the gray vertical the block mean centered outcome on the line). The thick diagonal lines are the predic- block mean centered covariate plus the treat- tions from a linear regression of the outcome ment indicator is equivalent to a linear regres- on an indicator of treatment and baseline out- sion of the observed outcome on the covariate come. The adjusted difference of means is plus treatment indicator plus indicator vari- the vertical difference between the regres- ables recording membership in blocks (i.e., sion lines, here, five. If there had been no “fixed effects” for blocks). relationship between baseline outcomes and In the leftmost plot, we see that adjust- post-treatment outcomes, the regression lines ment for baseline outcomes in addition to would have been flat and the vertical distances the blocking structure does little to change between those lines would have been the same the treatment effect. In fact, these blocks as the unadjusted difference of means (the were chosen with reference to baseline out- thin dark gray and black horizontal dashed comes; thus, adjusting for blocks is roughly 472 Jake Bowers

Unadj. Means Adj. Means ferences of means are not identical to what is happening with an analysis of covariance (although they are close and provide helpful intuition in this case). The middle and right-hand plots show two instances in which the intuitions using differences of means become more difﬁcult to 1 2 4 6.6 5 believe. In strata , , and , every control unit has a higher percent black than every treated unit. The unadjusted average treatment effect is 1.8, but after adjustment for percent black 1 2

Outcome (Strata Mean Centered) the effect is . . The middle plot, however, shows that the assumption of parallel effects is

16 18 20 22 24 hard to sustain and that there is little overlap 17 18 19 20 21 22 in the distributions of percent black between Baseline Outcome (Strata Mean Centered) the treatment and control groups. In this case, Figure 32.4. Covariance Adjustment in a Simple however, the adjustment makes little differ- Random Experiment ence in the qualitative interpretation of the Notes: Dark gray and black circles show treated treatment effect. and control units, respectively. The unadjusted The right-hand figure is a more extreme difference of means is 6.6. The thick diagonal case of the middle figure. This time there is = β + β + β = lines are Yˆ i ˆ0 ˆ1 Zi ˆ2 yi,t−1 with Zi no overlap at all between the distributions {0,1} and the adjusted average treatment effect is of median income between the controls (in ˆ | = 1, = − ˆ | = 0, = (Y i Zi yi,t−1 y¯t1 ) (Y i Zi yi,t−1 black, and all on the left side of the plot) and = 5. y¯t1 ) the treated units (in dark gray, and all on the right side of the plot). The adjustment causes equivalent to adjusting for baseline outcomes the treatment effect to change sign: from 1.8 (only it does not require what is clearly a dubi- to −2.1 percentage points of turnout. Is −2.1 ous parallel lines assumption). If the parallel a better estimate of the treatment effect than lines assumption held in this case, however, 1.8? Clearly, median income has a strong rela- then we might talk meaningfully about an tionship with outcomes and also, via random adjusted effect. The average treatment effect imbalance, with treatment assignment (recall in the first block is 6.6, but the treated units the test casting doubt on the null of balance were more likely to participate, on average, in Table 32.1). than the control units even at baseline (a mean What is the problem with covariance difference of 4.1). Some of the 6.6 points adjustment in this way? First, as noted pre- of turnout may well be due to baseline dif- viously, the assumption of parallel lines is not ferences (no more than 4.1, we assume, and correct. Second, we begin to notice another probably less so, because it would only mat- problem not mentioned in textbooks such as ter to the extent that baseline turnout is also Cox and Reid (2000)orCox(1958) – ran- related by chance in a given sample to treat- dom assignment will, in large samples, ensure ment assignment). In this case, the block- balance in the distributions of covariates but specific relationship between baseline out- will not ensure such balance in small sam- comes and treatment assignment is vanish- ples. This means that the distributions of the ingly small (difference of means is 1.9), so covariates, in theory, ought to be quite similar only about two points of the average treat- between the two groups. However, the the- ment effect is due to the baseline treatment ory does not exclude the possibility of ran- effect (where “due to” is in a very specific lin- dom imbalance on one of many covariates, ear smoothed conditional means sense). The and it is well known that random imbalance adjusted effect is actually closer to 5 than 4.6 can and does appear in practice. Adjustments because the intuitions provided here with dif- for such imbalance can be done in such a Unadj. Means Adj. Means Unadj. Means Adj. Means Unadj. Means Adj. Means

1.8 2 1.8 1.2 1.8 −2.1 Outcome Outcome Outcome −4 −2 0 2 4 −4 −2 0 2 4 −4 −2 0 2 4

−4−20246 −20 −10 0 10 20 −10 −5 0 5 10 15 Baseline Outcome Percent Black Median HH Income/1000 Figure 32.5. Covariance Adjustment in a Blocked Random Experiment Notes: Dark gray and black circles show treated and control units, respectively. All variables are block mean centered. 473 474 Jake Bowers way that adjusted mean differences are still ance adjustment (whether for precision or for meaningful representations of the treatment random imbalance) means linear regression. effect (as shown by the adjustment for base- In theory, counterfactual statistical inference line outcomes in the left plot of Figure 32.5). using the linear regression model for covari- But, as the distributions of covariates become ance adjustment estimator is biased (Freed- more unbalanced, the covariance adjustment man 2008a, 2008b, 2008c); however, in prac- can mislead. It is hard to claim, based on tice, it is often an excellent approximation these data, that adjusting for median house- (Green 2009; Schochet 2009). What should hold income is a meaningful operation – one we do when we worry about the approxi- just cannot imagine (in these data) finding mation: for example, when the experiment is groups of treated and control units with the small, there is great heterogeneity in effects same median household income.7 and/or variance of effects across blocks, or Thus, we can see where some criticisms great heterogeneity or discreteness in the out- of covariance adjustment might easily come come (such that the central limit theorem from: covariance adjustment done with multi- takes longer to kick in than one would pre- ple regression without additional diagnostics fer)? Rosenbaum (2002a) presents a simple poses a real problem for diagnosing whether argument that builds on the basics of Fisher’s the imbalance is so severe as to provide no randomization-based inference. Here, I pro- “common support” in the distributions of vide some brief intuition to guide study of the covariates. In such cases, one really can- that paper. This method of randomization- not “adjust for” the imbalance and must be justified covariance adjustment does not rely resigned to the fact that treatment versus con- on the linear model for statistical inference, trol comparisons, in the data, reflect some- but does “adjust” using the linear model. thing other than treatment assignment even Say an outcome is measured with noise if they would not in a larger sample or across caused in part by covariates. When we ran- repeated experiments. Of course, as sample domly assign treatment, we are attempting to size grows, given a finite set of covariates and isolate the part of the variation in the outcome a treatment with finite variance (i.e., a treat- due to the treatment from that due to other ment that only has a few values and does not factors. Say we are still interested in the dif- gain values as sample size grows), we would ference in mean outcomes between treatment expect the problem of common support to and control groups as assigned. The standard disappear in randomized studies. Luckily, in deviations of those means may be large (mak- many studies, one can assess such problems ing the treatment effect hard to detect) or before treatment is administered. small (making the treatment effect more easily manifest). If part of the noise in the outcome is due to covariates, then the residual Randomization Alone Can Justify from regressing the outcome on the covari- Statistical Inference Covariance-Adjusted ates represents a less noisy version of the Quantities outcome – the outcome without noise from A predominant use for covariance adjustment linear relationships with covariates. This is not to ameliorate random imbalance but to residual eib (for unit i in block b)ismea- enhance statistical precision. To the extent sured in units of the outcome (i.e., “percent that a covariate predicts outcomes, one may turning out to vote” in our running fake use it to reduce the noise in the outcome data example). The potential outcomes to unrelated to treatment assignment and thus treatment and control for units i in blocks help make treatment effects manifest. Covari- b, yTib and yCib, are fixed, and Yib is random by virtue of its relationship with ran- = dom assignment Z because Yib ZibyTib 7 This problem also occurred in Figure 32.4, but was + (1 – Zib)yCib. A null hypothesis tells not mentioned in order not to detract from the pedagogical task of describing the mechanism of covari- us what function of Yi and Zi would ance adjustment. recover yCi: that is, if the null hypothesis Making Effects Manifest in Randomized Experiments 475

τ is correct, then removing the effect (say, ib) that this is a method of hypothesis testing, from the treated units Yib, Z = 1 would tell us not of estimation. It would be quite incor- how the treated units would behave under rect to interpret the difference of means of control. Under a constant, additive model of residuals as an estimate of a treatment effect = + τ τ = effects, yTib yCib and so Yib –Zib ib because the residuals already have specific 8 yCib . Thus, considering our null hypothe- causal hypotheses built into them as just τ = τ sis for the sake of argument, H0 : 0 ib, described. τ regressing (Yib –Zib ib)onxi is regressing Figure 32.6 shows that the Rosenbaum a fixed quantity (i.e., yCib) on another fixed style covariance adjustment in these data is quantity, xib, and so the residuals from that well approximated by the direct regression– regression are a fixed quantity.9 One may style covariance adjustment in the unadjusted substitute e for x in Equations (1) and (4). case or when the adjustment is made for Fisher’s style of inference begins with a test baseline turnout and number of candidates – of a null hypothesis and inverts the hypothe- and the version adjusted for baseline turnout sis for a confidence interval: thus, the method and number of candidates (just) excludes zero allows us to infer consistent estimates of the from its ninety-five percent confidence inter- causal effect by testing a sequence of causal val. The two approaches differ when the dif- hypotheses τ 0. Loosely speaking, the point ference of means is adjusted for the census estimate is the causal effect hypothesized by variables. The most notable difference here the best-supported hypothesis tested.10 Note is for median household income, where the direct adjustment method is based entirely on 8 If this model is incorrect, then randomization-based the linear extrapolation between the groups, inferences will be conservative, but the coverage of whereas the Rosenbaum approach correctly the confidence intervals will still be correct as noted captures the sense in which there is no rea- independently by Gadbury (2001) and Robins (2002, Section 2.1). Other substantively meaningful mod- sonable way to adjust the treatment effect for els of effects are available (Rosenbaum 2002a,section this covariate. Because adjustment for percent 6; 2002b,ch.5; 2010,ch.2). For example, as Rosen- black also requires much linear extrapolation, baum (2002c, 323) notes, if the treatment effect varies by a binary covariate x coding 0 for group 1 and the randomization-based confidence interval 1 for group 2 (such that the parallel lines assump- again reflects the fact that the design itself tion is incorrect), then we would specify the potential has little information about what it means to responses to control as Y –Z (τ =1x + τ =0(1 − ib ib x ib x adjust for this variable. xib)) for treatment effects that differ by group. I use the constant additive effects model in this chapter to The advantages of this style of covariance map most closely onto the causal quantities implied adjustment are 1) that it sidesteps Freed- by the choice of a linear regression model for covari- man’s critiques of covariance adjustment for ance adjustment: indeed, for this very reason, I show 11 how both styles of covariance adjustment can produce experiments; 2) although we used large- identical quantities in Figure 32.6. Interested readers sample normal approximations to evaluate might find the discussion in Rosenbaum (2002c, section 3–6) useful for thinking about the equivalence of estimating an average treatment effect and test- ciated randomization-based variances, including a ing a sequence of hypotheses about individual causal method for randomization-based covariance adjust- effects. ment. Because the Hansen and Bowers method 9 For a single covariate x and a regression fit (Y ib approximates the results of testing sequences of − ZibTib) = βˆ0 + βˆ1xib, eib = (Y ib − Zibτib) − (βˆ0 hypotheses with simple means and variances, their + βˆ1 X ib). The residual is written e,noteˆ, because method requires an asymptotic justification. Their the regression fit is not an estimate of an unknown article and related reproduction materials (Bowers, quantity, but merely calculating a function of fixed Hansen, and Fredrickson 2008) also provide tools for features of the existing data. assessing the asymptotic justification. 10 See discussion of the Hodges-Lehmann point esti- 11 In particular, Freedman (2008b, 189) notes that the = mate in Rosenbaum (2002a; 2002b,ch.2)formore Fisher-style covariance adjustment is valid. “If Ti β ≡ formal discussion of randomization-justified point Ci for all i (the “strict null hypothesis”), then estimates of causal effects. In the context of a large, 0 and adjustment will help – unless αZ = 0, i.e., cluster randomized field experiment with binary the remaining variation (in Ci) is orthogonal to the outcomes and nonrandom noncompliance, Hansen covariate.” Another method, elaborated in Hansen and Bowers (2009) show how one may approxi- and Bowers (2009), also does not directly model the mate the results of testing sequences of hypothe- relationship between treatment and outcomes and so ses with simple calculations of means and asso- similarly avoids this critique. 476

Baseline turnout, Number of Candidates, Rosenbaum Percent Black, Median HH Income Regression

Rosenbaum Median Household Income Regression

Rosenbaum Percent Black Regression

Rosenbaum None Regression

Rosenbaum Baseline turnout Regression

Baseline turnout and Rosenbaum Number of Candidates Regression

−15 −10 −5 0 5 10 95% and 66% Confidence Intervals for Treatment Effect (Percentage Points Difference in Turnout) Figure 32.6. Covariance Adjusted Confidence Intervals for the Difference in Turnout between Treated and Control Cities in the Thirty-Two-City Turnout Experiment Data Notes: “Regression”-style adjustment regressed turnout on treatment indicator, block indicators, and covariates and referred to the standard iid + t based reference distribution for confidence intervals. “Rosenbaum”-style adjustment regressed turnout on block indicators and covariates, and then used the residuals as the basis for tests of the null hypotheses implied by these confidence intervals. The reference distributions for the Rosenbaum-style are large-sample approximations to the randomization distribution implied by the design of the experiment using the RItools (Bowers et al. 2009) package for R. Thin lines show the ninety-five percent intervals. Thick lines show the sixty-six percent intervals. The randomization-based confidence intervals for outcomesafter adjustment by functions including median household income are essentially infinite. Making Effects Manifest in Randomized Experiments 477 the differences of means here, neither dif- cussed here entirely prevents such criticism. ferences of means nor large-sample normal Of course, the easy way to avoid such crit- approximations are necessary (and the large- icism is to announce in advance what kinds sample approximations are checkable)12; and of random imbalance are most worrisome 3) unmodeled heteroskedasticity or incorrect and determine a plan for adjustment (includ- functional form does not invalidate the statis- ing a plan for assessing the assumptions of tical inference based on this method as it does the adjustment method chosen). Covariance for the standard approach. For example, the adjustment using the standard linear regres- parallel lines assumption is no longer relevant sion model requires that one believe the for this approach because the only job of the assumptions of that model. For example, this linear model is to reduce the variance in the model as implemented in most statistical soft- outcome. Finally, this particular data exam- ware requires a correct model of the relation- ple allows us to notice another side benefit ship between outcomes and covariates among of the statistical property of correct coverage: treatment and control units (i.e., a correct when there is no (or little) information avail- functional form), that the heteroskedasticity able in the design, the randomization-based induced by the experimental manipulation is approach will not overstate the certainty of slight, and that the sample size is large enough conclusions in the same way as the model- to overcome the problems highlighted by based approach.13 Freedman (2008a, 2008b, 2008c). As with any use of linear regression, one may assess many of these assumptions. If one or more of these Best Practices for Regression-Based assumptions appear tenuous, however, then Adjustment this chapter shows that one may still use the Adjusted treatment effect estimates always linear model for adjustment, but do so in a invite suspicion of data snooping or mod- way that avoids the need to make such com- eling artifacts. None of the techniques dis- mitments. Readers interested in the Rosen- baum (2002a) style of covariance adjustment 12 Rosenbaum (2002a) focuses on normal approxima- should closely study that paper. The code tions to the Wilcox rank sum statistic as his preferred summary of treatment effects (and the normal contained in the reproduction archive for this approximations there are not necessary either, but chapter may also help advance understanding merely convenient and often correct in large enough of that method. samples with continuous outcomes). 13 The disadvantages of this mode are not on display here (although they will be obvious to those who use the code contained in this chapter for their own 5. The More You Know, the More work). First, remember that this approach produces confidence intervals by testing sequences of hypothe- You Know ses. It does not “estimate” causal effects as would a standard regression estimator, but rather assesses the A randomized study that allows “the phe- plausibility of causal effects using tests. Of course, such assessments of plausibility are also implicit in nomenon under test to manifest itself” pro- confidence intervals for standard regression estima- vides particularly clear information and thus tors. However, the mechanics of the two methods enhances theory assessment, theory cre- of covariance adjustment are quite different. The randomization-based approach as implemented here ation, and policy implementation. Thus, builds a confidence interval by direct inversion of researchers should attend to those elements hypothesis tests: in this case, we tested hypotheses of the design and analysis that would increase about the treatment effect from τ 0 = –20 to τ 0 = 20 by 0.1. This can be computationally burdensome the precision of their results. This chapter if the number of hypotheses to test is large or if we points to only a small part of the enormous eschew large-sample approximations. Second, we did body of methodological work on the design not work to justify our choice of mean difference (rather than rank or other summary of observed out- of experiments. comes and treatment assignment). The standard lin- Random assignment has three main sci- ear regression estimator requires attention to mean entific aims: 1) it is designed to bal- differences as the quantity of interest, whereas any test statistic may be used in the randomization-based ance distributions of covariates (observed method of adjustment shown here. and unobserved) such that, across repeated 478 Jake Bowers randomizations, assignment and covariates Given random imbalance, what should one should be independent; 2) it is designed to do? Adjustment can help, but adjustment can allow assessment of the uncertainties of esti- also hurt. This chapter showed (using a fake mated treatment effects without requiring dataset built to follow a real dataset) a case in populations or models of outcomes (or lin- which adjustment can help and seems mean- earity assumptions); and 3) it is a method of ingful and a case in which adjustment does manipulating putatively causal variables in a not seem meaningful, as well as an inter- way that is impersonal and thus enhances the mediate case. One point to take away from credibility of causal claims. Political scientists these demonstrations is that some imbal- have recently become excited about experi- ance can be so severe that real adjustment ments primarily for the first and third aims, is impossible. Just as is the case in observa- but they have ignored the second aim. This tional studies, merely using a linear model chapter discusses some of the benefits (and without inspecting the data can easily lead pitfalls) of the use of covariates in random- an experimenter to mislead him- or herself – ized experiments while maintaining a focus and problems could multiply when more than on randomization as the basis for inference. one covariate is adjusted for at a time. Rosen- Although randomization allows statisti- baum’s (2002a) proposal for a randomization- cal inference in experiments to match the based use of linear regression models is causal inference, covariate imbalance can and attractive in that covariance-adjusted con- does occur in experiments. Balance tests are fidence intervals for the treatment effect designed to detect worrisome imbalances. do not depend on a correct functional One ought to worry about random imbal- form for the regression model. In this paper, ances when they are 1) large enough (and rele- all of the adjustment for median house- vant enough to the outcome) that they should hold income depended on a functional form make large changes in estimates of treatment assumption, so the randomization-based con- effects, and 2) large relative to their baseline fidence interval was essentially infinite (sig- values such that interpretation of the treat- naling that the design of the study had ment effect could be confused. no information available for such adjust- Small studies provide little information to ment) while the model-based regression con- help detect either treatment effects or imbal- fidence interval, although much wider than ance. The null randomization distribution the unadjusted interval, was bounded. Mod- for a balance test in a small study ought to ern matching techniques may also help with have larger variance than said distribution this problem (Keele et al. 2008). In this chap- in a large study. The same observed imbal- ter, precision was not enhanced by match- ance will cast more doubt on the null of bal- ing within blocks, but matchings including ance in a large study than it will in a small median household income did not radically study: the observed value will be farther into change confidence intervals for the treat- the tail of the distribution characterizing the ment effect, and balance tests before and after hypothesis for the large study than it will matching readily indicated that the matchings in the small study. The same relationship did not balance median household income. between small and large studies holds when This chapter did not engage with some the test focuses on the treatment effect itself. of the other circumstances in which covari- Thus, a p value larger than some acceptance ate information is important for random- threshold for the null hypothesis of balance ized studies. In particular, if outcomes are tells us that the imbalance is not big enough missing, then prognostic covariates become to cause detectable changes in assessments ever more important in experimental stud- of treatment effects. A p value smaller than ies given their ability to help analysts build some acceptance threshold tells us that the models of missingness and models of out- imbalance is big enough to cause detectable comes (Barnard et al. 2003; Horiuchi, Imai, changes when we gauge the effects of treat- and Taniguchi 2007). Thus, I understate ment. the value of collecting more information Making Effects Manifest in Randomized Experiments 479 about one’s units. The trade-offs between Fisher, Ronald A. 1935. The Design of Experiments. collecting more information about units ver- Edinburgh: Oliver & Boyd. sus including more units in a study ought to be Freedman, David A. 2008a. “On Regression Ad- understood from the perspectives long artic- justments in Experiments with Several Treat- ments.” Annals of Applied Statistics 2: 176–96. ulated in the many textbooks on experimental 2008 design: simple random assignment of units to Freedman, David A. b. “On Regression Ad- justments to Experimental Data.” Advances in two treatments (treatment vs. control) can be Applied Mathematics 40: 180–93. a particularly inefficient research design. Freedman, David A. 2008c. “Randomization Does Not Justify Logistic Regression.” Statistical Sci- ence 23: 237–49. References Gadbury, Gary L. 2001. “Randomization Infer- ence and Bias of Standard Errors.” American Barnard, John, Constantino E. Frangakis, Jennifer Statistician 55: 310–13. L. Hill, and Donald B. Rubin. 2003. “Principal Gelman, Andrew, and Jennifer Hill. 2007. Data Stratification Approach to Broken Randomized Analysis Using Regression and Multi-Level/ Experiments: A Case Study of School Choice Hierarchical Models. New York: Cambridge Vouchers in New York City.” Journal of the University Press. American Statistical Association 98 (462): 299– Green, Donald P. 2009. “Regression Adjustments 324. to Experimental Data: Do David Freedman’s Bowers, Jake, Mark Fredrickson, and Ben Hansen. Concerns Apply to Political Science?” Unpub- 2009. RItools: Randomization Inference Tools. R lished paper, Yale University. package version 0.1-6 ed. Hansen, Ben, and Mark Fredrickson. 2010. Opt- Bowers, Jake, Ben B. Hansen, and Mark Fredrick- match: Functions for Optimal Matching, Including son. 2008. Reproduction achive for: ‘Attribut- Full Matching. R package version 0.6-1 ed. ing Effects to a Cluster Randomized Get- Hansen, Ben B. 2004. “Full Matching in an Obser- Out-the-Vote Campaign’. Technical Report, vational Study of Coaching for the SAT.” Jour- University of Illinois and University of nal of the American Statistical Association 99 (467): Michigan. 609–18. Bowers, Jake, and Costas Panagopoulos. 2009. Hansen, Ben B. 2008. “Comment: The Essential ‘“Probability of What?’: A Randomization- Role of Balance Tests in Propensity-Matched Based Method for Hypothesis Tests and Con- Observational Studies.” Statistics in Medicine 27: fidence Intervals about Treatment Effects.” 2050–4. Unpublished paper, University of Illinois, Hansen, Ben B., and Jake Bowers. 2008. “Covari- Urbana-Champaign.RetrievedJanuary24,2011 ate Balance in Simple, Stratified and Clus- from http://www.jakebowers.org/PAPERS/ tered Comparative Studies.” Statistical Science BowPanEgap2009.pdf. 23: 219–36. Brady, Henry E. 2008. “Causation and Expla- Hansen, Ben B., and Jake Bowers. 2009. “Attribut- nation in Social Science.” In Oxford Hand- ing Effects to a Cluster Randomized Get-Out- book of Political Methodology,eds.JanetM. the-Vote Campaign.” Journal of the American Box-Steffensmeier, Henry E. Brady, and David Statistical Association 104: 873–85. Collier. New York: Oxford University Press, Horiuchi, Yusaku, Kosuke Imai, and Naoko chap. 10, 217–70. Taniguchi. 2007. “Designing and Analyzing Cochran, William G., and Gertrude M. Cox. 1957. Randomized Experiments: Application to a Experimental Designs. New York: Wiley. Japanese Election Survey Experiment.” Amer- Cox, David R., and Peter McCullagh. 1982. “Some ican Journal of Political Science 51: 669– Aspects of Analysis of Covariance (with Discus- 87. sion).” Biometrics 38: 541–61. Imai, Kosuke, Gary King, and Clayton Nall. Cox, David R., and Nancy Reid. 2000. The Theory 2009. “The Essential Role of Pair Match- of the Design of Experiments. Boca Raton, FL: ing in Cluster-Randomized Experiments, with Chapman & Hall/CRC Press. Application to the Mexican Universal Health Cox, David R. 1958. The Planning of Experiments. Insurance Evaluation.” Statistical Science 24: 29– New York: Wiley. 72. Fisher, Ronald A. 1925. Statistical Methods for Imai, Kosuke, Gary King, and Elizabeth Research Workers. Edinburgh: Oliver & Boyd. Stuart. 2008. “Misunderstandings among 480 Jake Bowers

Experimentalists and Observationalists about Rosenbaum, Paul R. 2002a. “Covariance Adjust- Causal Inference.” Journal of the Royal Statis- ment in Randomized Experiments and Obser- tical Society, Series A 111: 1–22. vational Studies.” Statistical Science 17: 286–327. Keele, Luke J., Corrine McConnaughy, and Rosenbaum, Paul R. 2002b. Observational Studies. Ismail K. White. 2008. “Adjusting Experimen- 2nd ed. New York: Springer-Verlag. tal Data.” Unpublished Manuscript, The Ohio Rosenbaum, Paul R. 2002c. “Rejoinder.” Statistical State University. Science 17: 321–27. Leisch, Friedrich. 2002. “Dynamic Generation Rosenbaum, Paul R. 2010. Design of Observational of Statistical Reports Using Literate Data Studies. New York: Springer. Analysis.” In Compstat 2002 – Proceedings Rubin, Donald B. 1974. “Estimating Causal in Computational Statistics, eds. Wolfgang Effects of Treatments in Randomized and Haerdle and Bernd Roenz. Heidelberg, Ger- Nonrandomized Studies.” Journal of Educa- many: Physika Verlag. tional Psychology 66: 688–701. Leisch, Friedrich. 2005. Sweave User Manual. Rubin, Donald B. 1990. “Comment: Neyman Retrieved from http://www.stat.uni-muenchen. (1923) and Causal Inference in Experiments de/∼leisch/Sweave/Sweave-manual.pdf. and Observational Studies.” Statistical Science 5: Neyman, Jerzy. 1990. “On the Application of 472–80. Probability Theory to Agricultural Experi- Schochet, Peter. 2009. “Is Regression Adjustment ments. Essay on Principles. Section 9 (1923).” Supported by the Neyman Model for Causal Statistical Science 5: 463–80. Inference?” Journal of Statistical Planning and Nickerson, David W. 2005. “Scalable Protocols Inference 140: 246–59. Offer Efﬁcient Design for Field Experiments.” Sekhon, Jasjeet S. 2008. “The Neyman-Rubin Political Analysis 13: 233–52. Model of Causal Inference and Estimation Panagopoulos, Costas. 2006. “The Impact of via Matching Methods.” In Oxford Handbook Newspaper Advertising on Voter Turnout: of Political Methodology, eds. Janet M. Box- Evidence from a Field Experiment.” Unpub- Steffensmeier, Henry E. Brady, and David lished Manuscript, Fordhan University. Collier. New York: Oxford University Press, Robins, James M. 2002. “Covariance Adjustment chap. 11, 271–99. in Randomized Experiments and Observational Senn, Stephen J. 1994. “Testing for Baseline Bal- Studies: Comment.” Statistical Science 17: 309– ance in Clinical Trials.” Statistics in Medicine 13: 21. 1715–26. CHAPTER 33

Design and Analysis of Experiments in Multilevel Populations

Betsy Sinclair

Randomized experiments, the most rigorous effects most often occur as a result of social methodology for testing causal explanations transmission of the treatment, which is par- for phenomena in the social sciences, are exper ticularly likely when the treatment consists of iencing a resurgence in political science. The information. This chapter explores the impli- classic experimental design randomly assigns cations of these multilevel settings to high- the population of interest into two groups, light the importance of careful experimental treatment and control. Ex ante these two design with respect to random assignment of groups should have identical distributions in the population and the implementation of the terms of their observed and unobserved char- treatment. There are potential problems with acteristics. Treatment is administered based analysis in this context, and this chapter sug- on assignment, and by the assumptions of gests strategies to accommodate multilevel the Rubin causal model, the average effect of settings. These problems are most likely to the treatment is calculated as the difference occur in field settings where control is lack- between the average outcome in the group ing, although they can sometimes occur in assigned to treatment and the average out- other settings as well. Multilevel settings have come in the group assigned to control. the potential to disrupt the internal validity of Randomized experiments are often con- the analysis by generating bias in the estima- ducted within a multilevel setting. These set- tion of the average treatment effect. tings can be defined at the level at which There are two common environments the randomization occurs, as well as at the where the standard experiment fails to adjust level at which the treatment is both directly for a multilevel structure and results in prob- and indirectly administered. These indirect lematic estimates, and both occur where the assignment to treatment is at the individual The author wants to thank Donald Green, Jamie Druck- level, but the administration of treatment is man, Thomas Leeper, Jon Rogowski, John Balz, Alex not. Both of these instances create problems Bass, Jaira Harrington, and participants of the West Coast Experimental Political Science Conference for for analysis. The first problem relates to their helpful comments in improving this manuscript. the selection of groups to receive treatment.

481 482 Betsy Sinclair

Suppose the random assignment of a voter Many randomized field experiments occur in mobilization treatment is conducted at the situations where interference between indi- individual level, but the implementation viduals is likely, such as within social set- of the treatment is directed at the group tings where social interaction is expected. If level, and suppose that the selection of ignored, SUTVA violations have the possi- which groups to treat is not random but is bility of adding bias to estimated treatment instead selected by an organization where the effects, and it is possible that these biases can selection is correlated with individual voting go in either a positive or a negative direction. probabilities. This produces bias in the infer- Multilevel randomized experiments rely ences that result from this setup.1 An exam- on existing social structures, which have the ple of this type of experiment would be one potential to provide solutions to the problems in which the randomization assigns individu- that arise from social transmission. A mul- als to treatment and control but administers tilevel randomized field experiment design treatment to particular ZIP codes. Inferences allows for the opportunity to estimate the in this context are problematic because of the treatment effect while allowing for the pos- selection of particular ZIP codes. This prob- sibility of SUTVA violations as well as an lem is solved with clustered randomization explicit test for the degree to which social and by making the appropriate adjustment to interactions occur. Making SUTVA an object the standard errors (Green and Vavreck 2008; of study instead of an assumption has the ben- Arceneaux and Nickerson 2009). This is not efit of providing new insights about inter- the problem addressed in this chapter. personal influence. Multilevel experiments The second problem is in terms of social provide an opportunity to better understand interactions – we could have possible spillover social transmission of politics. Although theo- effects from one individual to another within ries abound about the structure of social envi- the same household or, furthermore, from ronments, little empirical evidence exists to one household to another within the same explicitly validate that these structures influ- group. That is, the random assignment is con- ence an individual’s politics. Multilevel set- ducted at the individual level, but the imple- tings occur when individuals are assigned mentation of the treatment is indirectly at the to treatment but communicate with each group level. Again, an example of this type other within different social levels, such as of experiment would be one in which the households or precincts. Multilevel settings randomization assigns individuals to treat- require specific experimental designs and ment and control but administers treatment analyses, but allow us to estimate the effects of to households. Inferences at the individual interpersonal interactions. This chapter both level are then problematic because the treat- describes the advantages of a multilevel ran- ment can be socially transmitted within the domized field experiment and provides rec- household. Social science randomized exper- ommendations for the implementation and iments often rely on treatments that can be analysis of such experiments consistent with socially transmitted. Social transmission has the current best practices in the literature. the potential to result in violation of one of the fundamental assumptions in the analysis of these experiments, the stable unit treat- 1. Spillovers, Models of Diffusion, ment value assumption (SUTVA). SUTVA and the Reflection Problem states that there is no interference between units, such that the assignment of an indi- Multilevel contexts highlight the role that vidual to the treatment group should have social interactions may play in political behav- no effect on outcomes for other individuals. ior. Turnout decisions may diffuse through a population in much the same way that dis- 1 Because not all individuals who are assigned to receive ease or trends are also transmitted across the treatment will actually be treated, there is also a loss of efficiency if the contact rates differ greatly individuals who are socially connected. Dif- across groups. fusion processes have been clearly seen in Design and Analysis of Experiments in Multilevel Populations 483 marketing effects (Coleman, Katz, and Men- from assignment by neighborhood, congres- zel 1966; Nair, Manchanda, and Bhatia 2008) sional district, or precinct, often because it and are likely to also be present in political is difficult to administer treatment by indi- behavior. Formal models of spillover effects vidual (Arceneaux 2005; King et al. 2007; in political behavior suggest that the ways in Imai, King, and Nall 2009). For example, which individuals are socially connected are in experiments on the effects of campaign highly likely to determine their beliefs about advertising, randomization often occurs at candidates (DeMarzo, Vayanos, and Zwiebel the level of the media market (Panagopoulos 2003; Sinclair and Rogers 2010). and Green 2006; Green and Vavreck 2008). These models are difficult to estimate Some experiments have failed at the individ- using observational data due to the reflection ual level – for example, experiments on pol- problem (Manski 1993). This is an identifi- icy such as antipoverty efforts (Adato, Coady, cation problem: it is difficult to separate the and Ruel 2000) and reductions in class size selection into group membership from the (Krueger 1999) – because the subjects were effect of the group itself. Thus, individual- able to change their own assignment category level behavior appears correlated with net- within a particular group; thus, some types of work peers when in fact network peers do experiments must be conducted at the clus- not cause this correlation. Instead, the cause ter level. If randomization occurs at the clus- of this correlation could be a common factor ter level, then without additional assumptions that affects all members of the group or char- the administration of treatment and infer- acteristics that the members of the group are ences about treatment efficacy will also be likely to share. Standard regression models done at the cluster level. One difficulty in are exposed to the reflection problem because evaluating the treatments in these contexts it is impossible to determine whether the indi- is that individuals may have communicated vidual is having an effect on the group or to each other about the treatment; thus, the vice versa. Alternative estimation strategies observed treatment effect may result from an do exist, which include the use of instrumen- interaction between the direct and indirect tal variables and the role of time (Brock and administration of treatment. This has impli- Durlauf 1999; Conley and Udry 2010). Yet, cations for both the external and internal observational strategies are not generally suf- validity of the experiment because it is not ficient to identify causal effects of social inter- possible to separate the direct and indirect action. treatment effects. By using randomized experiments, we are Randomization occurs at the level of able to gain some leverage on the reflection the individual as well, for example, in the problem by providing a stimulus to one set bulk of experiments on voter mobilization of individuals and then observing the behav- tactics (for a review of the literature, see ior of the group. In particular, by looking for Gerber and Green 2008). The majority of empirical evidence of spillovers in this setting, this chapter addresses issues involving the we are then more able to directly test to what estimation of direct and indirect treatment extent models of diffusion are applicable to effects when inferences are drawn about the political behavior. By using a multilevel ran- behavior of the individual. When estimat- domized experiment, we are able to directly ing treatment effects, one’s statistical model evaluate both the effect of the treatment and must account for the level at which ran- peer effects. This occurs via the identifica- domization occurs. Inferences are generally tion of the appropriate multilevel context and drawn about the behavior of the individ- is described in the next section. ual, and it is possible to create inconsisten- With limited exceptions, random assign- cies between the assignment of individuals ment is typically done at either the individual to treatment and the implementation of the or cluster level in randomized field experi- treatment. ments. There are many examples where ran- Implementation of the treatment may domization occurs at the cluster level, ranging occur indirectly as the result of social 484 Betsy Sinclair interaction. Many randomized experiments to how we draw causal inferences about the administer treatments that could be socially efficacy of the treatment. The first part of transmitted, such as political information, SUTVA assumes that there is no interfer- a heightened sense of political interest, or ence between units; that is, it assumes that increased social trust. Empirical work on there are no spillover effects.2 This chapter social networks has suggested that many of focuses exclusively on the problem of infer- these treatments can be socially transmit- ence between units as a result of treatment, ted (Fowler and Christakis 2008; Nicker- specifically treatment spillovers. son 2008; Cacioppo, Fowler, and Christakis It is possible that the units interfere 2009). Experiments where the treatment is with each other in the course of receiving subject to social transmission are implicitly treatment. For example, suppose that some conducted within a multilevel setting. individuals are contacted and given additional political information – it seems likely that they might then discuss this information 2. Stable Unit Treatment Value with the other individuals they know in their Assumption household or in their neighborhood. In this case, there would be interference between Within a social setting, it is possible that units. In this chapter, we investigate the ex- spillovers from individuals assigned to treat- tent to which we can measure potential ment to those assigned to control could cre- spillovers and also design experiments in ate biases in the estimation of treatment order to be able to correct for their potential effects, or vice versa. Here we consider the effects. We contend that spillover effects standard setup for a political interest experi- could exist within households or within ment and examine ways in which these biases groups. can be eliminated. Our narrative, consistent We now look at an example where, in the with work by Sinclair, McConnell, and Green presence of spillovers, it would not be possible (2010), considers an experimental population to ascertain the exact treatment effect if the where individuals are members of a three- randomization is conducted at the individ- level multilevel setting. Individuals are res- ual level and there is communication between idents in a household, and each household individuals within the experimental popula- resides within a social group. The collection tion. We explore spillovers both within their of groups forms the population. This exam- households and within their groups. That is, ple could generalize to any number of lev- suppose we have a violation of SUTVA. We els or different settings, as long as each sub- consider the violation in terms of the intent- level is fully contained within the previous to-treat (ITT) effect. level. Our primary concern with the assignment and implementation of randomized 3. Intent-to-Treat Effect field experiments within a multilevel population is violations of the SUTVA, as labeled In our population of n individuals, let each in Rubin (1980). Units here are defined as individual i be randomly assigned to treat- the unit that is being evaluated (e.g., if the ment, t = 1, or control, t = 0. We investigate treatment is being evaluated at the individual the outcome for each individual Yi. We want level, then the individual is the unit, whereas if to know what the difference is between treat- the treatment is being evaluated at the group ment and control – the treatment effect – and, level, then the group is the unit). ideally, we want to calculate Yi,t=1 – Yi,t=0. SUTVA states that the potential outcomes for any unit do not vary with the treatments 2 The second part of SUTVA assumes that the treat- assigned to any other units, and there are ment is the same for each unit: “SUTVA is violated when, for example, there exist unrepresented versions no different versions of the treatment (Rubin of treatments or inference between units” (Rubin 1990). The assumption of SUTVA is key 1986, 961). Design and Analysis of Experiments in Multilevel Populations 485

∗ ∗ Yet, it is not possible to observe both states ITT = 1/6 (Y1|t1 = 1, t2 = 1) + 1/6 ∗ of the world at the individual level. However, (Y2|t1 = 1, t2 = 1) + 1/6 (Y4|t4 = 1, t3 ∗ due to the random assignment, the group of = 0)–1/2 (Y3|t3 = 0, t4 = 1). For indi- individuals who receive treatment are ex ante viduals 1 and 2, they may be more likely identical in terms of their characteristics to to change their behavior because they both those who receive control, and we are able receive treatment, so this suggests that in fact to look at the difference in terms of expected we could overestimate the treatment effect. = outcomes. We define the ITT effect as ITT Yet, individual 3 may also be more likely to | = 1 | = 0 E(Yi t )–E(Yi t ). SUTVA allows us change behavior because he or she shares a to consider only the assignment of individu- household with someone who was also in als. If we assume that there may be spillovers, the treatment group, so this suggests that however, then we define the ITT effect in we could in fact underestimate the treatment terms of the assignment of each individual effect. i and all other individuals k. Formally, we Extending this example to groups, we can then observe the ITT effect for those would then have an even more complicated instances where i is socially connected to k problem where there could be communi- others who do not receive treatment – that is, cation between many households within a = | = 1 = 0 | = 0 ITT E(Yi ti , tk )–E(Yi ti , tk group. The true ITT would then need to be = 0 ). When individuals are communicating, written based on all instances of communica- we must revise our statements to include the tion. assignments for the other individuals in our The consequences of these spillovers are sample as well. In the limiting case, we must such that it is possible that in the presence revise our statement to describe all n individ- of communication, the estimated treatment ual assignments. effect can be either an overestimate or an Recall that each individual i is either underestimate of the true effect. The direc- = 1 assigned to treatment, where ti , or con- tion of the bias will depend on the ways in = 0 trol, where ti . We observe the out- which communication occurs and the effect of come for each individual i as Yi. To con- communication on an individual’s decision – sider a case where individuals are commu- it may be the case that additional communica- nicating within their households, suppose we tions between treated individuals, for exam- consider a case where all individuals live in ple, will heighten the probability that they two-person households and the second per- behave in a given way. Violations of SUTVA son in the household is identified as j.When may produce biased estimates in light of com- measuring the expected outcome, we have to munication about treatment. It is not possi- consider the assignment to the second person ble to know whether the direction of the bias | in the household as well, E(Yi ti,tj). Thus, to will be positive or negative prior to conduct- estimate the treatment effect, we need a paring the experiment. Key to estimating ITT is ticular group of individuals who have been to understand which individuals are assigned, assigned to treatment where the other indi- either directly or indirectly, to treatment. viduals in their household have been assigned = | = = to control, so that ITT E(Yi ti 1, tj 0)– | = = E(Yi ti 0, tj 0). Yet, the standard inference 4. Identifying a Multilevel Context would not have incorporated the treatment assignment of j, so there is a chance that the We identify two types of multilevel con- inferences could be biased due to commu- texts. The focus of this chapter is to identify nication between individuals. That is, sup- instances where individuals are likely to com- pose there are four individuals where three municate to each other about the treatment, are assigned to treatment and one is assigned but other scholars may use this phrase in a to control, but that we do not draw infer- different type of situation. First, multilevel ences based on a multilevel structure. Then contexts are likely to exist where the random- we could misestimate the treatment effect as ization is conducted at a different level from 486 Betsy Sinclair the level at which treatment is administered. The focus of this chapter is the second area This type of multilevel context has the poten- where multilevel contexts are likely to exist. tial to generate correlations within groups Multilevel contexts are likely to be present about the treatment. Experiments like these where the treatment consists of something occur most often when the experimenter is that can be communicated across social ties, relying on the multilevel structure in order to such as information. This type of randomized implement the treatment. Examples include experiment need only be considered where voter mobilization experiments conducted by individuals in the population of study are precinct or household. The design of these members of the same social structure. That experiments must account for this structure. is, instances, where, for example, it is possi- If it is the case that there is any failure to ble that an individual assigned to control and treat – that is, if it is possible that there will an individual assigned to treatment could be be some groups where no attempt is made to residents in the same household. This type administer the treatment – then it is helpful of multilevel context requires a very spe- that the order of the attempts to contact each cific design because the treatment has the group be randomized.3 This randomization potential to be indirectly administered at the both allows for valid causal inferences and group level.5 This case has the potential to makes it impossible for the selection of par- be extremely problematic for drawing valid ticular groups to undermine the randomiza- causal inferences without additional adjust- tion. As long as all units will be treated, then ment. This case has the potential to violate the key in drawing inferences in these cases SUTVA. is that if treatment is administered at a differ- Empirically, scholars have observed social ent level than that of the randomization, the spillover, which could generate SUTVA vio- inferences must adjust for this correlation.4 lations in the classic experimental framework. For an example of within-household interfer- 2008 3 Failure-to-treat problems present challenges for ence, Nickerson ( ) finds higher levels of analysis, some of which can be mitigated via ran- turnout in two-person households when one domization inference (Hansen and Bowers 2008). of the individuals is contacted to get out to 4 If inconsistencies exist between the random assignment and the administration of the treatment, then vote via door-to-door canvassing in a voter we recommend two strategies for analysis. Suppose mobilization experiment. In this instance, that an experiment has been conducted where the there is interference within the households. randomization occurred at the individual level, but the treatment was administered at the group level. Nickerson finds that sixty percent of the In this case, we first recommend clustering the stan- increased propensity to vote can be passed dard errors at the group level when estimating ITT. onto the other member of the household – a This clustering explicitly acknowledges the correlation that is likely to exist within the group as a result precise measurement of treatment spillover. of the administration of the treatment and adjusts for Green, Gerber and Nickerson (2003) find the lack of independence of all observations within 2008 within-household spillover effects from door- the group (Green and Vavreck ; Arceneaux and 5 7 Nickerson 2009). Note, however, that this adjust- to-door canvassing: an increase of . per- ment is not sufficient to account for the potential centage points for other household members biases that have occurred as a result of social inter- among households of younger voters. In one actions. Correlation in the standard errors does not account for the possibility that individuals who are of the earliest mobilization experiments, the assigned to treatment, for example, may have been Erie County Study reported that although indirectly treated multiple times from other individuals assigned to treatment or the possibility that individuals assigned to control may have been indi- 2008; Sinclair, McConnell, and Michelson 2010). rectly treated from individuals assigned to treatment. Although this is more often a concern when there Our second recommendation, if there are a sufficient are failure-to-treat instances, there is still likely to be number of group-level observations, is to conduct group-level variation that is not properly accounted analysis either via a hierarchical linear model or via for via fixed effects. meta-analysis. Meta-analysis, for example, does not 5 If we conduct both our randomization at the group require homoscedasticity across groups, which is a level and our analysis on the group level, then this case necessary assumption with the inclusion of group- requires no additional shifts in experimental design level fixed effects, which assumes that there is a and is in fact eligible for the block group randomiza- group-specific effect (Gerber, Green, and Larimer tion (King et al. 2007). Design and Analysis of Experiments in Multilevel Populations 487 eight percent of Elmira, New York, resi- on geography requires no additional acquisi- dents had been contacted, turnout increased tion of data and also allows the researchers by ten percent, suggesting that mobilization to investigate to what extent there is spillover contact was socially transmitted (Berelson, within an individual’s physical context. The Lazarsfeld, and McPhee 1954).6 Other schol- choice of each of these methods – explicit ars have examined spillover effects in contexts observation, survey, and geography – should unrelated to political behavior (Besley and in large part be based on the type of treatment Case 1994; Miguel and Kremer 2004; Munshi administered. 2004; Duflo, Kremer, and Robinson 2006). Each instance would generate a SUTVA violation. 5. Designing a Multilevel Experiment To correctly identify instances where a multilevel structure exists to correct the The problem generated by communication of experimental design to adjust for potential the treatment with participants in the exper- SUTVA violations, it is necessary to have iment impels us to generate an alternative additional information about the population experimental design that relies on our knowl- of interest. There are several possibilities for edge of an individual’s social structure. A identifying multilevel social structures. The multilevel experimental design, where ran- first is to explicitly observe the level at which domization is conducted within an individ- social interactions occur. For example, for ual’s social structure, is the best approach for researchers conducting experiments where establishing causal inferences. Our proposed they have clear and explicit knowledge of an solution is to introduce additional random- individual’s full network (e.g., if the experi- ization, consistent with recommendations by ment was conducted via the social networking Sinclair, McConnell, and Green (2010). With web site Facebook), it is possible to explicitly these additional randomizations, we are able conduct randomizations across separate com- to establish via the Rubin potential outcomes ponents of personal networks so as to ensure framework a treatment framework that can against spillover. However, most experimen- identify both the spillovers and the direct tal frameworks do not allow for this type treatment effect. These additional random- of explicit specification of the full network izations are key to designing a multilevel structure. An alternative method for obser- experiment. vation is for researchers to rely on survey The first component of such a design is results where individuals self-identify their to identify all levels at which individuals will social ties. Randomization can then occur indirectly interact. For purpose of example, within an individual’s self-identified social we consider a population where individuals relationships. Survey data that solicit an indi- are likely to interact with each other on two vidual’s discussion partners – people with levels, household and group. We require that whom they are likely to communicate with these groups must be subsets of each other, about the treatment and thus where spillovers so that each individual in the population is are likely to occur – have demonstrated that part of exactly one group and one household. many of these discussion partners are geo- The social structure of our population can graphically proximate (Huckfeldt, Johnson, be seen in Figure 33.1. Key to this analysis and Sprague 2004;McClurg2006). Thus, the is to incorporate the full set of social struc- final method for observation of an individ- tures, where interactions are likely to occur. ual’s social structure is geography. Relying In our example, we model this as the group, but these group-level interactions could truly 6 The assumptions about social transmission of political information and positive and significant effects of be at the neighborhood level, the school dis- peers on individual political behavior date back to the trict level, or the city level. There must exist Erie County Study of 1940 and the Elmira Commu- a group level at which individuals will not nity Study of 1948, some of the earliest quantitative work in political science (Lazarsfeld, Berelson, and interact but levels where the experiment will Gaudet 1948; Berelson et al. 1954). take place; furthermore, each level must be 488 Betsy Sinclair

Population

Randomization Treatment Control Groups Groups

Randomization Treatment Control Households Households

Randomization

Treatment Control Individuals Individuals

These individuals actually receive treatment. Figure 33.1. Multilevel Experiment Design distinct – that is, an individual cannot belong Inferences drawn from experimental data that to multiple groups. One way to ensure this have not explicitly incorporated spillovers, is possible is to carefully select the experi- but where contagion of the treatment is likely, mental population; if an individual shares a may have greater problems with external household with individuals who are not part validity – the same treatment, administered in of the experiment, then it is not necessary to a different social context, is unlikely to gener- account for their presence in the estimation ate the same effect. However, if inferences are of the treatment effect, so individuals should presented at the group level where the poten- be eligible to participate in the experimental tial for spillovers is acknowledged but not population if their group memberships can incorporated into the experimental design, be summarized by the appropriate random- then in other contexts where the social struc- ization levels. It is also possible to increase tures are similar the treatment effects should the number of randomizations to include, as be similar. Thus, the caveat for the researcher an additional category, those individuals who should be to acknowledge the potential con- belong to multiple groups, although this sig- cern with external validity and to present niﬁcantly increases the number of individuals the group-level treatment effects at the level necessary to incorporate into the experiment. above which the social interactions have Identifying these social structures and iden- occurred. tifying the relevant experimental population In our example, we randomly assign is key to the design of a successful multilevel groups in the population to treatment and experiment. control. Within the groups assigned to Multilevel experimental design increases treatment, we then again repeat the ran- both internal and external validity. By dom assignment, randomly assigning house- acknowledging the presence of these social holds to treatment and control. Finally, structures, we increase the internal validity within the households assigned to treat- of our inferences. If it is not possible to ment, we again repeat the random assign- identify these social structures or to conduct ment, randomly assigning individuals to an experiment that incorporates variation in treatment and control. Treatment is admin- these structures, then the researcher needs to istered at the individual level to all mem- think carefully about the type of inference bers of the third stage of the randomization that is drawn from the analysis of these data. who are assigned to treatment. We conduct Design and Analysis of Experiments in Multilevel Populations 489

Table 33.1: Intent-to-Treat Effects

Assignment Assignment (Group, Household, Individual) (Group, Household, Individual) Quantity of Interest

Control, Control, Control Treatment, Control, Control Group-Level Spillover Treatment, Control, Control Treatment, Treatment, Household-Level Spillover Treatment, Treatment, Control Treatment Effect (Potentially Control Treatment, Treatment, Treatment Biased)

randomization at each level where we antic- Where the experiment has incorporated ipate social interactions will occur. Sinclair, the multilevel context into the experimen- McConnell, and Green (2010) follow this tal design, we recommend two strategies for recommendation for experimental analysis in analysis. First, we use the multiple random Los Angeles and Chicago. This process allows assignment variables to detect the presence for the identification of both the spillover of any social spillovers within our explicitly effects and the true treatment effect while specified social contexts. This allows us to removing the potential bias associated with evaluate whether there is enough evidence the spillover. Suppose that the true ITT is α, to reject the null that, for example, there but that we have positive indirect treatment are no household-level spillovers. If we can effects from individuals who receive treat- reject this null hypothesis, then this suggests ment to other individuals within their house- that in fact we do have spillovers – a clear hold that are equal to β and positive indi- test that then implies that we need to incorrect treatment effects from individuals who porate the multilevel structure in any addi- receive treatment to other individuals within tional analyses. If we reject this null, then their group that are equal to δ. A comparison we recommend a second strategy for anal- of individuals from those assigned to each cat- ysis. In this second stage, we estimate the egory in Figure 33.1 will enable us to estimate quantities resulting from these different ran- each of these three quantities. The SUTVA dom assignments. That is, again suppose that violations have become a quantity of inter- thetrueITTisα but that we have posi- est, allowing for inferences on interpersonal tive indirect treatment effects from individ- interactions. uals who receive treatment to other individuals within their household that are equal to β and positive indirect treatment effects from 6. Empirical Tests of Spillovers individuals who receive treatment to other and Estimation of the Intent-to- individuals within their group that are equal Treat Effect to δ. In the estimation of the ITT, it is then necessary to incorporate parameters to sep- α β δ Here we provide recommendations for esti- arately estimate , , and – that is, the mation of the ITT effect. These recommen- effects of each level of assignment. A visual dations are not statistical corrections in the demonstration for each estimate is included in 33 1 absence of design approaches, but instead are Table . . strategies for situations where the experimen- That is, consistent with our example, we tal design has explicitly adjusted for spillover. then need to estimate the effect of hav- These strategies are simple to adopt and ing been assigned to group-level treatment require minimal assumptions. but not household-level or individual-level 490 Betsy Sinclair treatment compared to the group-level con- group and thus cannot be reasonably control. We also estimate the effect of having ducted on a multilevel scale is to present been assigned to household-level treatment the group-level effects and to acknowledge but not individual-level treatment compared the potential presence of spillovers. Spillovers to household-level control and finally the can take on multiple forms; some individuals effect of having been assigned to individual- may be able to be treated multiple times, and level treatment compared to individual-level it is possible that both individuals assigned control. In Table 33.1, the difference between to treatment and control may be subject to the two columns produces the quantity of spillovers. We anticipate that most spillovers interest. If either of the first two rows pro- occur when the treatment group contacts the duces statistically significant effects, then the control group, but there are other kinds of sit- final row is likely to be biased. Thus, an esti- uations where the treatment group may also mate of the true treatment effect would sub- be indirectly treated. This makes it difficult to tract each of these spillover effects from the calculate the appropriate counterfactuals for final row to estimate the true ITT. how large a potential spillover effect could have been present with observed experimental data. Given the lack of knowledge about 7. Limitations and Best Practices the direction of the bias the spillovers could take, it is impossible to ex ante predict the There are two limitations that emerge from effects of potential SUTVA violations. Under conducting multilevel experiments. The first certain kinds of spillovers, the estimates could is simply the challenge in identifying the in fact converge to the actual true value of the levels at which social interactions are likely treatment effect, for example. to occur. Given that many experiments may One final limitation of the multilevel con- take advantage of geography, however, within text, albeit more applicable where there are which to conduct the experiment, this may failure-to-treat cases, is that the bias and loss not be a large hurdle for many types of exper- of efficiency from using instrumental vari- iments. Geography is likely to be a weak proxy ables when the contacts are made uniformly for a social environment; the best experiments across all groups is different than when the would be conducted using a preexisting social contacts are concentrated in some groups, as network. The second limitation that emerges is the potential case in the multilevel context. is that the construction of a multilevel experiment may require a larger experimental population in order to have sufficient statistical 8. Recommendations for power to identify the spillover effects. Thus, Experimental Design and Future the additional limitation is the loss of effi- Research ciency that emerges with the construction of multiple control groups. In many popula- We recommend that in multilevel contexts tions, this is not a concern because there are randomization occur not only at the indi- sufficiently many individuals that the addi- vidual level, but also at all appropriate social tion of more control group members does not structure levels. If treatment is then admin- limit the feasibility of the experiment. Yet, istered at the individual level, then it is there may be contexts within which either it clear how to draw inferences about spillovers, is difficult to locate additional control group allowing us great insight into the way in which participants or where the requirements of politics can be socially transmitted and the additional control groups that are geographi- role of interpersonal influence. In this sense, cally dispersed increases the cost of conduct- SUTVA violations have become a quantity of ing the experiment. interest. The best practice when the experiment is Most important, however, is the detailed limited by the potential size of the control exposition of the randomization in multilevel Design and Analysis of Experiments in Multilevel Populations 491 contexts. To the extent that it is possible, ment within the experimental population, as researchers must document whether their the assignment variable fails to capture the experimental treatments are administered at indirect treatment. a group level and acknowledge that these Multilevel experiments have the poten- strategies require shifts in their estima- tial to yield great insights into the ways in tion procedures. Furthermore, if researchers which humans interact; with careful experi- believe that their experiment is operating mental design, the SUTVA violations have within a multilevel context, then it is key that the potential to open up new avenues of this be documented so that future researchers research previously reliant on heroic assump- can incorporate these facts into future exper- tions. Each additional randomization does iments. not add to the technical difficulty of imple- Although much of this chapter has been menting the experiment because it is still pos- written from the perspective of field experi- sible for the experimental design to include ments, it animates much of the work done in the same number of participants assigned to the survey world (Stoker and Bowers 2002). be administered the treatment. It is the addi- There are many situations where experiments tion of the new control groups that allows are conducted within multilevel settings. for the identification of the spillover effects. Voter mobilization experiments, where treat- Researchers should be aware of the statistical ment and randomization occur at the level power to detect spillovers. Meta-analysis may of the individual, are clearly prey to poten- subsequently reveal spillovers, even if individ- tial SUTVA violations. Within existing social ual studies are inconclusive. and political organizations, there may be hier- This methodological improvement has the archical or geographically distributed groups potential to encourage different kinds of that are also subject to potential SUTVA inferences in randomized field experiments. violations, such as statewide organizations This is also a technique that allows the dis- with local chapters and individual mem- covery of supportive evidence for individuals bers. Clearly, these concerns are relevant for who study network analysis via survey data to experiments conducted on college campuses, understand social structure. This method can where individual participants may be resi- also be extended to include additional ran- dents in the same dorm, for example. Exper- domizations to study spillover in many direc- iments conducted using the multilevel ran- tions – for example, we could also include domization design will allow social science to a category where we compared individuals develop an empirical knowledgebase for how assigned to treatment who were paired with much spillover actually does occur and how control to individuals assigned to treatment much potential bias there could be. At this who were paired with treatment to individu- point, our collective knowledge of spillover is als assigned to control – this would allow us to fairly empty, and we do not know under what see if in fact the pairing of treated with treated conditions spillovers are likely to occur. would increase the effect of the treatment as Experimenters need to be sure to con- well. SUTVA violations have the potential to sider failure-to-treat situations and the ways be extremely interesting quantities of inter- in which they may further complicate these est. As we develop new and interesting ways to analyses. This chapter has not dealt explic- measure spillovers, these quantities will allow itly with failure to treat, but these instances us to inform which types of theories are most require additional assumptions under which applicable in the social transmission of poli- to draw inferences. In particular, many kinds tics. We do not yet know whether the instiga- of analyses use the random assignment vari- tion of political behavior is due to generated able as an instrumental variable in order conversations, heightened interest, or persua- to estimate the treatment-on-treated effect. sion. The measurement of spillover will offer This approach is not appropriate in cases one set of illustrations for where our theories where there is social transmission of treat- should focus. 492 Betsy Sinclair

References Gerber, Alan S., Donald P. Green, and Christo- pher W. Larimer. 2008. “Social Pressure and Adato, Michelle, David Coady, and Marie Ruel. Voter Turnout: Evidence from a Large-Scale 2000. An Operations Evaluation of PROGRESA Field Experiment.” American Political Science 94 653 63 from the Perspective of Beneficiaries, Promotoras, Review : – . School Directors and Health Staff. Washington, Green, Donald P., Alan S. Gerber, and David 2003 DC: International Food Policy Research Insti- W. Nickerson. . “Getting Out the Vote tute. in Local Elections: Results from Six Door-to- Arceneaux, Kevin. 2005. “Using Cluster Random- Door Canvassing Experiments.” Journal of Pol- 65 1083 96 ized Field Experiments to Study Voting Behav- itics : – . 2008 ior.” Annals of the American Academy of Political Green, Donald P., and Lynn Vavreck. . and Social Science 601: 169–79. “Analysis of Cluster-Randomized Experi- Arceneaux, Kevin, and David W. Nickerson. 2009. ments: A Comparison of Alternative Estima- 16 138 52 “Modeling Certainty with Clustered Data: A tion Approaches.” Political Analysis : – . 2008 Comparison of Methods.” Political Analysis 17: Hansen, Ben B., and Jake Bowers. . “Attribut- 177–90. ing Effects to a Cluster Randomized Get- Berelson, Bernard, Paul F. Lazarsfeld, and Out-the-Vote Campaign.” Journal of the Amer- 104 487 873 William N. McPhee. 1954. Voting. Chicago: ican Statistical Association ( ): – 85 The University of Chicago Press. . Besley, Timothy, and Anne Case. 1993. “Mod- Huckfeldt, Robert, Paul E. Johnson, and John 2004 eling Technology Adoption in Developing Sprague. . Political Disagreement.Cam- Countries.” American Economic Review 83: 396– bridge: Cambridge University Press. 402. Imai, Kosuke, Gary King, and Clayton Nall. 2009 Brock, William A., and Steven N. Durlauf. 1999. . “The Essential Role of Pair Matching in “A Formal Model of Theory Choice in Sci- Cluster-Randomized Experiments, with Appli- ence.” Economic Theory 14: 113–30. cation to the Mexican Universal Health Insur- 24 29 53 Cacioppo, John T., James H. Fowler, and Nicholas ance Evaluation.” Statistical Science : – . A. Christakis. 2009. “Alone in the Crowd: King, Gary, Emmanuela Gakidou, Nirmala Rav- The Structure and Spread of Loneliness in a ishankar, Ryan T. Moore, Jason Lakin, Manett Large Social Network.” Journal of Personality Vargas, Martha Maria Tellez-Rojo, Juan Euge- and Social Psychology 97: 977–91. nio Hernandez Avila, Mauricio Hernandez 2007 Coleman, James S., Elihu Katz, and Herbert Avila, and Hector Hernandez Llamas. .“A Menzel. 1966. Medical Innovation. Indianapolis: ‘Politically Robust’ Experimental Design for Bobbs-Merrill Press. Public Policy Evaluation, with Application to Conley, Timothy G., and Christopher R. Udry. the Mexican Universal Health Insurance Pro- 2010. “Learning about a New Technology: gram.” Journal of Policy Analysis and Management 26 479 509 Pineapple in Ghana.” American Economic Review : – . 1999 100: 35–69. Krueger, Alan B. . “Experimental Estimates DeMarzo, Peter M., Dimitri Vayanos, and Jeffrey of Education Production Functions.” Quarterly 114 497 532 Zwiebel. 2003. “Persuasion Bias, Social Influ- Journal of Economics : – . ence and Unidimensional Opinions.” Quarterly Lazarsfeld, Paul, Bernard Berelson, and Hazel 1948 Journal of Economics 118: 909–68. Gaudet. . The People’s Choice. New York: Duflo, Esther, Michael Kremer, and Jonathan Columbia University Press. 1993 Robinson. 2006. “Understanding Technol- Manski, Charles. . “Identification of Exoge- ogy Adoption: Fertilizer in Western Kenya.” nous Social Effects: The Reflection Problem.” 60 531 42 Unpublished manuscript, Massachusetts Insti- Review of Economic Studies : – . 2006 tute of Technology. McClurg, Scott. . “Political Disagreement in Fowler, James H., and Nicholas A. Christakis. Context: The Conditional Effect of Neighbor- 2008. “Dynamic Spread of Happiness in a hood Context, Discussion, and Disagreement Large Social Network: Longitudinal Analysis on Electoral Participation.” Political Behavior 28 349 66 over 20 Years in the Framingham Heart Study.” : – . 2004 British Medical Journal 337:a2338. Miguel, Edward, and Michael Kremer. . Gerber, Alan S., and Donald P. Green. 2008. Get “Worms: Identifying Impacts on Education Out the Vote! 2nd ed. Washington, DC: Brook- and Health in the Presence of Treatment 72 159 217 ings Institution Press. Externalities.” Econometrica : – . Design and Analysis of Experiments in Multilevel Populations 493

Munshi, Kaivan. 2004. “Social Learning in a Het- and Causal Inferences.’” Journal of the Amer- erogeneous Population: Technology Diffusion ican Statistical Association 81: 961–62. in the Indian Green Revolution.” Journal of Rubin, Donald B. 1990. “Formal Modes of Sta- Development Economics 73: 185–213. tistical Inference for Causal Effects.” Journal of Nair, Harikesh, Puneet Manchanda, and Tulikaa Statistical Planning and Inference 25: 279–92. Bhatia. 2008. “Asymmetric Social Interactions Sinclair, Betsy, Margaret A. McConnell, and Don- in Physician Prescription Behavior: The Role ald P. Green. 2010. “Detecting Spillover in of Opinion Leaders.” SSRN Working Paper Social Networks: Design and Analysis of Mul- 937021. tilevel Experiments.” Unpublished manuscript, Nickerson, David. 2008. “Is Voting Contagious? The University of Chicago. Evidence from Two Field Experiments.” Amer- Sinclair, Betsy, Margaret A. McConnell, and ican Political Science Review 102: 49–57. Melissa R. Michelson. 2010. “Strangers vs Panagopoulos, Costas, and Donald P. Green. Neighbors: The Efﬁcacy of Grassroots Voter 2006. “The Impact of Radio Advertisements Mobilization.” Unpublished manuscript, The on Voter Turnout and Electoral Competi- University of Chicago. tion.” Paper presented at the annual meeting Sinclair, Betsy, and Brian Rogers. 2010. “Polit- of the Midwest Political Science Association, ical Networks: The Relationship between Chicago. Candidate Platform Positions and Con- Rubin, Donald B. 1980. “Randomization Analysis stituency Communication Structures.” Unpub- of Experimental Data: The Fisher Randomiza- lished manuscript, The University of Chicago. tion Test Comment.” Journal of the American Stoker, Laura, and Jake Bowers. 2002. “Design- Statistical Association 57 (371): 591–93. ing Multilevel Studies: Sampling Voters Rubin, Donald B. 1986. “Which Ifs Have Causal and Electoral Contexts.” Electoral Studies 21: Answers? Discussion of Holland’s ‘Statistics 235–67. CHAPTER 34

Analyzing the Downstream Effects of Randomized Experiments

Rachel Milstein Sondheimer

The work in this volume provides profound of time frame and subject matter, as well evidence of the value of randomized experi- as high in cost. Although short-term exper- mentation in political science research. Lab- iments may incur costs similar to observa- oratory and field experiments can open up tional research, they often focus on a single new fields for exploration, shed light on or just a few variations in an independent vari- old debates, and answer questions previously able, seemingly limiting their applicability to believed to be intractable. Although acknowl- a breadth of topics that a survey could cover. edgment of the value of experimentation The high cost associated with long-term in political science is becoming more com- data collection and the necessity of main- monplace, significant criticisms remain. The taining contact with the subjects involved oft-repeated shortcomings of experimental impedes the likelihood of gathering infor- research tend to center on the practical and mation on long-term outcomes. There are ethical limitations of randomized interven- also few incentives to conduct interventions tions. I begin this chapter by detailing some of in which the impacts may only be deter- these criticisms and then explore one means mined years down the road. Such studies are of extending the value of randomized inter- not amenable to dissertation research unless ventions beyond their original intent to ame- graduate students extend their tours of duty liorate some of these same perceived limita- even longer, nor do they suit junior fac- tions. ulty trying to build their publication records. One of the most prominent critiques of The isolation of long-term effects necessi- this genre is that randomized experiments tates long-term planning, maintenance, and tend to be overly narrow in scope in terms funding, all of which tend to be in short supply for many researchers. The second practical critique of random- The views expressed in this chapter are solely those of ized experiments stems from the difficulty of the author and do not represent the views of the U.S. Military Academy, the Department of the Army, and/or such studies to capture the influence of vari- the Department of Defense. ables of interest to many political scientists.

494 Analyzing the Downstream Effects of Randomized Experiments 495

We cannot purposefully alter political cul- The list of interventions that, although ture to see its effect on democratization, nor useful for research and pedagogical purposes, can we alter the family background of a set would be simply impractical or unethical is of individuals to investigate its influence on seemingly endless, whereas the universe of political socialization. Perhaps, some might interventions that are both feasible and use- argue, experiments are only useful in studying ful appears somewhat limited in scope. Does narrow topics of political behavior in highly this mean that experiments will only be use- controlled settings, mitigating their applica- ful in answering questions where it is prac- bility to broad real-world questions. tical and ethical to manipulate variables of Randomized interventions also face ethi- interest? The answer is no for myriad rea- cal limitations and critiques. Prohibitions on sons discussed in this volume and elsewhere. potentially harmful interventions need little Here I focus on just one: we can expand discussion here. We must consider exper- the utility of initial interventions beyond iments testing political variables of inter- their original intent through examination of est that may fall into an ethical gray area. the long-term, and sometimes unforeseen, There are more possibilities of these than consequences of randomized interventions. one might initially assume. As in medical test- Using downstream analysis, political scien- ing, controversy exists concerning the prac- tists can leverage the power of one ran- tice of denying a known good to a subject domized intervention to examine a host of as an evaluative technique. Subjecting partic- causal relationships that they might otherwise ipants to varying treatment regimens might have never been able to study through means indicate an underlying assumption that one other than observational analysis. Although such regimen is “better” than another. If political scientists may not randomly assign this is indeed the case, then practitioners are some politicians to receive more money left open to the charge that they are some- than others, some other intervention, natu- how adversely affecting one group of peo- ral or intended, may produce such an out- ple over another by failing to provide them come. Researchers can exploit this variation with the “better” treatment option. For exam- achieved through random or near-random ple, intentionally boosting some individuals’ assignment to examine the consequences levels of educational attainment in compar- of this resulting discrepancy in campaign ison to others in an effort to examine the finances on other outcomes such as electoral ramifications of additional years of school- success. ing on political and behavioral outcomes In the next section, I further define and seems unethical given the widespread belief detail the key assumptions associated with in the positive externalities associated with downstream analysis of randomized exper- schooling. iments. I then highlight research that uses Also, ethically dubious is the notion that downstream analysis and outline some poten- some interventions could have the long-term tial research areas that may benefit from consequence of influencing social outcomes. this outgrowth of randomized interventions. Artificially enhancing randomly selected can- I conclude with a discussion of the method- didates’ campaign coffers to test the effects ological, practical, and ethical challenges of money on electoral success could affect posed by downstream analysis and offer electoral outcomes and the drafting and pas- some suggestions for overcoming these diffi- sage of legislation. Randomly assigning dif- culties. ferential levels of lobbying on a particular bill could sway the drafting and passage of legislation. Testing the effectiveness of gov- 1. Extending the Benefits of ernmental structures through random assign- Randomized Experiments ment of different types of constitutions to newly formed states could result in internal For most researchers, a randomized exper- strife and international instability. iment ends once the treatment is applied 496 Rachel Milstein Sondheimer and the outcome measures are collected. this strong relationship and an inability to This seeming finality belies the possibility isolate educational attainment from unob- that many, although perhaps not all, treat- served factors such as cognitive ability and ments produce ramifications that extend well family background. Observational analysis of beyond the original time frame and purpose the topic is faced with diminishing returns; of the study. The unintended consequence random assignment of varying levels of edu- of these interventions is that they often pro- cational attainment to otherwise similar sub- vide an exogenous shock to an outcome not jects would advance current knowledge in this normally amenable to classic randomization. field but is impossible for moral and prac- This allows researchers to extend the posi- tical reasons. It is possible to still leverage tive externalities of experimentation to dif- the benefits of experimental methods in this ferent fields of study and interest; this is field. Randomized trials to test different edu- especially useful for fields in which experi- cation techniques and policies often produce ments are often untenable for practical and differential levels of schooling between treat- ethical reasons. Green and Gerber (2002) ment and control cohorts. The second-order define these downstream benefits as “knowl- effects of these interventions, if observational edge acquired when one examines the indirect analysis is correct, should also produce dif- effects of a randomized experimental inter- ferential rates of political participation as vention” (394). a result of the boost given to the aver- Analyzing the second-order consequences age years of schooling among the treatment of randomized experiments can help jus- cohort. Although these studies were designed tify some of the perceived limitations often to examine the value of programs such as associated with such endeavors. Downstream public preschool and small class size, polit- analysis opens up the possibility of extend- ical scientists can use the exogenous shocks ing a narrowly construed topic or outcome to years of schooling brought about by ran- to apply to a broader range of fields. Experi- domized interventions to examine the effects ments on the utility of direct mail as a mobi- of high school graduation on voting behav- lization tool can become investigations into ior and other political and civic participation the causes of habitual voting (Gerber, Green, outcomes. and Shacher 2003). Random assignment of In the next section, I detail how we can a local political office reserved for women estimate such effects, continuing with the can become investigations into the long-term example of the downstream possibilities for effects of breaking the electoral glass ceil- using randomized educational interventions ing for women (Bhavnani 2009). This type of to untangle the causal relationship between application is best seen through an example educational attainment and political partici- involving the effects of schooling on participation. pation. The positive effect of formal schooling on political and civic participation is 2. A Model and Estimation widely reported in observational research of Downstream Effects of (e.g., Berelson, Lazarsfeld, and McPhee 1954; Randomized Experiments Campbell et al. 1960; Converse 1972; Verba Estimation of Local Average and Nie 1972; Wolfinger and Rosenstone Treatment Effect 1980; Delli Carpini and Keeter 1996; Miller and Shanks 1996; Nie, Junn, and Stehlik- The assumptions underlying the Rubin Barry 1996), but some recent research ques- causal model and the use of instrumen- tions the underpinnings of this relationship tal variables (IVs) estimation to overcome (Tenn 2007; Kam and Palmer 2008; Sond- unobserved heterogeneity are laid out by heimer and Green 2010). Many scholars tend Sovey and Green (2011) and adapted here to view this link as causal despite failure to for the downstream analysis of experi- clarify the causal mechanisms that produce ments. The IV estimator consists of a Analyzing the Downstream Effects of Randomized Experiments 497 two-equation model and is written as us to the fact that IV reveals the causal influ- follows: ence of the intervention for a subset of the population. We need not assume that the ini- = β + β + λ Y i 0 1 X i 1 Q1i tial intervention affected all treatment sub- +···+λ + jects in the same way or that the intervention K QKi ui (1) determined the outcome for all subjects in the X = γ0 + γ1 Z + δ1 Q1 i i i treatment cohort. + δ +···+δ + 2 1 Q1i K QKi ei ( ) The first step in estimating the LATE model is to conceptualize the effects of ran- In Equation (1), we assume that the individual domized assignment in the intervention on voter turnout (Yi) is the dependent variable, educational outcomes. In discussing estima- educational attainment (X1) is the regressor of tion techniques for randomized experiments, interest, Q1i Q2i,...QKi are covariates, and ui Imbens and Rubin (1997) group subjects is an unobserved disturbance term. Equation into four categories based on how the treat- 2 ( ) holds that the endogenous regressor (Xi), ments they receive depend on the treatments educational attainment in this case, is a lin- to which they are assigned. In the case of ear function of an instrumental variable (Zi), downstream analysis of randomized interven- the covariates, and an unobserved disturbance tions, the concept of compliance is applied term (ei ). Random assignment of Z allows us somewhat differently. In the case of down- to isolate exogenous variation in X, that is, the stream analysis of educational interventions, piece of X that is independent of other factors we define compliance in terms of whether that might influence Y, overcoming concerns people graduate high school in response to of omitted variable bias in our estimation of being assigned to the treatment group. In the the effect of X on Y. context of a downstream analysis, Imbens and To see what IV is estimating, it is useful Rubin’s four groups are as follows: to introduce the Rubin causal model as presented by Angrist, Imbens, and Rubin (1996). 1. Compliers graduate from high school if I apply this model to our running example of and only if they are assigned to the treat- = = = using a randomized educational intervention ment group (zi 1, xi 1)or(zi 0, = to isolate the effects of graduation from high xi 0); school on individual voter turnout. Here, 2. Never-takers do not graduate from high we make use of three dichotomous variables school regardless of the group to which 0 1 1 = = = coded as either or : ) individual assign- they are assigned (zi 0, xi 0)or(zi = ment to the treatment or control cohort of 1, xi 0); 2 the intervention (Zi), ) whether an individ- 3. Always-takers graduate from high school 3 ual graduated from high school (Xi), and ) regardless of the group to which they are = = = = whether an individual voted in a given elec- assigned (zi 0, xi 1)or(zi 1, xi tion (Yi). 1); and The preceding IV estimator appears to cal- 4. Defiers graduate from high school if and culate the causal effect of X on Y, but as only if they are assigned to the control = = = = we see in this chapter, closer examination group (zi 0, xi 1)or(zi 1, xi 0). shows that the estimator isolates the causal effect for X on Y for those subjects affected Note that we cannot look at a given indi- by the initial intervention – subjects whose vidual and classify him or her into one of X outcome changed due to the introduction these mutually exclusive groups because we of Z. Angrist and Krueger (2001) define this are limited to only observing one possible estimand as the local average treatment effect outcome per individual. In other words, if 1 (LATE). Speaking in terms of LATEs alerts a subject assigned to the treatment group graduates from high school, then we can- 1 Imbens and Angrist (1994) provide formal analysis of the distinction between LATE and average treatment not discern whether he or she is a complier effects. who could have only graduated if assigned to 498 Rachel Milstein Sondheimer the treatment group or an always-taker who can be expressed as either yi1 if the individ- would have graduated regardless of random ual graduated from high school or yi0 if the assignment in the intervention. subject did not graduate from high school. To draw causal inferences from an instru- The mechanics of downstream analysis can mental variables regression based on down- be illustrated by classifying subjects’ outcome stream data, one must invoke a series measures based on assignment and response of assumptions. The first is independence: to the educational intervention, producing potential outcomes of the dependent vari- four categories of subjects: able must be independent of the experimental group to which a person is assigned. This cri- 1. Individuals who voted regardless of terion is satisfied by random assignment. whether they graduated from high school The second is the exclusion restriction. (y 1 = 1, y 0 = 1); The instrumental variable, Z, only influences i i 2. Individuals who voted if they graduated Y through its influence on X. In other words, from high school but did not vote if they Y changes because of variation in X and not did not graduate from high school (y 1 = because of something else. In this case, we i 1, y 0 = 0); assume that random assignment in the orig- i 3. Individuals who did not vote if they grad- inal experiment has no influence on voting uated from high school but did vote if outcomes aside from that mediated by gradu- they did not graduate from high school ation from high school. Evaluating whether a (y 1 = 0, y 0 = 1); and given instrument satisfies this condition rests i i 4. Individuals who did not vote regardless of on understanding the nature of the relation- whether they graduated from high school ship between an observed variable (Zi) and = = (yi1 0, yi0 0). an unobserved variable (ui). Theoretically, random assignment can ensure the statistical independence of Z and u. However, the As detailed in Table 34.1, based on the nature of the randomized intervention may four compliance possibilities and the four lead to violations of the exclusion restriction – classifications of outcome measures mediated for example, subjects who realize that they are by educational attainment, we can create a being studied may be influenced by the sim- total of sixteen mutually exclusive groups into ple fact that they are part of an experimental which subjects may fall. The total subject treatment group. When evaluating the exclu- population share of each group is expressed π 16 π = 1 sion restriction, the researcher must consider as j such that j j . To estimate the causal pathways that might produce changes causal effect between our variables of inter- in Y through pathways other than X. est, we must further assume monotonicity Third, we invoke the stable unit treatment (Angrist et al. 1996) – that the treatment can value assumption (SUTVA), which holds only increase the possibility of graduating that an individual subject’s response is only from high school. This assumption holds a function of his or her assignment and that there are no defiers or that π13 = π14 = educational attainment and is not affected by π15 = π16 = 0. the assignment or outcomes of other subjects. Randomization allows us to leverage the Fourth, we assume monotonicity, that is, the variation brought about by an exogenous absence of defiers. Finally, we must assume shock to an otherwise endogenous measure that Z exerts some causal influence on X.Low to isolate the influence of that measure on correlation between Z and X can lead to small another outcome of interest. As such, we can sample bias, a problem discussed in Section 4. only identify the influence of schooling on We are primarily interested in the rela- the outcomes of a limited number of the tionship between graduation from high groups of subjects classified in Table 34.1. school and its subsequent effect on the like- We cannot evaluate the effects of school- lihood of voting. As such, we can say that ing on the voting behavior of the never- the dependent variable for each subject i takers who will not graduate from high school Analyzing the Downstream Effects of Randomized Experiments 499

Table 34.1: Classiﬁcation of Target Population in Downstream Analysis of Educational Intervention

Graduates Graduates Votes if from High from High Graduates Votes if Does School if School if from High Not Graduate Group Assigned to Assigned to School? from High Share of

Number Type Treatment? Control? (yi1) School? (yi0) Population

1 Never-takers No No No No π1 2 No No Yes No π2 a,b 3 No No No Yes π3 a,b 4 No No Yes Yes π4 5 Compliers Yes No No No π5 a 6 Yes No Yes No π6 b 7 Yes No No Yes π7 a,b 8 Yes No Yes Yes π8 9 Always-takers Yes Yes No No π9 a,b 10 Yes Yes Yes No π10 11 Yes Yes No Yes π11 a,b 12 Yes Yes Yes Yes π12 13 Deﬁers No Yes No No π13 b 14 No Yes Yes No π14 a 15 No Yes No Yes π15 a,b 16 No Yes Yes Yes π16

Source: Adapted from Sovey and Green (2011). a This share of the population votes if assigned to the treatment group. b This share of the population votes if assigned to the control group. regardless of assignment or the always-takers assignment on graduation rates, and 4) mono- who will graduate from high school regard- tonicity. As shown later in this chapter, if less of assignment. These outcomes are deter- these assumptions are met, then LATE esti- mined by factors that might be endogenous mation will also provide us with a consistent to our voting model. To isolate the effects of estimator of the CACE. educational attainment on voting, we can only The ﬁrst step is to estimate the voting estimate the schooling’s effects on the out- rates, V, for the treatment and control groups. comes of the compliers, who graduate from As the number of observations in the con- high school if assigned to the treatment but trol group approaches inﬁnity, the observed do not graduate if assigned to the control. voting rate in the assigned control group = 1 N This complier average causal effect (CACE) is (Vˆ c =1 yi0) can be expressed as N c i expressed as p lim Vˆ →∞ c E[y 1 − y 0|i ∈ Compliers] N c i i = π + π + π + π + π + π . 4 π 6 − π7 3 4 7 8 10 12 ( ) = . (3) π5 + π6 + π7 + π8 Similarly, the observed voting rate in the A randomized experiment and application assigned treatment group can be expressed of the Rubin causal model allows us to as estimate the LATE given four previously p lim Vˆ →∞ t discussed assumptions: 1) SUTVA, 2)the N t exclusion restriction, 3) a nonzero effect of = π3 + π4 + π6 + π8 + π10 + π12. (5) 500 Rachel Milstein Sondheimer

Next, we can find a consistent estimator of (2002). The concept of using existing ran- the proportion of compliers (α), subjects who domized and natural experiments to exam- only graduated from high school if and only ine second-order effects of interventions was if assigned to the treatment regimen, by sub- slow to build due, in large part, to the rela- tracting the graduation rate of the control tive dearth of suitable experiments in political group from the graduation rate of the treat- science. Now that experiments are becoming ment group: more widespread and prominent in political science literature, scholars are beginning to p lim αˆ cull the growing number of interventions to →∞ N test theories seemingly untestable through = p lim (πˆ 5 + πˆ 6 →∞ traditional randomization. In this section, I N t touch on just a few such examples and offer +πˆ 7 + πˆ 8 + πˆ 9 + πˆ 10 + πˆ 11 + πˆ 12) avenues for further exploration using simi- −p lim (πˆ 9 + πˆ 10 + πˆ 11 + πˆ 12) →∞ N c lar first-order experiments. This discussion is = π5 + π6 + π7 + π8. (6) meant to encourage those interested to seek out these and other works to explore the possibilities of downstream analysis. Finally, we can combine the estimators pre- As I discussed previously, an interesting sented in Equations (4)–(6) to provide a con- stockpile of experiments well suited for down- sistent LATE estimate that is the same as the stream analysis are interventions designed to CACE presented previously: test public policy innovations, programs in education in particular. While a key vari- Vˆ − Vˆ π6 − π7 p lim t c = . (7) able of interest to many is the influence of N →∞ αˆ π5 + π6 + π7 + π8 education on a wide array of political and social outcomes, one’s level of educational The estimator for the LATE also estimates attainment is itself a product of numerous the CACE, the effect of educational attain- factors, potentially impeding our ability to ment among those who would not have isolate schooling’s causal effect through stan- graduated save for the intervention (the dard observational analysis (Rosenzweig and compliers). Wolpin 2000). At the same time, it is nearly In conclusion, the IV estimator estimates impossible and potentially unethical to ran- CACE. We are not estimating the effect of domize the educational attainment of individ- high school for everybody, just the effect for uals or groups to gauge its effect. Sondheimer those who are influenced by a high school and Green (2010) examine two randomized inducing program. Of course, one can gener- educational interventions, the High/Scope alize beyond compliers, but this must be done Perry Preschool project examining the value cautiously and through replication unless one of preschool in the 1960s and the Student- is willing to make strong assumptions. The Teacher Achievement Ratio program testing downstream analysis of an individual inter- the value of small classes in Tennessee in the vention might be best interpreted as a LATE 1980s, in which the treatment groups wit- estimate, but, if a persistent pattern emerges nessed an increase in years of schooling in through the downstream analysis of multi- comparison control groups. They used these ple interventions with different target popu- differential levels of schooling produced by lations, we can then begin to extrapolate the the randomized interventions to isolate the results to a more broad population. effects of educational attainment on likelihood of voting, confirming the strong effect often produced in conventional observational 3. Downstream Analysis in Practice analysis. In addition to examinations of voter turnout, downstream analysis of educational “Downstream experimentation” is a term interventions can isolate the effects of years of originally coined by Green and Gerber schooling on a range of outcomes, including Analyzing the Downstream Effects of Randomized Experiments 501 views on government, party affiliation, civic tion and change can use downstream anal- engagement, and social networking. ysis to examine the second-order effects of As discussed throughout this volume (see, these exogenously induced variations on sub- in particular, Michelson and Nickerson’s sequent beliefs, opinions, and behaviors. For chapter in this volume), experimentation is example, Peffley and Hurwitz (2007)usea proliferating in the field of voter mobiliza- survey experiment to test the effect of dif- tion. Scores of researchers conduct random- ferent types of argument framing on support ized trials to estimate the effectiveness of dif- for capital punishment. Individual variation ferent techniques aimed at getting people to in support for capital punishment brought the polls. In doing so, these studies create the about by this random assignment could be opportunity for subsequent research on the used to test how views on specific issues influ- second-order effects of an individual casting ence attitudes on other political and social a ballot when he or she would not have done issues, the purpose and role of government so absent some form of intervention. Gerber generally, and evaluations of electoral candi- et al. (2003) use an experiment testing the dates. effects of face-to-face canvassing and direct Natural experiments provide further mail on turnout in a local election in 1998 opportunity for downstream examination of to examine whether voting in one election seemingly intractable questions. In this vein, increases the likelihood of voting in another scholars examine the second- and third-order election. Previous observational research on effects caused by naturally occurring random the persistence of voting over time is unable or near-random assignment into treatment to distinguish between the unobserved causes and experimental groups. Looking to legisla- of an individual voting in the first place tive research, Kellerman and Shepsle (2009) from the potential of habit formation. Gerber use the lottery assignment of seniority to mul- et al. find that the exogenous shock to vot- tiple new members of congressional commit- ing produced within the treatment group by tees to explore the effects of future senior- the initial mobilization intervention in 1998 ity on career outcomes such as passage of endured somewhat in the 1999 election, indi- sponsored bills in and out of the jurisdic- cating a persistence pattern independent of tion of the initially assigned committee and other unobserved causes of voting. Further reelection outcomes. Bhavnani (2009) uses extension of mobilization experiments could the random assignment of seats reserved for test the second-order effects of casting a bal- women in local legislative bodies in India to lot on attitudes (e.g., internal and external examine whether the existence of a reserved efficacy), political knowledge, and the likeli- seat, once removed, increases the likelihood hood of spillover into other forms of partici- of women being elected to this same seat in pation. the future. The original intent of the reser- Laboratory settings and survey manipu- vation system is to increase the proportion lation offer fruitful ground for downstream of women elected to local office. Bhavnani analysis of randomized experiments. Hol- exploits the random rotation of these reserved brook’s chapter in this volume discusses seats to examine the “next election” effects of experiments, predominantly performed in this program once the reserved status of a laboratories or lablike settings or in sur- given seat is removed and the local election is vey research, that seek to measure either again open to male candidates. He focuses his attitude formation or change as the depen- analysis on the subsequent elections in these dent variable. As she notes, understand- treatment and control wards, but one could ing the processes of attitude formation and imagine using this type of natural random- change is central to research in political sci- ization process to examine a host of second- ence because such attitudes inform demo- order effects of the forced election of female cratic decision making at all levels of gov- candidates ranging from changes in attitudes ernment and politics. Experiments seeking toward women to shifts in the distribution of to understand the causes of attitude forma- public goods in these wards. 502 Rachel Milstein Sondheimer

As the education experiments indicate, vouchers, possibly due to the disruption of randomized policy interventions used to test social networks that may result from reloca- new and innovative ideas provide fascinating tion. Future research in this vein could lever- opportunities to test the long-term ramifica- age this and similar interventions to examine tions of changes in individual and community how exogenously induced variations in the level factors on a variety of outcomes. Quasi- residency patterns of individuals and fami- experiments and natural experiments that lies affect social networks, communality, civic create as-if random placement into treat- engagement, and other variables of interest to ment and control groups provide additional political scientists. prospects. Many programs use lotteries to Other opportunities for downstream anal- determine which individuals or groups will ysis of interventions exist well beyond those receive certain new benefits or opportuni- discussed here. As this short review shows, ties. Comparing these recipients to nonre- however, finding these downstream possi- cipients or those placed on a waiting list bilities often entails looking beyond litera- evokes assumptions and opportunities sim- ture in political science to other fields of ilar to those of randomized experiments study. (Green and Gerber 2002). Numerous scholars in a range of disciplines (e.g., Katz, Kling, and Liebman 2001; Ludwig, Dun- 4. Challenges can, and Hirschfield 2001; Leventhal and Brooks-Gunn 2003; Sanbonmatsu et al. 2006) Downstream analysis of existing random- have examined the Moving to Opportunity ized interventions provides exciting possibil- (MTO) program that relocates participants ities for researchers interested in isolating from high-poverty public housing to pri- causation. We can expand the universe of vate housing in either near-poor or nonpoor relationships capable of study using assump- neighborhoods. tions generated by randomization and exper- Political scientists can and have bene- imental analysis. Although downstream anal- fited from this as-if random experiment as yses can be used to overcome some of well. Observational research on the effects the limitations of randomized interventions, of neighborhoods on political and social out- they do pose their own set of challenges. comes suffers from self-selection of subjects In this section, I discuss the methodolog- into neighborhoods. The factors that deter- ical, practical, and ethical challenges faced mine where one lives are also likely to influ- by those who want to perform downstream ence one’s proclivities toward politics, social analysis. networking tendencies, and other facets of political and social behavior. Downstream Methodological Challenges research into a randomly assigned residential voucher program allows political scien- Two key impediments to downstream anal- tists the opportunity to parse out the effects of ysis of randomized experiments stem from neighborhood context from individual-level two of the three conditions for instruments to factors that help determine one’s choice of maintain the conditions for instrumental vari- residential locale. Political scientists are just able estimation, specifically finding instru- beginning to leverage this large social experiments that meet the exclusion restriction and ment to address such questions. Gay’s (2010) provide a strong relationship between assign- work on the MTO allows her to examine ment and the independent variable of inter- how an exogenous shock to one’s residential est. First, a suitable instrument must meet environment affects political engagement in the exclusion restriction in that it should not the form of voter registration and turnout. exert an independent influence on the depen- She finds that subjects who received vouch- dent variable. Randomization of subjects into ers to move to new neighborhoods voted at treatment and control cohorts is not sufficient lower rates than those who did not receive to meet the exclusion restriction because the Analyzing the Downstream Effects of Randomized Experiments 503 specific nature of the treatment regimen can be due to habit but to another factor such as still violate this condition. Returning to our outreach to likely voters. investigation of educational interventions as There is no direct way to test whether this a means of isolating the influence of educa- condition is met. Instead, we must make the- tional attainment on political participation, oretical assumptions about the relationship we can imagine myriad situations in which between the instrument and potential unob- the intervention may influence participation, served causes of the variation in the depen- independent of the effects of schooling. If the dent variable. In-depth knowledge of the intervention works through a mentoring pro- original intervention and its possible effects gram, then mentors may discuss politics and on numerous factors relating to the outcome the importance of political involvement with variable of interest is necessary to uncover subjects in the treatment group, increasing possible violations. the likelihood of voting later in life. Similarly, The second methodological difficulty it is possible that an intervention intended to posed by researchers wanting to perform boost the educational attainment of a treat- downstream analysis relates to the second ment group also influences the family dynam- condition necessary for consistent estimates ics of the subjects’ home lives. If this occurs using instrumental variables – that the instru- and the exclusion restriction is violated, then ment must be correlated with the endoge- researchers will be unable to isolate the causal nous independent variable. In downstream influence of variations in years of schooling analysis, meeting this condition entails find- on political participation independent of fam- ing randomized experiments that “worked” ily background. insofar as random assignment produced vari- Another example of the potential viola- ation in the treatment group as compared tion of the exclusion restriction exists con- to the control. As Sovey and Green (2011) cerning Gerber et al.’s (2003) work on vot- and Wooldridge (2009) discuss, use of a weak ing and habit formation. Recall that Gerber instrument – an exogenous regressor with a et al. use a randomized mobilization inter- low partial correlation between Zi and Xi – vention to find that, all else equal, voting in can cause biased IV estimates in finite sam- one election increases the likelihood of vot- ples. Fortunately, weak instrumentation can ing in subsequent elections, indicating that be diagnosed using a first-stage F statistic, as electoral participation is habit forming. This outlined by Stock and Watson (2007). result hinges on the assumption that no other Finding a suitable instrument to meet changes took place for individuals in the ini- this condition is a delicate matter because tial experiment other than assignment to the although researchers need to find an exper- control or treatment group. A violation of iment with a strong result, we also need the exclusion restriction would occur if the to be wary of interventions with too-strong outcome induced due to assignment influ- results, an indication that something might enced other factors believed to influence sub- be awry. As Green and Gerber (2002)dis- sequent voting. For example, if political par- cuss, such results might indicate a failure of ties and other campaign operatives tend to randomization to create similar groups prior reach out to those who already seem likely to the intervention. Randomized experiments to vote (Rosenstone and Hansen 1993), then with large numbers of subjects and replicated voting in the first election may increase the results provide the best potential pool of stud- likelihood of being subject to increased mobi- ies for downstream analysis. lization in subsequent elections, increasing the likelihood of voting in subsequent elec- Practical Challenges tions. If this occurs and the exclusion restriction is violated, Gerber et al. (2003) would The most straightforward challenge of down- still be correct in arguing that voting in one stream analysis of randomized experiments election increases the likelihood of voting in is that, in most circumstances, the analyst subsequent elections, but this result may not has no control over the course of the initial 504 Rachel Milstein Sondheimer intervention. There are some opportunities necessity of maintaining such information at to improve on the analysis of an original inter- the time of the initial intervention, the ability vention, but, in general, the internal validity to reconnect with subjects cannot be under- of downstream analysis cannot be much, if valued. In the aftermath of his famous obe- any, better than that of the original exper- dience studies, Milgram (1974) followed up iment. Downstream analysis cannot recover with his subjects to evaluate their lingering the value of a botched experiment any more responses and reactions to the intervention. than it can mar the results of a well-executed Davenport et al. (2010) recently studied the intervention. enduring effects of randomized experiments, In some cases, scholars can alter the orig- testing the benefits of applying social pres- inal analysis of results to either narrow down sure as a voter mobilization tool by analyz- or expand the reach of the intervention, such ing registration rolls in subsequent elections. as by considering the effects of an intended Although collecting and keeping track of sub- treatment rather than the effects of the treat- ject contact information may slightly increase ment on the treated. The ability of down- the cost of the original experiment, they have stream analysts to make this type of deci- the potential to increase the payoffs in the sion depends on the depth and clarity of the long term. description of the original intervention. In Both practical challenges can be overcome examining the original Perry intervention, through cooperation among researchers. Schweinhart, Barnes, and Weikart (1993) This advice may seem prosaic; however, detailed their decision to move two sub- in the case of experiments, there is usu- jects from the treatment group to the control ally only one opportunity to perform an group because of the subjects’ unique fam- intervention, and optimal execution of said ily concerns. Such documentation allowed experiment will influence future scholars’ Sondheimer and Green (2010)tomove ability to build off of the results. Shar- these subjects into the original intent-to-treat ing experimental designs prior to imple- group for use in their own downstream anal- mentation can open up important dialogue ysis of the program. This example speaks among scholars, dialogue that can improve to the necessity of meticulous record keep- initial interventions and heighten prospects ing and documentation of decisions made for future downstream possibilities. More- prior to, during, and following a randomized over, the maintenance of subject contact intervention. information may not be a first-order prior- A second and related practical challenge ity for researchers, but it may provide long- is that of subject attrition. Depending on term benefits to one’s own research team the topic of the investigation, there can be and others. The sharing of results among a a long lag time between the initial inter- broad swath of the research community can vention and the collection of new data for also increase the likelihood of extending the downstream analysis. The longer the win- findings beyond their original intent. As seen, dow between the initial intervention and the many interventions give way to downstream downstream analysis, the more likely one is analysis in entirely different subject areas. to face high levels of attrition. This loss of Cross-disciplinary collaboration on random- data becomes increasingly problematic if it ized experiments will help scholars approach is nonrandom and associated with the initial previously intractable puzzles. This will be intervention. easier to do if researchers cooperate from the Concerns over attrition can be ameliorated outset. if those performing the original intervention collect and maintain contact and other iden- Ethical Challenges tifying information on their original subjects. If these data are lost or never collected, then Although downstream analysis of preex- the possibilities for downstream analysis dis- isting randomized interventions provides sipate. Even if researchers do not foresee the researchers the opportunity to study exoge- Analyzing the Downstream Effects of Randomized Experiments 505 nous changes in independent variables of in the face of downstream analysis because interest normally deemed off-limits to out- researchers might never be sure when, if side manipulation, this type of analysis does ever, data collection is completed. Scholars pose some interesting ethical considerations must consider whether a full or even limited with reference to subject consent and the debriefing will impede the ability of future use of deception. Concerns in both realms researchers to conduct downstream analy- derive from the unending nature of exper- sis of the original experiment and what the iments subject to future downstream analy- proper course of behavior should be given this sis. The issues raised with regard to consent potentiality. and deception in considering how researchers planning new interventions ought to proceed to ensure that their contemporary research 5. Conclusion has value beyond its original intent. In terms of consent, we should consider Researchers conducting randomized experi- whether it is problematic for participants to ments ought to consider potential long- and be subjected to tests and data collection post short-term downstream analyses of their ini- hoc if they did not accede to further exam- tial interventions. Doing so expands our esti- ination at the time of the original inter- mates of the costs and benefits associated vention. Such downstream research might with randomized experimentation and probe unobtrusive and not involve interaction vides unique research prospects for those who with the original subjects, but these con- are unable to feasibly conduct such interven- cerns remain the same. Moreover, encour- tions. Awareness of these additional research aging researchers to share data on experi- opportunities can help structure the inter- ments to allow for later analysis may violate vention and data collection process in ways Institutional Review Board guidelines stipu- amenable to downstream analysis. Down- lating that research proposals detail the names stream analysis is only possible if data are of any investigators who will have access to maintained and made available to others. data containing names and other identifiable Moreover, consideration of downstream pos- information of participants in the original sibilities prior to implementing a particular study. protocol will help researchers brainstorm the Issues raised over deception and full disclo- range of measures collected at the outset of sure closely parallel concerns over consent. a project. This will expand the value of the Although consent focuses on what the origi- experiment to the individual researcher in the nal researcher should do prior to the interven- short term and to the broader reaches of the tion, challenges concerning deception center research community in the long term. around behavior following the initial intervention. The American Psychological Asso- ciation’s “Ethical Principles of Psycholo- References gists and Code of Conduct” stipulates that researchers using deception provide a full American Psychological Association. 2003. Ethi- debriefing to participants as soon as possi- cal Principles of Psychologists and Codes of Con- ble following the intervention, but “no later duct. Washington, DC: American Psychologi- than the conclusion of the data collection” cal Association. 2003 Angrist, Joshua D., Guido W. Imbens, and Don- (American Psychological Association , 1996 section 8.07). Ideally, this disclosure should ald B. Rubin. . “Identification of Causal Effects Using Instrumental Variables.” Journal occur immediately following the conclusion of the American Statistical Association 91: 444–55. of the intervention, but researchers are per- Angrist, Joshua D., and Alan B. Krueger. 2001. mitted to delay the debriefing to maintain the “Instrumental Variables and the Search for integrity of the experiment. However, this Identification: From Supply and Demand to stipulation mandating a debriefing at the con- Natural Experiments.” Journal of Economic Per- clusion of the data collection is problematic spectives 15: 69–85. 506 Rachel Milstein Sondheimer

Berelson, Bernard, Paul F. Lazarsfeld, and ity Experiment.” Quarterly Journal of Economics William N. McPhee. 1954. Voting: A Study 116: 607–54. of Opinion Formation in a Presidential Cam- Kellermann, Michael, and Kenneth A. Shep- paign. Chicago: The University of Chicago sle. 2009. “Congressional Careers, Committee Press. Assignments, and Seniority Randomization in Bhavnani, Rikhil R. 2009. “Do Electoral Quotas the US House of Representatives.” Quarterly Work after They Are Withdrawn? Evidence Journal of Political Science 4: 87–101. from a Natural Experiment in India.” American Leventhal, Tama, and Jeanne Brooks-Gunn. 2003. Political Science Review 103: 23–35. “Moving to Opportunity: An Experimental Campbell, Angus, Philip E. Converse, Warren E. Study of Neighborhood Effects on Mental Miller, and Donald E. Stokes. 1960. The Amer- Health.” American Journal of Public Health 93: ican Voter. New York: Wiley. 1576–82. Converse, Philip E. 1972. “Change in the Amer- Ludwig, Jens, Greg J. Duncan, and Paul ican Electorate.” In The Human Meaning of Hirschfield. 2001. “Urban Poverty and Juvenile Social Change, eds. Angus Campbell and Philip Crime: Evidence from a Randomized Housing- E. Converse. New York: Russell Sage Founda- Mobility Experiment.” Quarterly Journal of Eco- tion, 263-337. nomics 116: 655–79. Davenport, Tiffany C., Alan S. Gerber, Donald Milgram, Stanley. 1974. Obedience to Authority: An P. Green, Christopher W. Larimer, Christo- Experimental View. New York: Harper & Row. pher B. Mann, and Costas Panagopoulos. Miller, Warren E., and J. Merrill Shanks. 1996. 2010. “The Enduring Effects of Social Pres- The New American Voter. Cambridge, MA: Har- sure: Tracking Campaign Experiments over a vard University Press. Series of Elections.” Political Behavior 32: 423– Nie, Norman H., Jane Junn, and Kenneth Stehlik- 30. Barry. 1996. Education and Democratic Citizen- Delli Carpini, Michael X., and Scott Keeter. 1996. ship in America. Chicago: The University of What Americans Know about Politics and Why Chicago Press. It Matters. New Haven, CT: Yale University Peffley, Mark, and Jon Hurwitz. 2007. “Persuasion Press. and Resistance: Race and the Death Penalty in Gay, Claudine. 2010. “Moving to Opportunity: America.” American Journal of Political Science The Political Effects of a Housing Mobility 51: 996–1012. Experiment.” Working paper, Harvard Uni- Rosenstone, Steven J., and John Mark Hansen. versity. 1993. Mobilization, Participation, and Democracy Gerber, Alan S., Donald P. Green, and Ron in America. New York: Macmillan. Shacher. 2003. “Voting May Be Habit- Rosenzweig, Mark R., and Kenneth I. Wolpin. Forming: Evidence from a Randomized Field 2000. “Natural ‘Natural Experiments’ in Eco- Experiment.” American Journal of Political Sci- nomics.” Journal of Economic Literature 38: 827– ence 27: 540–50. 74. Green, Donald P., and Alan S. Gerber. 2002. Sanbonmatsu, Lisa, Jeffery R. Kling, Greg J. Dun- “The Downstream Benefits of Experimenta- can, and Jeanne Brooks-Gunn. 2006. “Neigh- tion.” Political Analysis 10: 394–402. borhoods and Academic Achievement: Results Imbens, Guido W., and Joshua D. Angrist. 1994. from the Moving to Opportunity Experi- “Identification and Estimation of Local Aver- ment.” Journal of Human Resources XLI: 649– age Treatment Effects.” Econometrica 62: 467– 91. 75. Schweinhart, L. J., Helen V. Barnes, and David P. Imbens, Guido W., and Donald B. Rubin. 1997. Weikart. 1993. Significant Benefits: The High- “Bayesian Inference for Causal Effects in Ran- Scope Perry Preschool Study through Age 27. domized Experiments with Noncompliance.” Monographs of the High/Scope Educational Annals of Statistics 25: 305–27. Research Foundation, vol. 10. Ypsilanti, MI: Kam, Cindy, and Carl L. Palmer. 2008. “Recon- High/Scope Press. sidering the Effects of Education on Civic Par- Sondheimer, Rachel Milstein, and Donald P. ticipation.” Journal of Politics 70: 612–31. Green. 2010. “Using Experiments to Estimate Katz, Lawrence F., Jeffery R. Kling, and Jeffery the Effects of Education on Voter Turnout.” B. Liebman. 2001. “Moving to Opportunity in American Journal of Political Science 54: 174– Boston: Early Results of a Randomized Mobil- 89. Analyzing the Downstream Effects of Randomized Experiments 507

Sovey, Allison J., and Donald P. Green. 2011. Verba, Sidney, and Norman H. Nie. 1972. Partic- “Instrumental Variables Estimation in Political ipation in America: Political Democracy and Social Science: A Reader’s Guide.” American Journal Equality. New York: Harper & Row. of Political Science 55: 188–200. Wolﬁnger, Raymond E., and Steven J. Rosen- Stock, James H., and Mark W. Watson. 2007. stone. 1980. Who Votes? New Haven, CT: Yale Introduction to Econometrics. 2nd ed. Boston: University Press. Pearson Addison Wesley. Wooldridge, Jeffrey M. 2009. Introductory Eco- Tenn, Steven. 2007. “The Effect of Education on nometrics: A Modern Approach. 4th ed. Mason, Voter Turnout.” Political Analysis 15: 446–64. OH: South-Western Cengage Learning. CHAPTER 35

Mediation Analysis Is Harder Than It Looks

John G. Bullock and Shang E. Ha

Mediators are variables that transmit causal Unfortunately, the most common pro- effects from treatments to outcomes. Those cedures are not very good. They call for who undertake mediation analysis seek to indirect effects – the portions of treatment answer “how” questions about causation: how effects that are transmitted through medi- does this treatment affect that outcome? Typ- ators – to be estimated via multiequation ically, we desire answers of the form “the regression frameworks. These procedures treatment affects a causally intermediate vari- do not require experimental manipulation able, which in turn affects the outcome.” of mediators; instead, they encourage the Identifying these causally intermediate vari- study of mediation with data from unma- ables is the challenge of mediation analysis. nipulated mediators (MacKinnon et al. 2002, Conjectures about political mediation 86; Spencer, Zanna, and Fong 2005). The effects are as old as the study of politics. procedures are therefore prone to producing But codification of procedures by which to biased estimates of mediation effects. Warn- test hypotheses about mediation is a rela- ings about this problem have been issued for tively new development. The most common decades by statisticians, psychologists, and procedures are now ubiquitous in psychology political scientists. (Quinones-Vidal˜ et al. 2004) and increasingly Recognizing that nonexperimental meth- popular in the other social sciences, not least ods of mediation analysis are likely to be political science. biased, social scientists are slowly turning to methods that involve experimental manipulation of mediators. This is a step in the right A previous version of this chapter was presented at direction. But experimental mediation anal- the Experimentation in Political Science Conference at Northwestern University, May 28–29, 2009. We thank ysis is difficult – more difficult than it may Jamie Druckman, Alan Jacobs, Scott Matthews, Nora seem – because experiment-based inferences Ng, David Nickerson, Judea Pearl, Dustin Tingley, about indirect effects are subject to impor- and Lynn Vavreck for comments. Particular thanks to Donald Green, our coauthor on several related papers, tant but little-recognized limitations. The for many helpful discussions. point of this chapter is to explain the bias to

508 Mediation Analysis Is Harder Than It Looks 509 which nonexperimental methods are prone social psychology, where its influence is now and to describe experimental methods that hard to overstate.1 And within political sci- hold out more promise of generating credi- ence, it is most prominent among articles that ble inferences about mediation. But it is also have explicitly psychological aims. For exam- to describe the limits of experimental media- ple, Brader, Valentino, and Suhay (2008)use tion analysis. the procedure to examine whether emotions We begin by characterizing the role that mediate the effects of news about immigra- mediation analysis plays in political science. tion on willingness to write to members of We then describe conventional methods of Congress. Fowler and Dawes (2008, 586–88) mediation analysis and the bias to which they use it to test hypotheses about mediators of are prone. We proceed by describing exper- the connection between genes and turnout. imental methods that can reliably produce And several political scientists have used it accurate estimates of mediation effects. The to understand the mechanisms that underpin experimental approach has several important priming and framing effects in political con- limitations, and we end the section by explain- texts (e.g., Nelson 2004; Malhotra and Kros- ing how these limitations imply both best nick 2007). practices and an agenda for future research. To some, the increasing use of the Baron- We consider objections to our argument Kenny method in political science seems a in the next section, including the common good thing: it promises to bring about “valu- objection that manipulation of mediators is able theoretical advances” and is just what often infeasible. Our last section reviews and we need to “push the study of voting up a concludes. notch or two in sophistication and conceptual payoffs” (Malhotra and Krosnick 2007, 250, 276). But increasing use of the Baron- 1. Mediation Analysis in Kenny method is not a good thing. Like Political Science related methods that do not require manipulation of mediators, it is biased, and in turn it The questions that animate political scientists leads researchers to biased conclusions about can be classified epistemologically. Some are mediation. purely descriptive. Others – the ones to which experiments are especially well suited – are 2. Nonexperimental Mediation about treatment effects. (“Does X affect Y? Analyses Are Prone to Bias How much? Under what conditions?”) But questions about mediation belong to a dif- Like many related procedures, the method ferent category. When social scientists seek proposed by Baron and Kenny (1986) is based information about “processes” or demand on three models: to know about the “mechanisms” through which treatments have effects, they are asking M = α1 + aX + e1, (1) about mediation. Indeed, when social scien- Y = α2 + cX + e2, and (2) tists speak about “explanation” and “theory,” mediation is usually what they have in mind. Y = α3 + dX + bM + e3, (3) Social scientists often try to buttress their claims about mediation with data. They use where Y is the outcome of interest, X is a variety of methods to do so, but nearly all a treatment, M is a potential mediator of are based on crosstabulations or multiequation regression frameworks. In this chapter, 1 Quinones-Vidal˜ et al. (2004) show that the article is already the best-cited in the history of the Journal we focus on one such method: the one proof Personality and Social Psychology. Our own search posed by Baron and Kenny (1986). We focus turned up more than 20,000 citations. Analogous on it because it is simple, by far the most searches suggest that Downs (1957) has been cited fewer than 14,000 times and that The American Voter common method, and similar to almost all (Campbell et al. 1960) has been cited fewer than 4,000 other methods in use today. It originated in times. 510 John G. Bullock and Shang E. Ha the treatment, and α1, α2, and α3 are inter- The OLS estimator of d is also biased: cepts. For simplicity, we assume that X and 0 1 cov(e1, e3) M are binary variables coded either or . E[dˆ] = d − a · . The unobservable disturbances e1, e2, and var(e1) e3 are mean-zero error terms that represent the cumulative effect of omitted variables. It (A proof is given in Bullock, Green, and Ha 2008 39 40 is not difficult to extend this framework to [ , – ].) OLS estimators of direct and include multiple mediators and other covari- indirect effects will therefore be biased as ates, and our criticisms apply with equal force well. to models that include such variables. For In expectation, the OLS estimators of b notational clarity and comparability to previ- and d produce accurate estimates only if = 0 3 ous articles about mediation analysis, we limit cov(e1, e3) . But this condition is unlikely our discussion to the three-variable regres- to hold unless both X and M are randomly sion framework. assigned. The problem is straightforward: if For simplicity, we assume throughout this an unobserved variable affects both M and Y, chapter that X is randomly assigned such that it will cause e1 and e3 to covary. And even if it is independent of the disturbances: e1, e2, e3 no unobserved variable affects both M and ⊥⊥X. As we shall see, randomization of X Y, these disturbances are likely to covary if alone does not ensure unbiased estimation M is merely correlated with an unobserved of the effects of mediators. Consequently, we variable that affects Y, e.g., another mediator. refer to designs in which only X is randomized This “multiple-mediator problem” is a seri- as nonexperimental for the purpose of media- ous threat to social-science mediation analysis tion analysis, reserving experimental for stud- because most of the effects that interest social ies in which both X and M are randomized. scientists are likely to have multiple corre- The coefficients of interest are a, b, c, and lated mediators. Indeed, we find it difficult to think of any political effects that do not fit d. The total effect of X on Y is c. To see 4 how c is typically decomposed into “direct” this description. and “indirect” effects, substitute Equation (1) The standard temptation in nonexperi- into Equation (3), yielding mental analysis is to combat this problem by controlling for potential mediators other than M. But it is normally impossible to mea- Y = α3 + X (d + ab) + (a1 + e1)b + e3. sure all possible mediators. Indeed, it may be impossible to merely think of all possible The direct effect of X is d. The indirect mediators. And controlling for some poten- or “mediated” effect is (or, equivalently, ab tial mediators but not all of them is no guaran- c − d).2 tee of better estimates; to the contrary, it may Baron and Kenny (1986) do not say how make estimates worse (Clarke 2009). Fighting the coefficients in these equations are to be estimated; in practice, ordinary least squares 3 Imai, Keele, and Yamamoto (2010, 3) show that the (OLS) is almost universally used. But the indirect effect ab is identified under the assumption of 3 sequential ignorability, i.e., independence of X from OLS estimator of b in Equation ( ) is biased: the potential outcomes of M and Y, and independence of M from the potential outcomes of Y. This is a stronger identifying assumption than cov(e1, e3) = 0 (Imai, Keele, and Yamamoto 2010, 10), but it has cov(e1, e3) E[bˆ] = b + . the virtue of being grounded in a potential-outcomes var(e1) framework. 4 An occasional defense of the Baron-Kenny method is that the method itself is unbiased: the problem lies in its application to nonexperimental data, and it would 2 This discussion of direct and indirect effects elides vanish if the method were applied to studies in which a subtle but important assumption: the effect of M both X and M are randomized. This is incorrect. In on Y is the same regardless of the value of X. This fact, when both X and M are randomized, the Baron- additivity or “no-interaction” assumption is implied Kenny method calls for researchers to conclude that in linear models, e.g., Equation (3). See Robins (2003, M does not mediate X even when M strongly mediates 76–77) for a detailed consideration. X. For details, see Bullock et al. (2008, 10–11). Mediation Analysis Is Harder Than It Looks 511 endogeneity in nonexperimental mediation has been discussed in statistics and politi- analysis by adding control variables is a cal science (e.g., Rosenbaum 1984, 188–94; method with no clear stopping rule or way King and Zeng 2006, 146–48), but its rele- todetectbias–ashaky foundation on which vance to mediation analysis has gone largely to build beliefs about mediation. unnoticed. At root, it is one instance of an Political scientists who use the Baron- even more general rule: estimators of the Kenny (1986) method and related methods parameters of regression equations are likely often want to test hypotheses about several to be unbiased only if the predictors in potential mediators rather than one. In these those equations are independent of the dis- cases, the most common approach is “one-at- turbances. And in most cases, the only way a-time” estimation, whereby Equation (3) is to ensure that M is independent of the dis- estimated separately for each mediator. This turbances is to randomly assign its values. practice makes biased inferences about medi- By contrast, “the benefits of randomization ation even more likely. The researcher, who are generally destroyed by including post- already faces the spectre of bias due to the treatment variables” (Gelman and Hill 2007, omission of variables over which she has no 192). control, compounds the problem by inten- Within the past decade, statisticians and tionally omitting variables that are likely to political scientists have advanced several dif- be important confounds. Nonexperimental ferent methods of mediation analysis that mediation analysis is problematic enough, but do not call for manipulation of mediators. one-at-a-time testing of mediators stands out These methods improve on Baron and Kenny as an especially bad practice. (1986), but they do not overcome the prob- The Baron-Kenny method and related lem of endogeneity in nonexperimental medi- methods are often applied to experiments in ation analysis. For example, Frangakis and which the treatment has been randomized Rubin (2002) propose “principal stratifica- but the mediator has not, and there seems to tion,” which entails dividing subjects into be a widespread belief that such experiments groups on the basis of their potential out- are sufficient to ensure unbiased estimates of comes for mediators. Causal effects are then direct and indirect effects. But randomiza- estimated separately for each “principal stra- tion of the treatment is not enough to pro- tum.” The problem is that some poten- tect researchers from biased estimates. It can tial outcomes for each subject are necessar- ensure that X bears no systematic relation- ily unobserved, and those who use principal ship to e1, e2,ore3, but it says nothing about stratification must infer the values of these whether M is systematically related to those potential outcomes on the basis of covari- variables, and thus nothing about whether ates. In practice, “this reduces to making the 5 cov(e1, e3) = 0. same kinds of assumptions as are made in typ- Stepping back from mediation analysis ical observational studies when ignorability is to the more general problem of estimating assumed” (Gelman and Hill 2007, 193). causal effects, note that estimators tend to be In a different vein, Imai, Keele, and biased when one controls for variables that Yamamoto (2010) show that indirect effects are affected by the treatment. One does this can be identified even when the mediator is whenever one controls for M in a regres- not randomized – provided that we stipu- sion of Y on X, which the Baron-Kenny late the size of cov(e1, e3). This is helpful: if method requires. This “post-treatment bias” we are willing to make assumptions about the covariance of unobservables, then we may be 5 This warning is absent from Baron and Kenny (1986), able to place bounds on the likely size of but it appears clearly in one of that article’s prede- cessors, which notes that what would come to be the indirect effect. But in no sense is this known as the Baron-Kenny procedure is “likely to method a substitute for experimental manip- yield biased estimates of causal parameters . . . even ulation of the mediator. Instead, it requires when a randomized experimental research design has been used” (Judd and Kenny 1981, 607, emphasis in origi- us to make strong assumptions about the nal). properties of unobservable disturbances, just 512 John G. Bullock and Shang E. Ha as other methods do when they are applied entails direct manipulation of treatments and to nonexperimental data. Moreover, Imai, mediators. We have described such a design Keele, Tingley, and Yamamoto (2010, 43) elsewhere (Bullock, Green, and Ha 2008), note that even if we are willing to stipulate but in many cases, limited understanding of the value of cov(e1, e3), the method that they mediators precludes direct manipulation. For propose cannot be used whenever the media- example, although we can assign subjects to tor of interest is directly affected by both the conditions in which their feelings of efficacy treatment and another mediator. This point are likely to be heightened or diminished, we is crucial because many effects that interest do not know how to gain direct experimental political scientists seem likely to be trans- control over efficacy. That is, we do not know mitted by multiple mediators that affect each how to assign specific levels of efficacy to dif- other. ferent subjects. The same is true of party iden- None of these warnings implies that tification, emotions, cultural norms, modes all nonexperimental mediation research is of information processing, and other likely equally suspect. All else equal, research in mediators of political processes. These vari- which only a treatment is randomized is ables and others are beyond direct experimen- preferable to research in which no variables tal control. are randomized; treatment-only randomiza- But even when mediators are beyond tion does not make accurate mediation infer- direct experimental control, we can often ence likely, but it does clarify the assumptions manipulate them indirectly. The key in such required for accurate inference. And in gen- cases is to create an instrument for M,the eral, nonexperimental research is better when endogenous mediator. To be a valid instru- its authors attempt to justify the assumption ment for M, a variable must be correlated that their proposed mediator is uncorrelated with M but uncorrelated with e3. Many vari- with other variables, including unobserved ables are likely to satisfy the first condition: variables, that may also be mediators. This whatever M is, it is usually not hard to think of sort of argument can be made poorly or well. a variable that is correlated with it, and once But even the best arguments of this type typ- we have measured this new variable, estimat- ically warrant far less confidence than arguing the correlation is trivial. But satisfying the ments about unconfoundedness that follow second condition is more difficult. Because directly from manipulation of both the treat- e3 is unobservable, we can never directly test ment and the mediator. whether it is uncorrelated with the potential This discussion should make clear that the instrument. Worse, almost every variable that solution to bias in nonexperimental media- is correlated with M is likely to be correlated tion analyses is unlikely to be another non- with other factors that affect Y, and thus likely 6 experimental mediation analysis. The prob- to be correlated with e3. lem is that factors affecting the mediator and Fortunately, a familiar class of variables the outcome are likely to covary. We are not meets both conditions: assignment-to- likely to solve this problem by controlling for treatment variables. Use of these instrumen- more variables, measuring them more accu- tal variables is especially common in analyses rately, or applying newer methods to nonex- of field experiments, where compliance perimental data. To calculate unbiased esti- with the treatment is likely to be partial. mates of mediation effects, we should look to For example, Gerber and Green (2000) experiments. use a field experiment to study various means of increasing voter turnout. They cannot directly manipulate the treatments of 3. Experimental Methods of interest: they cannot compel their subjects Mediation Analysis

6 See Angrist et al. (1996) for a thorough discussion of The simplest experimental design that per- the conditions that a variable must satisfy to be an mits accurate estimation of indirect effects instrument for another variable. Mediation Analysis Is Harder Than It Looks 513 to read mail, answer phone calls, or speak to directly manipulate the accessibility of par- face-to-face canvassers. Instead, they use ticular thoughts, but they do know how to random assignments to these treatments as indirectly manipulate accessibility by prim- instruments for the treatments themselves. ing people in different ways (e.g., Burdein, Doing so permits them to recover accurate Lodge, and Taber 2006, esp. 363–64; see also estimates of treatment effects even though Lodge and Taber’s chapter in this volume). the treatments are beyond direct experimen- Experimental analysis of the hypothesis is tal control. (For elaboration of this point, therefore possible. Following Equation (3), see Angrist, Imbens, and Rubin [1996] and consider the model: Gerber’s chapter in this volume.) Although the instrumental variables attitudes = α3 + d(framing) approach is increasingly used to estimate + + average treatment effects, it has not yet been b(accessibility) e3. used in political science to study mediation. We think that it should be. It has already In this model, framing indicates whether sub- been used multiple times to study media- jects were assigned to a control condition tion in social psychology, and its use in that (framing = 0) or an issue frame (framing = 1); discipline suggests how it might be used accessibility is reaction times in milliseconds in ours. For example, Zanna and Cooper in a task designed to gauge the accessibility (1974) hypothesize that attitude-behavior of particular thoughts about the issue; and conflict produces feelings of unpleasant ten- e3 is a disturbance representing the cumula- sion (“aversive arousal”), which in turn pro- tive effect of other variables. Crucially, acces- duces attitude change. They cannot directly sibility is not randomly assigned. It is likely manipulate levels of tension, so they use an to be affected by framing and to be corre- instrument to affect it indirectly: subjects lated with unobserved variables represented swallow a pill and are randomly assigned by e3: age, intelligence, and political predis- to hear that it will make them tense, make positions, among others. them relax, or have no effect. In a related The OLS estimator of b, the effect of vein, Bolger and Amarel (2007) hypothesize accessibility, is therefore likely to be biased. that the effect of social support on the stress (The OLS estimator of d, the direct effect of levels of recipients is mediated by efficacy: the framing manipulation, is also likely to be support reduces recipients’ stress by raising biased.) But suppose that in addition to the their feelings of efficacy. Bolger and Amarel framing manipulation and the measurement cannot directly assign different levels of effi- of accessibility, some subjects are randomly cacy to different participants in their experi- assigned to a condition in which relevant con- ment. Instead, they randomly assign subjects siderations are primed. This priming manip- to receive personal messages that are designed ulation may make certain thoughts about the to promote or diminish their feelings of effi- issue more accessible. In this case, accessibil- cacy. In this way, they indirectly manipulate ity remains nonexperimental, but the priming efficacy. intervention generates an instrumental vari- To see how such instruments might be able that we can use to consistently estimate created and used in political science, con- b. If we also estimate a – for example, by consider research on issue framing. A controver- ducting a second experiment in which only sial hypothesis is that framing an issue in a framing is manipulated – our estimator of ab, particular way changes attitudes by increas- the extent to which priming mediates framing the accessibility of particular thoughts ing, will also be consistent. about the issue, i.e., the ease with which par- The most common objection to exper- ticular thoughts come to mind (see Iyengar imental mediation approaches is that they and Kinder 1987, esp. ch. 7; Nelson, Claw- often cannot be used because mediators son, and Oxley 1997; Miller and Krosnick often cannot be manipulated. We take up 2000). Political scientists do not know how to this objection later in this chapter, but 514 John G. Bullock and Shang E. Ha for the moment, we stress that researchers Green, Ha, and Bullock 2010); here, we offer need not seek complete experimental control a brief overview of each.7 over mediators. They need only seek some An experimental intervention is useful for randomization-based purchase on mediators. mediation analysis if it affects one mediator Consider, for example, one of the best-known without affecting others. If the intervention and least tractable variables in political behav- instead affects more than one mediator, it ior research: party identification. The his- violates the exclusion restriction – in terms tory of party ID studies suggests that it of Equation (3), it is correlated with e3 – should be difficult to manipulate. It is one and is not a valid instrument. In this case, of the most stable individual-level influences the instrumental variables estimator of the on votes and attitudes, and no one knows how indirect effect will be biased. For example, to assign different levels of party ID to differ- issue frames may affect attitudes not only by ent subjects. But party ID can be changed by changing the accessibility of relevant consid- experiments, and such experiments are the erations, but also by changing the subjective key to understanding its mediating power. relevance of certain values to the issue at hand For example, Brader and Tucker (2008)use (Nelson et al. 1997). In this case, an experi- survey experiments to show that party cues mental intervention can identify the medi- can change Russians’ party IDs. And Ger- ating role of accessibility only if it primes ber, Huber, and Washington (2010)usea relevant considerations without affecting the field experiment to show that registering with subjective relevance of different values. And a party can produce long-term changes in by the same token, an experimental interven- party ID. The most promising path to secure tion will identify the mediating role of value inferences about party ID as a mediator is weighting only if it affects the subjective rel- to conduct studies in which interventions evance of different values without changing like these are coupled with manipulations of the accessibility of considerations. The gen- policy preferences, candidate evaluations, or eral challenge for experimental researchers, other treatments. And in general, the most then, is to devise manipulations that affect promising path to secure inferences about one mediator without affecting others.8 mediation is to design studies that include experimental manipulations of both treatments and mediators. 7 We do not take up two other limitations. One is the unreliability of instrumental-variable approaches to mediation in nonlinear models (Pearl 2010). The other is the “weak instruments” problem: when 4. Three Limitations of Experimental instruments are weakly correlated with the endoge- Mediation Analysis nous variables, IV estimators have large standard errors, and even slight violations of the exclusion restriction (cov[Z, e3] = 0 where Z is the instrument Despite its promise, the experimental for the endogenous mediator) may cause the esti- approach has limitations that merit more mator to have a large asymptotic bias (Bartels 1991; attention than they typically receive. It Bound, Jaeger, and Baker 1995). This is a large concern in econometric studies, where instruments are requires researchers to devise experimental often weak and exclusion-restriction violations likely. manipulations that affect one mediator with- But instruments that are specifically created by ran- out affecting others. Even if researchers suc- dom assignment to affect endogenous mediators are likely to meet the exclusion restriction and unlikely ceed, their estimates of indirect effects will to be “weak” by econometric standards. typically apply only to a subset of the exper- 8 Econometric convention permits the use of multiple imental sample. Finally, if causal effects are instruments to simultaneously identify the effects of a single endogenous variable. But estimators based on not identical for all members of a sample, multiple instruments have no clear causal interpre- then even a well-designed experiment may tation in a potential-outcomes framework; they are lead to inaccurate inferences about indirect instead difficult-to-interpret mixtures of local average treatment effects (Morgan and Winship 2007, effects. We discuss these limitations at length 212). This is why we recommend that experimenters in other work (Bullock, Green, and Ha 2010; create interventions that isolate individual mediators. Mediation Analysis Is Harder Than It Looks 515

Even if researchers isolate particular medi- lation of M – aˆbˆ is the causal effect of X on ators, they must confront another dilemma: M for one group of people times the causal some subjects never take a treatment even if effect of M on Y for another group of people. they are assigned to take it, and a treatment This product has no causal interpretation. It effect cannot be meaningfully estimated for is just an unclear mixture of causal effects for such people. Consequently, the experimental different groups.9 approach to mediation analysis produces esti- A related problem is that experiments can- mates of the average treatment effect not for not lead to accurate estimates of indirect all subjects but only for “compliers” who can effects when the effects of X on M are not the be induced by random assignment to take it same for all subjects or when the effects of M (Imbens and Angrist 1994). For example, if on Y are not the same for all subjects. When some subjects are assigned to watch a presi- we are not studying mediation, the assump- dential campaign advertisement while others tion of unvarying effects does little harm: if are assigned to a no-advertisement control the effect of randomly manipulated X on Y group, then the average effect of the ad can varies across subjects, and we regress Y on X, be identified not for all subjects but only for then the coefficient on X simply indicates the 1) treatment-group subjects who are induced average effect of X. But if the effects of X by random assignment to watch the ad, and and M vary across subjects, it will typically be (2) control-group subjects who would have difficult to estimate an average indirect effect been induced to watch the ad if they had (Glynn 2010). To see why, consider an exper- been assigned to the treatment group. One imental sample in which there are two groups may assume that the average indirect effect of subjects. In the first group, the effect of X is the same for these subjects as for others, on M is positive, and the effect of M on Y is but this is an assumption, not an experimen- also positive. In the second group, the effect tal result. Strictly speaking, estimates of the of X on M is negative, and the effect of M average indirect effect apply only to a subset on Y is also negative. In this case, the indi- of the sample. We can usually learn some- rect effect of X is positive for every subject in thing about the characteristics of this subset the sample: to slightly adapt the notation of 2009 166 72 (Angrist and Pischke , – ), but we Equations (1) and (3), aibi is positive for every can never know exactly which subjects belong subject. But aˆ, the estimate of the average to it. effect of X on M, may be positive, negative, An unintuitive consequence follows: even or zero. And bˆ, the estimate of the average if we use experiments to manipulate both a effect of M on Y, may be positive, negative, treatment and a mediator, we may not be able or zero. As a result, the estimate of the average to estimate an average indirect effect for our indirect effect, aˆbˆ, may be zero or negative – experimental sample or any subset of it. To even though the true indirect effect is positive see why, recall that the indirect effect of X for every subject. 1 3 on Y in Equations ( )–( )isab. By manipu- Such problems may arise whenever differ- lating X, we can recover aˆ, an estimate of the ent people are affected in different ways by average effect of X on M among those whose X and M. For example, Cohen (2003) wants value of X can be affected by the X manipula- to understand how reference-group cues (X) tion. And by manipulating M, we can recover affect attitudes toward social policy (Y). In his bˆ, an estimate of the average effect of M on Y experiments, politically conservative subjects among those whose value of M can be affected receive information about a generous wel- by the M manipulation. If these two popula- fare policy; some of these subjects are told tions are the same, aˆbˆ is a sensible estimate that the policy is endorsed by the Republican of the local average treatment effect. But if Party, while others receive no endorsement these two populations differ – if one set of subjects is affected by the manipulation of X 9 The same problem holds if we express the indirect but a different set is affected by the manipu- effect as c − d rather than ab. 516 John G. Bullock and Shang E. Ha information. Cohen’s findings are consis- of subjects is more sensitive than the rest of tent with cues (endorsements) promoting the sample to changes in X and to changes in systematic thinking (M) about the policy M, estimates of indirect effects will be biased. information, and with systematic thinking This problem cannot be traced to a deficiency in turn promoting positive attitudes toward in the methods that are often used to calcu- the policy (Cohen 2003, esp. 817).10 On the late indirect effects: it is fundamental, not a other hand, Petty and Wegener (1998, 345) matter of statistical technique (Robins 2003; and others suggest that reference-group cues Glynn 2010). inhibit systematic thinking about information, and that such thinking promotes the influence of policy details – which might be 5. An Agenda for Mediation Analysis expected to lead, in this case, to negative attitudes toward the welfare policy among the These limitations of experimental mediation conservative subjects. For present purposes, analysis – it requires experimenters to isolate there is no need to favor either of these the- particular mediators, produces estimates that ories or to attempt a reconciliation. We need apply only to an unknown subset of subjects, only note that they suggest a case in which and cannot produce meaningful inferences causal effects may be heterogeneous, and in about mediation when causal effects covary which mediation analysis is therefore diffi- within a sample – are daunting. Experiments cult. Let some subjects in an experiment be are often seen as simplifying causal inference, “Cohens”: for these people, exposure to refer- but taken together, these limitations imply ence group cues heightens systematic think- that strong inferences about mediation are ing (ai is positive), and systematic thinking likely to be difficult even when researchers makes attitudes toward a generous welfare use fully experimental methods of mediation policy more favorable (bi is positive). But analysis. Still, none of our cautions implies other subjects are “Petties”: for them, expo- that experiments are useless for mediation sure to reference group cues limits systematic analysis. Nor do they imply that experimental thinking (ai is negative), and systematic think- mediation analysis is no better than the non- ing makes attitudes toward a generous wel- experimental alternative. Unlike nonexperi- fare policy less favorable (bi is negative). Here mental methods, experiments offer – albeit again, the indirect effect is positive for every under limited circumstances – a systematic > subject because aibi 0 for all i.Butifthe way to identify mediation effects. And the experimental sample includes both Cohens limitations that we describe are helpful inas- and Petties, aˆ and bˆ may each be positive, much as they delineate an agenda for future negative, or zero. Conventional estimates of mediation analysis. the average indirect effect – aˆbˆ and related First, researchers who do not manipulate quantities – may therefore be zero or even mediators should try to explain why the medi- negative. ators are independent of the disturbances in Moreover, causal effects need not differ their regression equations – after all, the accu- so sharply across members of a sample to racy of their estimates hinges on this assump- make mediation analysis problematic. Con- tion. In practice, justifying this assumption ventional estimates of indirect effects will be entails describing unmeasured mediators that biased if a and b merely covary among sub- may link X to Y and explaining why these jects within a sample. For example, if a subset mediators do not covary with the measured mediators. Such efforts are rarely undertaken, but without them it is hard to hold out hope 10 This is only one aspect of Cohen (2003). As far as mediation is concerned, Cohen’s main suggestion is that nonexperimental mediation analysis will that reference-group cues affect policy attitudes not generate credible findings about mediation. by changing the extent to which people think system- Second, researchers who experimentally atically about policy information but by otherwise changing perceptions of the policies under consider- manipulate mediators should explain why ation. they believe that their manipulations are Mediation Analysis Is Harder Than It Looks 517 isolating individual mediators. This entails ulated. In this case, researchers should aim to describing the causal paths by which X may make multiple inferences for relatively homo- affect Y and explaining why each experimental geneous subgroups rather than single infer- manipulation affects only one of these paths. ences about the size of an indirect effect for The list of alternative causal paths may be an entire sample. extensive, and multiple experiments may be needed to demonstrate that a given intervention tends not to affect the alternative paths 6. Defenses of Conventional Practice in question. Third, researchers can improve the state In different ways, statisticians (Rosenbaum of mediation analysis simply by manipulating 1984; Rubin 2004; Gelman and Hill 2007, treatments and then measuring the effects of 188–94), social psychologists (James 1980; their manipulations on many different out- Judd and Kenny 1981, 607), and political sci- comes. To see how this can improve media- entists (King and Zeng 2006, 146–48; Glynn tion analysis, consider studies of the effects of 2010) have all warned that methods like the campaign contact on voter turnout. In addi- one proposed by Baron and Kenny (1986)are tion to assessing whether a particular kind likely to produce meaningless or inaccurate of contact increases turnout, one might also conclusions when applied to observational survey participants to determine whether this data. Why have their arguments not taken kind of contact affects interest in politics, feel- hold? Some of the reasons are mundane: the ings of civic responsibility, knowledge about arguments are typically made in passing, their where and how to vote, and other potential relevance to mediation analysis is not always mediators. In a survey or laboratory experi- clear, there are few such arguments in any ment, this extra step need not entail a new sur- one discipline, and scholars rarely read out- vey: relevant questions can instead be added side their own disciplines. But these are not to the post-test questionnaire. Because this the only reasons. Another part of the answer kind of study does not include manipulations lies with three defenses of nonexperimental of both treatments and mediators, it cannot mediation analysis, which can also be framed reliably identify mediation effects. But if some as criticisms of the experimental approach. variables seem to be unaffected by the treat- The first and most common defense is ment, one may begin to argue that they do that many mediators cannot be manipulated not explain why the treatment works. and that insistence on experimental media- Fourth, researchers should know that if the tion analysis therefore threatens to limit the effects of X and M vary from subject to subject production of knowledge (e.g., James 2008; within a sample, it may be impossible to esti- Kenny 2008). Manipulation of mediators is mate the average indirect effect for the entire indeed difficult in some cases, but we think sample. To determine whether this is a prob- that this objection falls short on several counts lem, one can examine the effects of X and (Bullock et al. 2008, 28–29). First, it follows M among different types of subjects. If the from a misunderstanding of the argument. effects differ little from group to group (e.g., No one maintains that unmanipulable vari- from men to women, whites to nonwhites, the ables should not be studied or that causal wealthy to the poor), we can be relatively con- inferences should be drawn only from exper- fident that causal heterogeneity is not affect- iments. The issue lies instead with the accu- ing our analysis.11 In contrast, if there are racy of nonexperimental inferences and the large between-group differences in the effects degree of confidence that we should place in of X or M, then mediation estimates made them. In the absence of natural experiments, for an entire sample may be inaccurate even dramatic effects, or precise theory about data- if X and M have been experimentally manip- generating processes – that is, in almost all situations that social scientists examine – non- 11 This is exactly the approach that Angrist, Lavy, and Schlosser (2010) take in their study of family size and experimental studies are likely to produce the long-term welfare of children. biased estimates of indirect effects and to 518 John G. Bullock and Shang E. Ha justify only weak inferences. Moreover, the do about learning the size of effects. But this objection is unduly pessimistic, likely because indifference to the size of effects is regret- it springs from a failure to see that many table. Our stock of accumulated knowledge variables that cannot be directly manipulated speaks much more to the existence of effects can be indirectly manipulated. Perhaps some than to their size, and this makes it difficult mediators defy even indirect manipulation, to know which effects are important. And but in light of increasing experimental cre- even if the emphasis on bounding results away ativity throughout the discipline – exempli- from zero were appropriate, there would not fied by several other chapters in this vol- be reason to think that conventional media- ume – we see more cause for optimism than tion analysis does a good job of helping us for despair. to learn about bounds. As Imai, Keele, Tin- A second objection is that the problem of gley, and Yamamoto (2010, 43) note, even bias in mediation analysis is both well under- a well-developed framework for sensitivity stood and unavoidable. The solution, accord- analysis cannot produce meaningful informa- ing to those who make this objection, is not to tion about mediation when important omit- embrace experimentation but to “build better ted variables are causally subsequent to the models” (e.g., James 1980). The first part of treatment. this objection is implausible: those who analyze mediation may claim to be aware of the threat of bias, but they typically act as though 7. Conclusion they are not. Potential mediators other than the one being tested are almost never dis- Experiments have taught us much about cussed in conventional analyses, even though treatment effects in politics, but our ability their omission makes bias likely. When sev- to explain these effects remains limited. Even eral mediators are hypothesized, it is com- when we are confident that a particular vari- mon to see each one analyzed in a separate set able mediates a treatment effect, we are usu- of regressions rather than collectively, which ally unable to speak about its importance in further increases the probability of bias. either an absolute sense or relative to other This makes the second part of the objec- mediators. Given this state of affairs, it is not tion – that the way to secure inferences about surprising that many political scientists want mediation is to “build better models” – infea- to devote more attention to mediation. sible. In the absence of experimental bench- But conventional mediation analysis, marks (e.g., LaLonde 1986), it is difficult to which draws inferences about mediation from know what makes a model better. Merely unmanipulated mediators, is a step back- adding more controls to a nonexperimental ward. These analyses are biased, and their mediation analysis is no guarantee of better widespread use threatens to generate a store estimates, common practice to the contrary. of misleading inferences about causal pro- It may well make estimates worse (Clarke cesses in politics. The situation would be 2009). better if we could hazard guesses about the A more interesting argument is that social size and direction of the biases. But we can scientists are not really interested in point rarely take even this small step with confi- estimation of causal effects (Spencer et al. dence because conventional mediation analy- 2005, 846). They report precise point esti- ses rarely discuss mediators other than those mates in their tables, but their real con- that have been measured. Instead, conven- cern is statistical significance, i.e., bounding tional analyses are typically conducted as effects away from zero. And for this pur- though they were fully experimental, with no pose, the argument goes, conventional meth- consideration of threats to inference. ods of mediation analysis do a pretty good A second, worse problem is the impres- job. The premise of this argument is cor- sion conveyed by the use and advocacy of rect: many social scientists care more about these methods: the impression that mediation bounding effects away from zero than they analysis is easy, or at least no more difficult Mediation Analysis Is Harder Than It Looks 519 than running a few regressions. In reality, not realize that they are conducting causal secure inferences about mediation typically analyses” and fail to justify the assumptions require experimental manipulation of both that underpin those analyses (Kenny 2008, treatments and mediators. But experimental 356). It would be a shame if political scien- inference about mediation, too, is beset by tists went the same route. We can stay on limitations. It requires researchers to craft track by remembering that inference about interventions that affect one mediator with- mediation is difficult – much more difficult out affecting others. If researchers succeed in than conventional practice suggests. this, their inferences will typically apply only to an unknown subset of subjects in their sample. And if the effects of the treatment and the References mediator are not the same for every subject in the sample, even well-designed experiments Angrist, Joshua D., Guido W. Imbens, and Don- may be unable to yield meaningful estimates ald B. Rubin. 1996. “Identification of Causal of average mediation effects for the entire Effects Using Instrumental Variables.” Journal sample. In the most difficult cases, it may be of the American Statistical Association 91: 444–55. impossible to learn about mediation without Angrist, Joshua D., Victor Lavy, and Analia making strong and untestable assumptions Schlosser. 2010. “Multiple Experiments for the about the relationships among observed and Causal Link between the Quantity and Qual- unobserved variables. ity of Children.” Journal of Labor Economics 28: 773 824 The proper conclusion is not that media- – . Angrist, Joshua D., and Jorn-Steffen¨ Pischke. tion analysis is hopeless but that it is difficult. 2009 Experiments with theoretically refined treat- . Mostly Harmless Econometrics: An Empiri- cist’s Companion. Princeton, NJ: Princeton Uni- ments can help by pointing to mediators that versity Press. merit further study. Experiments in which Baron, Reuben M., and David A. Kenny. 1986. mediators are manipulated are even more “The Moderator-Mediator Variable Distinc- promising. And analysis of distinct groups tion in Social Psychological Research: Concep- of subjects can strengthen mediation analy- tual, Strategic, and Statistical Considerations.” sis by showing us whether it is possible to Journal of Personality and Social Psychology 51: estimate average indirect effects for general 1173–82. populations or whether we must instead tai- Bartels, Larry M. 1991. “Instrumental and ‘Quasi- lor our mediation analyses to specific groups. Instrumental’ Variables.” American Journal of Political Science 35: 777–800. But because of the threats to inference that 2007 we describe, any single experiment is likely Bolger, Niall, and David Amarel. . “Effects of Social Support Visibility on Adjustment to to justify only the most tentative infer- Stress: Experimental Evidence.” Journal of Per- ences about mediation. Understanding the sonality and Social Psychology 92: 458–75. processes that mediate even a single treat- Bound, John, David A. Jaeger, and Regina M. ment effect will typically require a research Baker. 1995. “Problems with Instrumental program comprising multiple experiments – Variables Estimation When the Correlation experiments that address the challenges between the Instruments and the Endogenous described here. Explanatory Variable Is Weak.” Journal of the It is worthwhile to draw a lesson from American Statistical Association 90: 443–50. 2008 other social sciences, where manipulation of Brader, Ted A., and Joshua A. Tucker. . mediators is rare but mediation analysis is “Reflective and Unreflective Partisans? Exper- ubiquitous. In these disciplines, promulga- imental Evidence on the Links between Infor- mation, Opinion, and Party Identification.” tion of nonexperimental procedures has given Manuscript, New York University. rise to a glut of causal inferences about medi- Brader, Ted, Nicholas A. Valentino, and Elizabeth ation that warrant little confidence. Even Suhay. 2008. “What Triggers Public Opposi- the scholar who has arguably done most to tion to Immigration? Anxiety, Group Cues, and promote nonexperimental mediation analysis Immigration Threat.” American Journal of Polit- now laments that social scientists often “do ical Science 52: 959–78. 520 John G. Bullock and Shang E. Ha

Bullock, John G., Donald P. Green, and Shang Imai, Kosuke, Luke Keele, Dustin Tingley, and E. Ha. 2008. “Experimental Approaches to Teppei Yamamoto. 2010. “Unpacking the Mediation: A New Guide for Assessing Black Box: Learning about Causal Mechanisms Causal Pathways.” Unpublished manuscript, from Experimental and Observational Stud- Yale University. ies.” Unpublished manuscript, Princeton Uni- Bullock, John G., Donald P. Green, and Shang E. versity. Retrieved from http://imai.princeton. Ha. 2010. “Yes, But What’s the Mechanism? edu/research/files/mediationP.pdf (November (Don’t Expect an Easy Answer).” Journal of Per- 21, 2010). sonality and Social Psychology 98: 550–58. Imai, Kosuke, Luke Keele, and Teppei Yamamoto. Burdein, Inna, Milton Lodge, and Charles Taber. 2010. “Identification, Inference, and Sensitivity 2006. “Experiments on the Automaticity of Analysis for Causal Mediation Effects.” Statis- Political Beliefs and Attitudes.” Political Psychol- tical Science 25: 51–71. ogy 27: 359–71. Imbens, Guido W., and Joshua D. Angrist. 1994. Campbell, Angus, Philip E. Converse, Warren “Identification and Estimation of Local Aver- Miller, and Donald Stokes. 1960. The Ameri- age Treatment Effects.” Econometrica 62: 467– can Voter. Chicago: The University of Chicago 75. Press. Iyengar, Shanto, and Donald R. Kinder. 1987. Clarke, Kevin A. 2009. “Return of the Phantom News That Matters: Television and American Menace: Omitted Variable Bias in Political Opinion. Chicago: The University of Chicago Research.” Conflict Management and Peace Sci- Press. ence 26: 46–66. James, Lawrence R. 1980. “The Unmeasured Cohen, Geoffrey L. 2003. “Party over Policy: Variables Problem in Path Analysis.” Journal The Dominating Impact of Group Influence of Applied Psychology 65: 415–21. on Political Beliefs.” Journal of Personality and James, Lawrence R. 2008. “On the Path to Media- Social Psychology 85: 808–22. tion.” Organizational Research Methods 11: 359– Downs, Anthony. 1957. An Economic Theory of 63. Democracy. New York: HarperCollins. Judd, Charles M., and David A. Kenny. 1981. Fowler, James H., and Christopher T. Dawes. “Process Analysis: Estimating Mediation in 2008. “Two Genes Predict Voter Turnout.” Treatment Evaluations.” Evaluation Review 5: Journal of Politics 70: 579–94. 602–19. Frangakis, Constantine E., and Donald B. Rubin. Kenny, David A. 2008. “Reflections on Media- 2002. “Principal Stratification in Causal Infer- tion.” Organizational Research Methods 11: 353– ence.” Biometrics 58: 21–29. 58. Gelman, Andrew, and Jennifer Hill. 2007. Data King, Gary, and Langche Zeng. 2006.“The Analysis Using Regression and Multilevel Hierar- Dangers of Extreme Counterfactuals.” Political chical Models. New York: Cambridge University Analysis 14: 131–59. Press. LaLonde, Robert J. 1986. “Evaluating the Econo- Gerber, Alan S., and Donald P. Green. 2000. metric Evaluations of Training Programs with “The Effects of Canvassing, Telephone Calls, Experimental Data.” American Economic Review and Direct Mail on Voter Turnout: A Field 76: 604–20. Experiment.” American Political Science Review MacKinnon, David P., Chondra M. Lockwood, 94: 653–62. Jeanne M. Hoffman, Stephen G. West, and Gerber, Alan S., Gregory A. Huber, and Ebonya Virgil Sheets. 2002. “A Comparison of Meth- Washington. 2010. “Party Affiliation, Partisan- ods to Test Mediation and Other Intervening ship, and Political Beliefs: A Field Experiment.” Variable Effects.” Psychological Methods 7: 83– American Political Science Review 104: 720–44. 104. Glynn, Adam N. 2010. “The Product and Dif- Malhotra, Neil, and Jon A. Krosnick. 2007. ference Fallacies for Indirect Effects.” Unpub- “Retrospective and Prospective Performance lished manuscript, Harvard University. Assessments during the 2004 Election Cam- Green, Donald P., Shang E. Ha, and John G. Bul- paign: Tests of Mediation and News Media lock. 2010. “Enough Already about ‘Black Box’ Priming.” Political Behavior 29: 249–78. Experiments: Studying Mediation Is More Dif- Miller, Joanne M., and Jon A. Krosnick. 2000. ficult Than Most Scholars Suppose.” Annals of “News Media Impact on the Ingredients of the American Academy of Political and Social Sci- Presidential Evaluations.” American Journal of ences 628: 200–8. Political Science 44: 295–309. Mediation Analysis Is Harder Than It Looks 521

Morgan, Stephen L., and Christopher Winship. Robins, James M. 2003. “Semantics of Causal 2007. Counterfactuals and Causal Inference.New DAG Models and the Identification of York: Cambridge University Press. Direct and Indirect Effects.” In Highly Struc- Nelson, Thomas E. 2004. “Policy Goals, Public tured Stochastic Systems, eds. Peter J. Green, Rhetoric, and Political Attitudes.” Journal of Nils Lid Hjort, and Sylvia Richardson. Politics 66: 581–605. New York: Oxford University Press, 70– Nelson, Thomas E., Rosalee A. Clawson, and Zoe 81. M. Oxley. 1997. “Media Framing of a Civil Lib- Rosenbaum, Paul R. 1984. “The Consequences of erties Conflict and Its Effect on Tolerance.” Adjustment for a Concomitant Variable That American Political Science Review 91: 567–94. Has Been Affected by the Treatment.” Journal Pearl, Judea. 2010. “The Mediation Formula: of the Royal Statistical Society, Series A 147: 656– A Guide to the Assessment of Causal Path- 66. ways in Non-Linear Models.” Unpublished Rubin, Donald B. 2004. “Direct and Indi- manuscript, University of California, Los rect Causal Effects via Potential Outcomes.” Angeles. Retrieved from http://ftp.cs.ucla.edu/ Scandinavian Journal of Statistics 31: 161– ∼kaoru/r363.pdf (November 21, 2010). 70. Petty, Richard E., and Duane T. Wegener. 1998. Spencer, Steven J., Mark P. Zanna, and Geof- “Attitude Change: Multiple Roles for Persua- frey T. Fong. 2005. “Establishing a Causal sion Variables.” In The Handbook of Social Psy- Chain: Why Experiments Are Often More chology. vol. 1, 4th ed., eds. Daniel T. Gilbert, Effective Than Mediational Analyses in Exam- Susan T. Fiske, and Gardner Lindzey. New ining Psychological Processes.” Journal of York: McGraw-Hill, 323–90. Personality and Social Psychology 89: 845– Quinones-Vidal,˜ Elena, Juan J. Lopez-Garcia,´ 51. Maria Penaranda-Ortega,˜ and Francisco Zanna, Mark P., and Joel Cooper. 1974. “Disso- Tortosa-Gil. 2004. “The Nature of Social and nance and the Pill: An Attribution Approach Personality Psychology as Reflected in JPSP, to Studying the Arousal Properties of Disso- 1965–2000.” Journal of Personality and Social nance.” Journal of Personality and Social Psychol- Psychology 86: 435–52. ogy 29: 703–9.

AFTERWORD

CHAPTER 36

Campbell’s Ghost

Donald R. Kinder

Congratulations, intrepid reader! You have was never far from my mind, and he is back come a long way. After thirty-ﬁve chapters, again now.1 many hundreds of pages of text, and cascades This is presumptuous of me to say, but of footnotes, what more is there to say on I am going to go ahead and say it anyway: the subject? Exactly. My remarks are thus I believe that Campbell would have been brief. thrilled with both the conference proceedings The conference that led to this volume and the handbook that you are doing your began with remarks by Jamie Druckman, who best to keep balanced in your hands. Surely reminded us how right and proper it was he would have been delighted by the rapid that Northwestern University host our gath- rise to prominence of experimental meth- ering. Northwestern was Donald Campbell’s ods in political science, the sheer range and home for more than three decades, and no ingenuity in the application of experimental one was more important than Campbell in methods to political questions, and the dis- bringing the philosophy, logic, and practice play of sophistication in the statistical anal- of experimental methods to the social sci- yses of experimental data. For anyone with ences. In graduate school, I was trained in experimental methods by Barry Collins, a 1 Campbell wrote on an astonishing range of impor- student of Campbell’s. For more than thirty tant subjects and left a mark on half a dozen disci- years, ﬁrst at Yale and later at Michigan, I plines. Within the treasure trove that was Campbell’s contribution to methodological excellence are sem- have been teaching a graduate seminar on inal papers on the logic of experimentation (Camp- research design that draws heavily on Camp- bell 1957; Campbell and Stanley 1966); measurement bell’s work. Those who survived this experi- (Campbell and Fiske 1959; Webb et al. 1966); types of validity (Cook and Campbell 1979,ch.1); the design ence have taught similar courses elsewhere. and analysis of quasiexperiments (Campbell and Ross Alterations and additions there have been 1968; Cook and Campbell 1979); case studies in com- along the way, of course, but the core ideas parative politics (Campbell 1975); the experimenting society (Campbell 1969; Riecken and Boruch 1974); and the heart of the syllabus go back to Camp- and the philosophy of science (Campbell 1966, 1970, bell. Throughout the conference, Campbell 1974). Campbell died in 1996 at the age of 79.

525 526 Donald R. Kinder

18 16 14 12 10 8 6 4 Number of articles 2 0

Years Figure 36.1. Number of Articles Featuring Experiments Published in American Political Science Review, 1906–2009 an interest in what an experimental political results are published regularly, cited widely, science might mean, the handbook is brim- and discussed in increasingly sophisticated ming with exemplary illustrations and excel- ways (Druckman et al. 2006). The conference lent advice. served to illustrate this progress and to mark It was not always so. Thirty years ago, the occasion. Those of us who have pushed when Shanto Iyengar and I began to cook up for experimentation in political science are our research on television news (eventually no longer a beleaguered insurgency; we have published in 1987 as News That Matters), we arrived. From time to time, the proceedings chose experiments as our principal method in Evanston understandably took on an air of of inquiry. Trained as a social psychologist, festive celebration – in as much as political experimentation is what I knew. Trained as scientists are capable of such a thing. a political scientist, Iyengar was simply ahead I grew up in a small town in the Mid- of his time. As a team, we gravitated more or west. Celebration does not come easily to me. less naturally to experimentation – but at the Predictably, I worried about us celebrating time, experiments were far from common- too much. I remembered Campbell’s warn- place. When we began, experiments were ing to not expect too much from experiments. seen by much of the political science estab- Those of us determined to march under the lishment as exotic or irrelevant; experimen- experimental banner, Campbell cautioned, tation was a subject not fit for serious dis- should be prepared for a “poverty” of results cussion. Experiments? Experiments were what (Campbell and Stanley 1966): went on over in the chemistry building or in psychology laboratories – they had nothing If, as seems likely, the ecology of our science is one in which there are available many more to do with how political scientists conducted wrong responses than correct ones, we may their business. The science of politics, accord- anticipate that most experiments will be disap- ing to the standard assumption of the time, pointing. . . . We must instill in our students could not be an experimental one. the expectation of tedium and disappointment Things change. Experiments are no longer and the duty of thorough persistence, by now 2 eccentric (Figure 36.1). Today, experimental so well achieved in the biological and physical sciences. (3) 2 Figure 36.1 updates a figure presented by Druckman Experimentation is a powerful method of and colleagues (2006) in their essay on experimentation for the 100th anniversary issue of the American testing, but it is no magic elixir. An exper- Political Science Review. iment is no substitute for a good idea. An Campbell’s Ghost 527 experiment cannot compensate for muddy Give a small boy a hammer, and he will find conceptualization. It cannot do the work that that everything he encounters needs hammer- must be done by measurement. Even the best ing. It comes as no particular surprise to dis- experiment cannot answer a question poorly cover that a scientist formulates problems in posed. We should not ask too much of our a way which requires for their solution just experiments. those techniques in which he himself is especially skilled. (28) There is a larger and more important point here, I think. In our enthusiasm for It was on these grounds that Palfrey and experimental methods, we may be in dan- I argued for expanding the methodologi- ger of overlooking an important epistemo- cal repertoire of political science to include logical premise, one that is central to Camp- experimentation. We did not envision the day bell’s work. Some years ago, thrown together when experiments would dominate the scien- in that best of all possible worlds, the Center tific study of politics. Such a goal seemed to us for Advanced Study in the Behavioral Sci- not only unrealistic, but also undesirable. Our ences, Tom Palfrey and I collaborated on an goal was diversification. We believed that “a edited volume on experimentation for polit- political science based on a variety of empiri- ical science (Kinder and Palfrey 1993). We cal methods, experimentation prominent among identified and reprinted a set of excellent them, is both within our reach and well worth applications of experimental methods to core reaching for” (Kinder and Palfrey 1993, 1; problems in politics, and we wrote an intro- italics in original). ductory essay making the case for includ- I find it surprising, and not a little ironic, ing experimentation within the method- that I now find myself obliged to repeat that ological repertoire of modern political argument, but in reverse. Experimentation is science. a powerful tool. Indeed, for testing causal The essay drew heavily on Campbell. This propositions, there is nothing quite like a is perhaps a polite way to put it. For my part, I well-designed experiment. The analysis of triedmybesttochannel Campbell. Palfrey and nonexperimental data to test causal claims, I argued that all methods are fallible. None as Green and Gerber (2002) put it in their can provide a royal road to truth. What to fine essay on field experiments, is informa- do in the face of this inescapable predica- tive only to the extent that “nature performs ment? Campbell says to pursue research ques- a convenient randomization on our behalf or tions from a variety of methodological angles, that statistical correctives enable us to approx- each one fallible, but fallible in different ways. imate the characteristics of an actual exper- Dependable knowledge is grounded in no sin- iment” (808). Experiments have additional gle method, but rather in convergent results virtues: they enable the analytic decomposi- across complementary methods. Hypotheses tion of complex forces into component parts, prove their mettle by surviving confrontation turn up “stubborn facts” that provoke the- with a series of equally prudent but distinct oretical invention, apply to various levels of tests. aggregation equally, and accelerate interdis- Depending on a single method is risky ciplinary conversations (Kinder and Palfrey business, epistemologically speaking. It is [1993] develop each of these points). I believe also a narrowing business. Relying exclu- all this. Experiments are exceedingly useful, sively on one method constricts the range of but they are not infallible. questions that seem worth pursuing. Which Like all methods, experiments are accom- questions are interesting and which are not panied by both shortcomings and strengths. is seen through the filter of what one is These include the inevitable ambiguity that able to do. Methodological preoccupations surrounds the meaning of experimental treat- inevitably and insidiously shape substantive ments, the unsuitability of experimental agendas. This is the “law of the instrument” methods to some of our core questions, at work (Kaplan 1964): and the hazards that inescapably attend 528 Donald R. Kinder generalizing from experimental results. All not only to the issue at hand, but also to methods are fallible. the way the frames resonate with deeply Going forward, this means taking Camp- held beliefs about race and gender. Winter bell’s insistence on multiple methods seri- establishes this point with well-designed, sub- ously. Going forward, it means that we should tle experiments and with sharp analysis of be aware not only of what experiments can carefully chosen national survey data. Again, tell us, but also of what they cannot. Going Winter is much more convincing (in my forward, it means carrying out programs of book) on how everyday ideas about gender research that incorporate methods in addi- and race become implicated in the public’s tion to experimentation. I think this is more views on matters of policy for having pro- difficult than it might seem. It requires not vided two kinds of empirical demonstrations, just extra effort, but a special kind of thinking: not just one. moving back and forth between ever deeper conceptual analysis of the question under “Politics is an observational, not an experi- investigation, on the one hand, and judi- mental science.” So said A. Lawrence Lowell cious exploitation of particular methodologi- in his presidential address to the American 3 cal complementarities, on the other. Perhaps Political Science Association in 1909. One an example or two of what I believe that we hundred years later, this is no longer so. Now should be up to will help clarify the point I we would say – we would be required to say – am straining to make. “Politics is an observational science and an Given multiple sources of cultural differ- experimental science.” No doubt, Campbell ence in society, when and why does one would be pleased. difference become the basis for political competition and conflict rather than another? Posner’s (2004, 2005) answer to this impor- References tant and often overlooked question hinges on the degree to which cultural groups are Campbell, Donald T. 1957. “Factors Relevant to useful vehicles for political coalition build- the Validity of Experiments in Social Settings.” Psychological Bulletin 54: 297–312. ing. To test his theory, Posner takes clever 1966 advantage of a natural experiment, the draw- Campbell, Donald T. . “Pattern Matching as an Essential in Distal Knowing.” In The Psy- ing of the border between the African nations chology of Egon Brunswik, ed. Kenneth R. Ham- of Zambia and Malawi by the British South mond. New York: Holt, Rinehart and Winston, 1891 African Company in – a border drawn 81–106. for purely administration purposes, with no Campbell, Donald T. 1969. “Reforms as Experi- attention to the geography of the indigenous ments.” American Psychologist 24: 409–29. peoples living nearby. The differences that Campbell, Donald T. 1970. “Natural Selection Posner observes associated with living on one as an Epistemological Model.” In A Handbook side of the border as against the other sup- of Methods in Cultural Anthropology, eds. Raoul port his account, but – and this is the point Naroll and Ronald Cohen. New York: Natural History Press, 51–85. I want to stress here – the results of the nat- 1974 ural experiment are much more convincing Campbell, Donald T. . “Evolutionary Epis- temology.” In The Philosophy of Karl Popper,ed. when fortified by convergent results emanat- Paul Arthur Schilpp. La Salle, IL: Open Court ing from Posner’s analysis of campaigns and Press., 413–63. election returns. Campbell, Donald T. 1975. “‘Degrees of Free- A second example, equally excellent, dom’ and the Case Study.” Comparative Political comes from Winter’s (2008) research on Studies 8: 178–93. “dangerous frames.” Winter is interested in Campbell, Donald T., and Donald W. Fiske. 1959. the intersection between elite rhetoric and “Convergent and Discriminant Validation by public opinion. He shows that politicians, interest groups, and parties routinely frame 3 Lowell’s speech quoted by Druckman et al. (2006, issues in ways that prime audiences to respond 627). Campbell’s Ghost 529

the Multitrait-Multimethod Matrix.” Psycholog- Kaplan, Abraham. 1964. The Conduct of Inquiry. ical Bulletin 56: 81–105. San Francisco: Chandler Press. Campbell, Donald T., and H. L. Ross. 1968.“The Kinder, Donald R., and Thomas R. Palfrey. 1993. Connecticut Crackdown on Speeding: Time- “On Behalf of an Experimental Political Sci- Series Data in Quasi-Experimental Analysis.” ence.” In Experimental Foundations of Political Law and Society Review 3: 33–53. Science, eds. Donald R. Kinder and Thomas Campbell, Donald T., and Julian C. Stanley. 1966. R. Palfrey. Ann Arbor: University of Michigan Experimental and Quasi-Experimental Designs for Press, 1–39. Research. Chicago: Rand McNally. Posner, Daniel N. 2004. “The Political Salience of Cook, Thomas D., and Donald T. Camp- Cultural Differences: Why Chewas and Tum- bell. 1979. Quasi-Experimentation: Design and bukas Are Allies in Zambia and Adversaries in Analysis for Field Settings. Chicago: Rand Malawi.” American Political Science Review 98: McNally. 529–45. Druckman, James N., Donald P. Green, James Posner, Daniel N. 2005. Institutions and Ethnic Pol- H. Kuklinski, and Arthur Lupia. 2006.“The itics in Africa. New York: Cambridge University Growth and Development of Experimental Press. Research in Political Science.” American Politi- Riecken, Henry W., and Robert F. Boruch. 1974. cal Science Review 100: 627–35. Social Experimentation: A Method for Planning Green, Donald P., and Alan S. Gerber. 2002. and Evaluating Social Innovations. New York: “Reclaiming the Experimental Tradition in Academic Press. Political Science.” In Political Science: State of Webb, Eugene J., Donald T. Campbell, Richard the Discipline, eds. Ira Katznelson and Helen V. D. Schwartz, and Lee B. Sechrest. 1966. Unob- Milner. New York: Norton, 805–32. trusive Measures: Nonreactive Research in the Iyengar, Shanto, and Donald R. Kinder. 1987. Social Sciences. Chicago: Rand McNally. News That Matters: Television and American Winter, Nicholas J. G. 2008. Dangerous Frames: Opinion. Chicago: The University of Chicago How Ideas about Race and Gender Shape Public Press. Opinion. Chicago: The University of Chicago Press.

Name Index

Aarts, Henk, 156 Anderson, John, 158 Abelson, Robert, 145, 163, 431 Anderson, Lisa, 247 Abrajano, Marisa, 326 Anderson, Mary, 176 Abramowitz, Alan, 118 Anderson, Norman, 157 Achen, Christopher, 130, 274 Andreoli, Virginia, 150, 203 Adams, William, 118, 120, 229 Anen, Cedric, 248 Adato, Michelle, 483 Angrist, Joshua, 22, 30, 31, 117, 118, 126, 127, Addonizio, Elizabeth, 124 235, 448, 453, 497, 498, 512, 513, 515 Ajzen, Icek, 143 Annan, Jeannie, 393 Albertson, Bethany, 17, 448 Ansolabehere, Stephen, 45, 76, 77, 78, 81, 118, Alchian, Armen, 340 120, 130, 191, 192–193, 196, 217, 219, 220, Aldrich, John, 54–55, 194, 296, 451 224, 451 Alexander, Deborah, 291 Apostle, Richard, 104 Alger, Dean, 194, 195 Appiah, O., 325 Alker, Henry, 431 Aragones, Enriqueta, 374, 381 Allen, Terry, 282 Arai, Tatsushi, 424 Allport, Gordon, 141, 142, 306 Arceneaux, Kevin, 31, 43, 126, 132, 145, 225, 229, Allum, Nick, 176 231, 235, 237, 482, 483, 486 Almond, Gabriel, 250, 251, 370 Argyle, Lisa, 266 Althaus, Scott, 204 Aronson, Elliot, 28, 29, 34, 35, 44, 202 Alvarez, R. Michael, 326 Aronson, Jessica, 248 Amarel, David, 513 Aronson, Joshua, 175 Amengua, Matthew, 260 Arpan, Laura, 325 American Psychological Association, 505 Asch, Solomon, 208, 276 Amodio, David, 163 Ascherio, A., 273 Anderhub, Vital, 246 Ash, Ronald, 54 Andersen, Kristi, 291 Ashraf, Nava, 250 Andersen, Susan, 193 Austen-Smith, David, 94 Anderson, Craig, 43, 209 Avila, Juan, 483, 486, 489–490

531 532 Name Index

Avila, Mauricio, 483, 486, 489–490 Benton, Alan, 417, 418–419 Axelrod, Robert, 340, 342, 416 Benz, Matthias, 42 Azari, Julia, 17 Bereby-Meyer, Yoella, 402 Berelson, Bernard, 166, 273, 278, 370, 487, 496 Babbitt, Paul, 21, 77, 451 Berent, Matthew, 142, 144 Bachtiger, Andre, 260 Berg, Joyce, 243, 245 Back, K., 278 Bergan, Daniel, 125, 238 Baer, Denise, 118, 120, 229, 231 Berger, Jonah, 130, 166 Bahry, Donna, 246, 250, 348 Beriker, Nimet, 417, 424 Bailenson, Jeremy, 8, 80 Berinsky, Adam, 143, 192 Bailey, Delia, 85 Berkowitz, Leonard, 45 Baker, Regina, 514 Berman, Maureen, 418 Banaji, Mahzarin, 161 Bertrand, Marianne, 386 Banerjee, Abhijit, 390, 392 Besley, Timothy, 388 Banerjee, Rukmini, 392 Betsch, Tilman, 158 Banker, Brenda Bettinger, Eric, 453 Banks, Curtis, 277 Bhatia, Tulikaa, 483 Banuri, Sheheryar, 252 Bhavnani, Rikhil, 496, 501 Banwart, Mary, 292, 296 Bianco, William, 243, 360, 361, 362–363 Barabas, Jason, 263 Billig, Michael, 60 Barber, Benjamin, 259 Bizer, George, 150 Bargh, John, 156, 159, 163, 164, 165, 166 Bjorkman, Martina, 392 Barnard, John, 478 Black, Duncan, 95, 166 Barnes, Helen, 504 Blair, Irene, 193 Baron, David, 400–401, 402, 404 Blake, Robert, 417, 419 Baron, Reuben, 20, 46, 509, 511, 517 Blattman, Christopher, 386, 393 Barr, Abigail, 250 Bloom, Erik, 453 Barreto, Matt, 320 Blundell, Richard, 85 Barrios, Alfred, 157 Bobo, Lawrence, 320, 323, 328, 329 Bartels, Larry, 76, 105, 130, 187, 202, 216, Bobrow, Davis, 414 514 Bodenhausen, Galen, 155 Bartos, Otto, 419 Boettcher, William, 202 Bass, Bernard, 417 Bohan, Lisa, 192 Bassi, Anna, 63, 64 Bohman, James, 259 Battaglini, Marco, 378, 379, 402 Bohnet, Iris, 247, 248, 250, 251 Baum, Mathew, 204 Boisjoly, Johanne, 282 Baumgartner, Hans, 142–143 Boix, Charles, 384 Bazerman, Max, 406 Bolger, Niall, 513 Beach, Rebecca, 232 Bolls, Paul, 208 Beasley, Ryan, 436, 438, 440 Bolton, Gary, 402, 404 Beasley, Timothy, 487 Bond, Charles, 276 Beck, Paul, 201 Bond, Joe, 116, 117 Beckman, Matthew, 79, 80 Bond, Rod, 276 Beer, Francis, 433 Bonetti, Shane, 66 Beersman, Bianca, 437 Bonham, Matthew, 422 Behr, Roy, 202 Boninger, David, 149 Beilenson, Jeremy, 192 Bonoma, Thomas, 416 Bellemare, Charles, 247 Borenstein, Michael, 21 Belli, Robert, 143 Borgida, Eugene, 194 Bem, Daryl, 141 Bornstein, Gary, 252 Ben-Ner, Avner, 252 Bortolotti, Lisa, 68 Bench, Lawrence, 282 Boruch, Robert, 525 Benhabib, Seyla, 260 Bos, Angela, 192 Benjamin, Daniel, 232 Bositis, David, 73, 118, 120, 229, 231 Bentley, Arthur, 339 Bottom, William, 364 Name Index 533

Boudreau, Cheryl, 178, 182 Camerer, Colin, 6, 8, 29, 58, 63, 248, 340, 341, Bound, John, 514 347, 349, 389 Bourne, Lyle, 433 Camp, Scott, 282 Bowers, Jake, 447, 448, 461, 466, 467, 468, 470, Campbell, Angus, 53, 144–147, 148, 190, 274, 471, 476, 486, 491 496, 509 Bowles, Samuel, 6, 8, 347, 349 Campbell, David, 283, 453 Boyd, Robert, 6, 347, 349 Campbell, Donald, 5–6, 7, 18–19, 28, 30, 42, 46, Boydston, John, 148 48, 107, 261, 262, 268, 525, 526 Bradburn, Norman, 76 Campbell, James, 197 Brader, Ted, 60, 67, 150, 151, 209, 222, 224, 225, Cann, Arnie, 143 312, 509, 514 Cantril, Hadley, 142 Brady, Henry, 105, 120, 130, 228, 460 Capella, Joe, 263 Brannan, Tessa, 124 Cardenas, Juan-Camilo, 349 Brau, Shawn, 5, 157, 158, 218 Carew Johnson, Jessica, 249 Brauer, Markus, 436 Carlsmith, J. Merrill, 28, 29, 34, 35, 44 Braumoeller, Bear, 447 Carmines, Edward, 105, 107, 109–112, 307–308 Braverman, Julia, 208 Carnevale, Peter, 420 Brehm, John, 248 Carnot, Catherine, 232 Breuning, Marijke, 144, 145, 158, 188 Carpenter, Jeffrey, 250 Brewer, Marilyn, 44, 149 Carpentier, Francesca, 81 Brians, Craig, 77 Carrington, Peter, 275 Brock, Timothy, 149 Carroll, Susan, 296 Brock, William, 483 Carruba, Clifford, 404 Brody, Richard, 4, 54, 197 Carson, Richard, 142, 144 Broockman, David, 196 Case, Anne, 487 Brooks-Gunn, Jeanne, 502, 505 Caselli, Graziella, 85 Broome, Benjamin, 419, 422, 424 Cason, Timothy, 252 Brown, Bert, 415 Cassese, Erin, 205 Brown, Rupert, 437 Cassino, Daniel, 157 Brown, Stephen, 4 Castillo, Marco, 249 Browne, Eric, 404 Caucutt, Elizabeth, 453 Brownstein, Charles, 4 Cesarini, David, 248 Bruck, H. W., 430 Chadwick, Richard, 422 Bruin, Ralph, 163 Chaiken, Shelly, 141, 142, 143, 155, 156, 159, Brumberg, Bruce, 196 193 Buchan, Nancy, 250, 344 Chalmers, Iain, 130 Buck, Alfred, 29 Chambers, Simone, 258, 259, 260–262 Bullock, John, 510, 512, 514, 517 Chammah, Albert, 91 Burdein, Inna, 513 Chan, Steve, 434 Burgess, Robin, 388 Charness, Gary, 252 Burhans, David, 399 Chatt, Cindy, 84 Burkhalter, Stephanie, 259 Chattopadhyay, Raghabendra, 387, 388, 390 Burkhauser, Richard, 85 Chen, Annette, 156, 165 Burnstein, Eugene, 80 Chen, Yan, 63 Burrell, Barbara, 295, 296 Cheng, Clara, 156, 159, 161, 163 Burrows, Lara, 156, 165 Chin, Michelle, 116, 117 Bushman, Brad, 43, 209 Chollet, Derek, 430 Butemeyer, John, 292 Chong, Alberto, 390, 391 Butler, Daniel, 15, 125, 196 Chong, Dennis, 43, 106, 150, 151, 218, 224, 320, Butler, Frances, 420, 421, 425 330, 331 Bystrom, Dianne, 292, 296 Christakis, Nicholas, 273, 275, 484 Citrin, Jack, 244, 314 Cacioppo, John, 96, 97, 149, 158, 207, 331, 484 Clark, M. H., 452 Caldeira, Gregory, 172, 173, 174, 273 Clarke, Kevin, 510, 518 Calvert, Randall, 373 Claussen-Schultz, A., 419 534 Name Index

Clawson, Rosalee, 150, 208, 324, 329, 513, Dale, Allison, 124, 125, 230 514 Dalton, Russell, 201 Clayton, Laura, 259 Danielson, Anders, 247 Clingingsmith, David, 22 Daniere, Amrita, 250 Clinton, Joshua, 219, 221, 224, 225, 364 Darcy, Robert, 290 Coady, David, 483 Darley, John, 36 Coate, Stephen, 374, 376 Dasgupta, Sugato, 374, 377 Cobb, Michael, 15, 144 Dasovic, Josip, 274 Cobo-Reyes, N. Jimenez, 252 Davenport, Tiffany, 115, 504 Cochran, William, 460 Davidson, Brian, 159 Cohen, Dov, 36 Davis, Belinda, 176 Cohen, Geoffrey, 515, 516 Davis, Douglas, 403 Cohen, Jonathan, 248 Davis, James, 143 Cohen-Cole, Ethan, 275 Dawes, Christopher, 248, 509 Colditz, G. A., 273 Dawes, Robyn, 250, 341, 453 Coleman, Eric, 346 Dawson, Michael, 110, 301, 320 Coleman, James, 483 De Dreu, Carsten, 421, 437 Coleman, Stephen, 377 De Figueiredo, Rui, 313 Colleau, Sophie, 191 De La O, Ana, 388, 390, 391 Collett, Derek, 431 de Quervain, Dominique, 29 Collier, Kenneth, 372, 373 De Vries, Nanne, 437 Collier, Paul, 386 Dean, Norma, 292 Collins, Allan, 159 Deaton, Angus, 126, 391 Collins, Barry, 30 DeBell, Matthew, 174 Collins, Nathan, 8, 80, 192 Deess, E. Pierre, 259, 260 Conger, Kimberly, 151, 189, 195 Delgado, Mauricio, 29, 248 Conley, Timothy, 483 Delli Carpini, Michael, 190, 261, 496 Conlon, Donald, 420 DeMarzo, Peter, 483 Conover, Pamela, 163, 192, 261, 302, 304 Dennis, J. Michael, 84 Converse, Jean, 54 Denton, Frank, 210 Converse, Philip, 53, 94, 108, 142, 145, 190, 259, Depositario, Dinah, 54 274, 370, 496, 509 Deshpande, Rohit, 325 Cook, Fay, 261 Deutsch, Morton, 417, 419, 421 Cook, Karen, 250 Deutsch, Roland, 163 Cook, Thomas, 18–19, 28, 42, 46, 48, 149, 261, Devine, Dennis, 259 525 Devine, Patricia, 163 Cook, Timothy, 194, 195 Dickhaut, John, 44, 243, 245 Cooper, Joel, 513 Dickson, Eric, 59, 60, 61, 62, 64 Cosmides, Leda, 409 Dickson, William, 36 Cover, Albert, 196 Diermeier, Daniel, 399, 400, 401, 403, 404, 405, Cowden, Jonathan, 39 406, 407, 408, 409 Cox, David, 460, 463, 468, 469, 471, 472 Dietz-Uhler, Beth, 220–221 Cox, Gary, 360 Dijksterhuis, Ap, 156, 165 Cox, Gertrude, Dipboye, Robert, 7 Cox, James, 61, 246, 341 Dittmar, Kelly, 296 Cramer Walsh, Katherine, 260, 261 Djankov, Simeon, 386 Crandall, Christian, 80 Dodd, Stuart, 280 Crewe, Ivor, 261 Dodge, Richard, 120 Crigler, Ann, 194, 195, 203, 208 Doherty, David, 117, 123, 124, 125 Cronk, Lee, 250 Dolan, Kathleen, 295 Croson, Rachel, 247, 249, 250, 252, 344 Dolan, Thomas, 194, 195 Crow, Wayman, 416 Donakowski, Darrell, 174 Cunningham, William, 161 Donnerstein, Edward, 45 Curhan, Jared, 406, 408 Dovidio, John, 163 Curley, Nina, 29 Dowling, Conor, 117, 124, 125 Name Index 535

Downs, Anthony, 166, 196, 370, 509 Fershtman, Chaim, 249 Druckman, Daniel, 4, 6, 21, 354, 416, 417, Festinger, L., 278 418–419, 420, 421, 422, 423, 424, 425, 526 Fiala, Nathan, 386 Druckman, James, 3, 4, 8, 15, 16, 29, 37, 42, 43, Finan, Frederico, 9, 388 53, 62–63, 67, 75, 106, 108, 143, 149, 150, 151, Finkel, Steven, 78 207, 218, 224, 265, 330, 331, 404, 423, 424, 528 Fiorina, Morris, 91–94, 145, 196, 354, 355, Dryzek, John, 258, 260 356–357, 364, 399 DuBois, William E. B., 306 Fischbacher, Urs, 29, 247, 248, 249, 250 Duﬂo, Esther, 387, 388, 390, 392, 487 Fischle, Mark, 190 Dufwenberg, Martin, 252 Fish, L., 420, 421, 425 Dumay, Adrie, 406 Fishbein, Martin, 143, 232 Duncan, Greg, 282, 502, 505 Fisher, Roger, 418 Dunford, Benjamin, 259 Fisher, Ronald, 230, 447, 469 Dunwoody, Sharon, 208 Fishkin, James, 258, 259, 260, 262, 263–264, 277 Durlauf, Steven, 483 Fiske, Donald, 525 Fiske, Susan, 145, 191 Eagly, Alice, 141, 142, 143, 166 Fitzgerald, Jennifer, 274 Eavey, Cheryl, 6, 364 Flament, Claude, 60 Ebner, Noam, 425 Flanagan, Michael, 7 Eccles, Jacque, 282 Fleming, Monique, 325 Eckel, Catherine, 246, 247, 249, 252 Fletcher, Jason, 275 Eldersveld, Samuel, 4, 15, 117, 118–119, 120, 229 Fletcher, Joseph, 105, 109 Eliasoph, Nina, 260, 261, 269 Fling, Sheila, 292 Ellsworth, Phoebe, 28, 29, 34, 35 Follett, Mary, 418 Elster, Jon, 346 Fong, Geoffrey, 508, 518 Endersby, James W., 361, 367 Forehand, Mark, 325 Engelmann, Dirk, 246 Forgette, Richard, 151 Engle-Warnick, Jim, 246 Forlenza, Samuel, 420, 421, 425 Entman, Robert, 201 Forsythe, Robert, 375, 376, 378, 403 Eraslan, Hulya,¨ 399, 400, 405 Fouraker, Lawrence, 415 Erikson, Robert, 118, 138, 196 Fowler, C., 192 Erisen, Cengiz, 157, 158, 166 Fowler, James, 8, 22, 248, 273, 275, 484, 509 Eron, Leonard, 202 Fox, Richard, 296 Esterling, Kevin, 9, 261, 263 Foyle, Douglas, 435 Etzioni, Amitai, 416, 417 Frangakis, Constantine, 478, 511 Eulau, Heinz, 436 Frank, Robert, 248 Eveland, William, 208 Franklin, Charles, 45, 46 Everts, Philip, 438 Franklin, Mark, 404 Franz, Michael, 225 Falk, Armin, 9, 250 Franzese, Robert, 46 Farnham, Barbara, 435 Frazetto, Giovanni, 39 Farrar, Cynthia, 277 Frechette,´ Guillaume, 5, 6, 402, 404, 405, 406, 408 Fazio, Russell, 156, 159, 163, 331 Frederick, Brian, 295, 296 Fearon, James, 386, 391 Frederickson, Mark, 470, 471, 476 Feddersen, Timothy, 97, 379 Freedman, David, 106, 462, 470, 477 Fehr, Ernst, 6, 8, 29, 59, 60, 61, 63, 247, 248, 249, Freedman, Paul, 78–79, 225 250, 251, 253, 341, 347, 349, 404 Freeze, Melanie, 296 Feingold, Alan, 166 Freidman, Daniel, 51, 55 Feldman, Stanley, 157, 163, 192, 193, 310, 311, Frey, Bruno, 251 312, 329 Fridkin, Kim, 295, 296 Fendreis, John, 404 Fried, Linda, 85 Fenno, Richard, 195, 243 Friedman, Daniel, 341 Ferejohn, John, 93, 340, 400–401, 402 Friedman, Milton, 44, 56 Ferraz, Claudio,´ 9, 388 Frohlich, Norman, 6, 94, 95, 266, 344 Ferree, Myra, 290 Frontera, Edward, 420, 421, 425 536 Name Index

Fudenberg, Drew, 340 Gneezy, Uri, 63, 247, 249 Fukui, Haruhiro, 438 Goldberg, Bernard, 201 Fung, Archon, 260, 261, 263 Goldberg, Philip, 290 Funk, Carolyn, 190 Goldgeier, James, 430, 431 Goldstein, Kenneth, 78–79, 225, 249 Gachter,¨ Simon, 29, 59, 60, 61, 63, 247, 341 Gollwitzer, Peter, 232 Gaertner, Samuel, 163 Gonzales, Marti, 28, 29, 34, 35 Gaes, Gerald, 282 Gonzales-Ocantos, Ezequiel, 392 Gafﬁe,´ B., 150 Goodin, Robert, 205 Gailmard, Sean, 379, 403, 404, 406, 408 Gordon, H. Scott, 345 Gaines, Brian, 7, 43, 77, 109, 231, 330, 450 Gordon, Michael, 54 Gakidou, Emmanuela, 483, 486, 489–490 Goren, Amir, 193 Galinsky, Adam, 419 Gosnell, Harold, 117, 118, 119, 229 Gamson, William, 261, 262–263, 399, 404 Gould, Stephen, 173, 174 Garcıa´ Bedolla, Lisa, 124, 231, 232, 234, Gouldner, Alvin, 417 327 Gouws, Amanda, 109 Gardner, Roy, 5, 15, 341, 345, 346–347 Govender, Rajen, 159 Gardner, Scott, 151 Govorun, Olesya, 156, 159, 161, 163 Garramone, Gina, 221 Graber, Doris, 171, 194 Garrison, Jean, 437 Green, Donald, 3, 4, 5, 6, 7, 15, 24, 31, 32, 38, 42, Garst, Jennifer, 149 43, 73, 75, 115, 116, 117, 118, 119, 121, Gartner, Scott, 150 122–123, 124, 125, 126, 133, 138, 218, 229, Gastil, John, 259, 260 230–232, 234, 235, 236, 238, 249, 277, 280, Gaudet, Hazel, 273, 487 314, 327, 328, 331, 354, 386, 389, 390, 391, Gaun, Mei, 124 447, 462, 474, 482, 483, 486, 487, 489, 496, Gawronski, Bertram, 155, 163 499, 500, 501, 502, 503, 504, 510, 512, 514, Gay, Claudine, 502 517, 527, 528 Geer, John, 78 Green, Erick, 393 Gelman, Andrew, 468, 511, 517 Green, Jennifer, 277, 390 Genovese, Michael, 295, 296 Green, Justin, 424 Gerber, Alan, 6, 7, 15, 24, 31, 32, 38, 42, 43, 115, Greenberg, Jerald, 7 116, 117, 118, 119, 121, 122–123, 124, 125, Greenwald, Anthony, 156, 159, 163, 232 126, 133, 218, 229, 230, 231, 232, 235, 236, Greig, Fiona, 248, 250 237, 238, 280, 328, 331, 447, 483, 486, 496, Grelak, Eric, 364 500, 501, 502, 503, 504, 512, 514, 527 Griggs, Richard, 61 Gerber, Elisabeth, 376 Groseclose, Tim, 374 Geva, Nehemia, 6, 116, 117, 434 Grosser, Jens, 379 Gibson, James, 109, 172, 173, 174, 313 Grossman, Matthew, 223 Gigerenzer, Gerd, 29 Groves, Robert, 54 Gilbert, Daniel, 191 Gruder, Charles, 149 Gilens, Martin, 5, 15, 180, 312, 320 Guala, Francesco, 17–18, 44, 51 Gilliam, Franklin, 79, 80, 311, 320, 322, 330 Guan, Mei, 124 Gillis, J. Roy, 325 Guernaschelli, Serena, 97 Gimpel, James, 43, 218, 235, 236, 238 Guetzkow, Harold, 4, 422 Gintis, Herbert, 6, 347, 349 Guilford, J. P., 423 Giovannucci, E., 273 Gunther, Albert, 208 Gjerstad, Steven, 341 Gunther, Barrie, 76 Glaeser, Edward, 244, 246 Gurin, Gerald, 301, 331 Glaser, Jack, 190 Gurin, Patricia, 301, 331 Glaser, James, 124, 150 Guth,¨ Werner, 246, 402, 403 Glennerster, Rachel, 392 Gutig, Robert, 158 Glock, Charles, 104 Gutmann, Amy, 259, 260 Glunt, Erik, 325 Glynn, Adam, 515, 516, 517 Ha, Shang, 234, 510, 512, 514, 517 Glynn, Kevin, 191 Habermas, Jurgen,¨ 259 Name Index 537

Habyarimana, James, 6, 116, 348, 389, 391 Hermann, Charles, 73, 436, 437, 438 Hacker, Kenneth, 187 Hermann, Margaret, 73, 431, 435, 436, 438 Hacking, Ian, 106 Herrmann, Benedikt, 247, 248 Haddock, Geoffrey, 220 Herrmann, Richard, 440 Hafer, Catherine, 59, 60, 61, 259 Hetherington, Marc, 244 Hagan, Joe, 438 Hey, John, 65, 66 Hagendoorn, Look, 6 Hibbing, John, 244, 259, 261 Hahn, Kyu, 202 Higgins, Julian, 21 Haile, Daniel, 249 Highton, Benjamin, 313 Hais, Michael, 205 Hill, Jennifer, 468, 478, 511, 517 Hajnal, Zoltan, 192 Hill, Seth, 85 Hall, Crystal, 193 Hill, Timothy, 204 Hallam, Monica, 166 Himmelfarb, Samuel, 142 Halpern, Sydney, 8 Hirschﬁeld, Paul, 502 Hamilton, James, 201, 203 Ho, Shirley, 205 Hamilton, William, 340, 342 Hodges, Sarah, 157 Hamman, John, 344, 345–347 Hoekstra, Valerie, 189 Hammond, Kenneth, 419 Hoffman, Jeanne, 508 Hammond, Thomas, 6 Hogarth, Robin, 63 Hamner, W. Clay, 416 Holbert, R. Lance, 204 Han, Hahrie, 237 Holbrook, Allyson, 142, 144, 145, 147, 148 Haneman, W. Michael, 142, 144 Holbrook, Robert, 204 Haney, Craig, 277 Holland, Paul, 16, 23, 445 Hanna, Rema, 386 Holm, Hakan, 247 Hansen, Ben, 447, 448, 461, 466, 467, 470, 471, Holt, Charles, 403 476, 486 Hopmann, P. Terrance, 415, 416, 417, 418, 419, Hansen, Glenn, 204 421, 424, 425 Hansen, John, 117–121, 141, 228, 503 Horiuchi, Yasuka, 478 Hardin, Garrett, 340 Horowitz, Joel, 403 Hardin, Russell, 244, 245, 246, 339 Houlette, Missy Harinck, Fieke, 419 Houser, Daniel, 248, 251, 253, 374, 380 Harkins, Stephen, 325 Hovland, Carl, 76, 77, 215, 217, 452 Harris, Kevin, 192 Howell, William, 283, 453 Harris, Richard, 416, 417, 420, 422, 423, 424 Hsu, Ming, 29 Harrison, Glenn, 116, 348 Huang, Li-Ning, 149 Harsanyi, John, 343 Huber, Gregory, 145, 206, 317, 321, 514 Hartman, Jon, 192, 223 Huck, Steffen, 251 Hartz-Karp, Janette, 260 Huckfeldt, Robert, 201, 259, 274, 276, 487 Hasecke, Edward, 151, 189, 195 Huddy, Leonie, 190, 191, 291, 292, 310, 311, 312 Hastie, Reid, 157, 193, 195, 259, 277 Hudson, Valerie, 430 Hatchett, Shirley, 301 Huffacker, David, 408 Hauck, Robert, 8 Humphreys, Macartan, 6, 9, 116, 125, 126, Hauser, David, 205 127–129, 348, 385, 386, 387–388, 389, 391 Hayes, Danny, 192 Hurley, Norman, 150, 326 Healy, Alice, 433 Hurwitz, Jon, 105, 150, 311, 323, 328, 329, 501 Healy, Andrew, 130 Hutchings, Vincent, 150, 192, 317 Heckman, James, 9, 77, 451 Hyde, Susan, 15, 125 Hedges, Larry, 21 Heine, Steven, 54 Imai, Kosuke, 230, 464, 478, 483, 511, 512, 518 Heinrichs, Markus, 248 Imbens, Guido, 30, 31, 126, 127, 235, 448, 497, Heldman, Caroline, 21, 77, 451 498, 512, 513, 515 Henrich, Joseph, 6, 8, 54, 347, 349 Inglehart, Ronald, 248 Henry, P. J., 143, 307, 310, 311 Institute for Personality and Ability Testing, 433 Herek, Gregory, 325 Irmer, Cynthia, 421 Hermalin, Albert, 85 Irwin, Kyle, 249 538 Name Index

Isaac, R. Mark, 342, 343 Kaplan, Abraham, 527 Isenberg, Daniel, 436 Kaplan, Edward, 24, 116, 123, 133 Ito, Tiffany, 158 Kapteyn, Arie, 85 Iyengar, Shanto, 6, 8, 15, 45, 75, 76, 77, 78, 79, 80, Karau, S. J., 276 81, 120, 150, 190, 191, 192–193, 196, 201, 202, Kardes, Frank, 7, 156, 159, 163 205, 217, 219, 220, 224, 309, 311, 320, 322, Karlan, Dean, 66, 125, 234, 246, 248, 250, 390, 330, 451, 513 391 Karpowitz, Christopher, 260, 262–263, 266, 268 Jackman, Simon, 109 Katz, Elihu, 483 Jackson, James, 301 Katz, Lawrence, 283, 502 Jacobs, Lawrence, 196, 261 Kawachi, I., 273 Jacobson, Dan, 424 Kawakami, Kerry, 163 Jacobson, Gary, 118, 138 Kayali, Amin, 248 Jaeger, David, 514 Kearns, M., 276 James, Lawrence, 517, 518 Keating, Caroline, 192 James, William, 54, 149 Keefer, Philip, 250, 384 Jamison, Julian, 66, 393 Keele, Luke, 469, 478, 511, 512, 518 Janda, Kenneth, 428 Keeter, Scott, 190, 496 Janis, Irving, 437 Kegler, Elizabeth, 324, 329 Janssen, Marco, 340 Kellermann, Michael, 501 Jeliazkov, Ivan, 360 Kelley, Stanley, 194 Jennings, M. Kent, 274 Kelshaw, Todd, 259 Jeong, Gyung-Ho, 364, 365, 371 Kendrick, Timothy, 192 Jerit, Jennifer, 179, 180, 182 Kenney, Patrick, 78, 295, 296 Johannesson, Magnus, 248 Kenny, David, 20, 46, 509, 511, 517, 519 Johansson-Stenman, Olof, 246, 247, 250 Keohane, Robert, 4 John, Peter, 9, 124 Keppel, Geoffrey, 8 Johnson, David, 419 Kern, Holger, 24 Johnson, Devon, 323, 328, 329 Kern, Mary, 406, 407, 409 Johnson, Dominic, 39 Kern, Montague, 194, 195 Johnson, Eric, 188, 344 Keser, Claudia, 252 Johnson, George, 8 Key, Vladimir, 196 Johnson, Kevin, 225 Khemani, Stuti, 392 Johnson, Marcia, 208 Khwaja, Asim, 22 Johnson, Martin, 43 Kiers, Marius, 406 Johnson, Noel, 246 Kiesler, Charles, 30 Johnson, Paul, 487 Kiewiet de Jonge, Chad, 392 Jones, Jeffrey, 196 Kile, Charles, 6 Jowell, Roger, 262, 263, 277 Kim, Dukhong, 320 Judd, Charles, 436, 511, 517 Kim, Young Mie, 204 Junn, Jane, 496 Kim, Sung-youn, 168 Just, Marion, 194, 195, 203, 208 Kinder, Donald, 6, 15, 43, 73, 75, 106, 144, 145, Juster, Thomas, 85 150, 165, 190, 191, 195, 201, 202, 203, 205, 210, 216, 261, 307, 308, 309, 310, 320, 321, Kaarbo, Juliet, 436, 437, 438, 440 513, 527 Kagel, John, 5, 6, 58, 402, 404, 405, 406, 408 King, David, 223, 294, 295, 296, 486, 489–490, Kahn, Kim, 78, 293 517 Kahn, Robert, 85 King, Elizabeth, 453 Kahneman, Daniel, 29, 259, 435 King, Gary, 4, 464, 483, 511 Kaid, Lynda, 148, 221, 224, 225, 292, 296 King-Casas, Brooks, 248 Kaku, Michio, 209 Kirshner, Alexander, 17 Kalakanis, Lisa, 166 Kitayana, Shinobu, 80 Kalish, G. K., 92 Kline, Reuben, 252 Kam, Cindy, 42, 46, 51, 150, 189, 223, 496 Kling, Jeffrey, 283, 502, 505 Kameda, Tatsuya, 437, 438 Knack, Stephen, 130–131, 250 Name Index 539

Knight, Jack, 243 Laver, Michael, 399, 404 Kobrynowicz, Diane, 175 Lavine, Howard, 164, 166, 193, 195 Kocher, Martin, 247, 252 Lawless, Jennifer, 295, 296 Koford, Kenneth, 364 Lawrence, Adria, 17, 448 Kogan, Nathan, 436, 437 Lawrence, Regina, 296 Kohut, Andrew, 205, 262 Lazarsfeld, Paul, 166, 273, 274, 278, 370, 487, 496 Kollock, Peter, 252 Lazer, David, 263 Koop, Royce, 22 Ledyard, John, 66, 378 Kopelman, Shirli, 416 Lee, Daniel, 54–55, 451 Kopp, Raymond, 142, 144 Lee, Taeku, 261, 263 Korfmacher, William, 310 Leeper, Mark, 291 Korper, Susan, 419, 422, 424 Lehrer, Steven, 5, 6, 402, 406, 408 Kosfeld, Michael, 248 Leonardelli, G. L., 419 Kosoloapov, Michail, 348 Levati, M. Vittoria, 344 Koutsourais, Helen, 420 Leventhal, Tama, 502 Kowert, Paul, 435 Levi, Margaret, 244 Kozyreva, Pelina, 348 Levin, Henry, 453 Krambeck, Hans-Juergen, 252 Levine, Adam, 97 Krasno, Jonathan, 118, 138 Levine, David, 59, 378, 380 Kravitz, David, 276–278 Levine, Felice, 8 Kremer, Michael, 22, 273, 281, 282, 391, 453, 487 Levine, Peter, 260 Kressel, Kenneth, 420, 421, 425 Levitt, Steven, 7, 118, 138 Kroger,¨ Sabine, 247 Levy Paluck, Elizabeth, 386, 389, 391 Krosnick, Jon, 130, 142, 143, 144, 145, 147, 148, Levy, Dan, 282 150, 151, 158, 174, 188, 190, 237, 509, 513 Lewis, Steven, 418, 419, 421, 423, 425 Krueger, Alan, 117, 483, 497 Li, Rick, 84 Kruglanski, Arie, 45, 194 Li, Sherry, 63 Krupnikov, Yanna, 219, 220–221 Lichtenstein, Paul, 248 Kugler, Tamar, 252 Lickteig, Carl, 142 Kuhberger,¨ Anton, 7 Liebman, Jeffrey, 283, 502 Kuklinski, James, 3, 4, 7, 15, 29, 42, 43, 75, 77, Lien, Pei-te, 320 109, 144, 150, 179, 180, 182, 189, 190, 192, Lijphart, Arend, 4 231, 315, 326, 330, 354, 450, 528 Limongi Neto, Fernando, 384 Kurzban, Robert, 248 Lin, Xihong, 452 Kwon, Seungwoo, 421 Lindsay, James, 209 Lindsey, Samuel, 155, 165 Ladd, Helen, 453 Lindskold, Svenn, 420 LaFleur, S., 157 Linz, Gabriel, 317 Laibson, David, 244, 246 Lipsitz, Keena, 223 Lakin, Jason, 483, 486, 489–490 List, John, 7, 116, 348 LaLonde, Robert, 117, 518 Little, Roderick, 452 Lamm, Helmut, 436, 437 Livingstone, J. Leslie, 44 Landa, Dimitri, 59, 60, 61 Liyanarachchi, Gregory, 44, 45 Langlios, Judith, 166 Lizotte, Mary-Kate, 295 Lapinksi, John, 206, 219, 224, 225, 317, 321 Llamas, Hector, 483, 486, 489–490 Larimer, Christopher, 125, 280, 331, 447, 486, Lo, James, 85 504 Lockwood, Chandra, 508 Larson, Andrea, 166 Lodge, Milton, 5, 6, 146, 148, 150, 157, 158, 163, Lassen, David, 380 164, 165, 166, 188, 189, 193, 194, 195, 196, Latane, Bibb, 36 218, 513 Lau, Richard, 21, 42, 77, 156, 179, 182, 188, 189, Loewen, Peter, 22 193, 194, 195, 207, 214, 216, 217, 218, 220, Loftus, Elizabeth, 159 224, 249, 250, 451 Londregan, John, 374 Lauda, Dimitri, 259 Long, Qi, 452 Laude, Tiffany, 54 Lopez-Garcia,´ J. J., 509 540 Name Index

Loudon, Irvine, 131 McClone, Matthew, 175 Lovrich, Nicholas, 76 McClurg, Scott, 487 Lowell, A. Lawrence, 3 McCombs, Maxwell, 202 Ludwig, Jens, 502 McConahay, John, 307 Lupia, Arthur, 3, 4, 6, 29, 37, 42, 62, 63, 64, 75, McConnaughy, Corrine, 469, 478 90, 94, 95, 97, 143, 156, 171, 172, 173, 174, McConnell, Margaret, 124, 232, 484, 486, 487, 175, 177, 243, 354 489 Luskin, Robert, 262, 263, 277 McCormick, Joseph, 307 Lyall, Jason, 15 McCubbins, Mathew, 62, 63, 64, 90, 95, 172, 177, Lybrand, Steven, 191 243 Lynch, Michael, 360, 361, 362–363 McCullagh, Peter, 469 Lytle, Anne, 416 McDermott, Monika, 191 McDermott, Rose, 8, 28, 39, 42, 44–45, 53, 58, MacDonald, Paul, 44 435 Macedo, Stephen, 259 McGhee, Debbie, 156, 159 Mackie, Diane, 34 McGonagle, Katherine, 143 MacKinnon, David, 508 McGraw, Kathleen, 5, 146, 148, 150, 163, 189, MacKuen, Michael, 196 190, 193, 194, 195, 196 Madden, Mary, 205 McGrimmon, Tucker, 249 Maheswaran, Durairaj, 156 McGuire, William, 96, 210 Mahmud, Minhaj, 246, 247, 250 McKelvey, Richard, 15, 93, 94, 95, 97, 353, 355, Mahoney, Robert, 4 356, 359, 360, 362–363, 366, 372, 401 Maison, Dominika, 163 McKersie, Robert, 418 Makhijani, Mona, 166 McKinnon, David, 508 Malanchuk, Oksana, 190, 331 McNulty, John, 130 Malesky, Edmund, 9 McPhee, William, 166, 273, 278, 370, 487, 496 Malhotra, Neil, 130, 509 Meade, Jordan, 260 Mameli, Matteo, 68 Mechanic, David, 85 Manchanda, Puneet, 483 Medvec, Victoria, 406, 407, 409, 419 Manin, Bernard, 260 Meier, Stephen, 42 Mann, Christopher, 504 Meirowitz, Adam, 259, 364 Mannix, Elizabeth, 406 Melamed, Lawrence, 4 Mansbridge, Jane, 259, 260, 268, 269 Melendez,´ Carlos, 392 Manski, Charles, 450, 483 Mellor, Jennifer, 247 Maraden, Peter, 143 Mendelberg, Tali, 76, 165, 192, 206, 224, 259, Marcus, George, 303 260, 261, 262–263, 266, 316, 317, 321, 329 Markus, Gregory, 146, 190 Menning, Jesse, 97 Marmot, Michael, 85 Menzel, Herbert, 483 Marschall, Melissa, 331 Meredith, Mark, 130, 166 Martel Garcia, Fernando, 392 Merelman, Richard, 191 Martin, Barbara, 276–278 Merkle, Daniel, 262 Martin, Lanny, 399 Merlo, Antonio, 399, 400, 404, 405, 407 Martin, Linda, 85 Merton, Robert, 104, 274 Martinez, S., 386 Messick, David, 406 Martinsson, Peter, 246, 247, 250 Metcalf, Kelly, 192, 223 Masters, William, 386 Michelson, Melissa, 124, 231, 232, 234, 237, 327, Matland, Richard, 223, 294, 295, 296 484, 486 Matthews, Douglas, 220–221 Middleton, Joel, 236 Matzner, William, 248 Miettinen, Topi, 344 Mayhew, David, 196 Miguel, Edward, 32, 273, 281, 391, 487 McCabe, Kevin, 243, 245, 248 Milgram, Stanley, 35, 44, 68, 279, 504 McCafferty, Patrick, 166, 192 Milinski, Manfred, 252 McClain, Paula, 249 Miller, Arthur, 190, 244, 301, 331 McCleod, Douglas, 205 Miller, Cheryl, 307 McClintock, Charles, 425 Miller, Gary, 6, 360, 361, 362–363, 364, 365, 371 Name Index 541

Miller, George, 155 Naef, Michael, 247, 249, 250 Miller, Joanne, 130, 143, 150, 151, 237, 513 Nagler, Jonathan, 326 Miller, Melissa, 176, 177 Nair, Harikesh, 483 Miller, Nicholas, 359 Nall, Clayton, 483 Miller, Norman, 30 Nannestad, Peter, 244, 246 Miller, Richard, 146 Nash, John, 91, 92 Miller, Roy, 118, 120, 229, 231 National Research Council (NRC), 351 Miller, Warren, 53, 145, 190, 274, 496, 509 Nayga, Rodolfo, 54 Milnor, J. W., 92 Neblo, Michael, 9, 263 Milyo, Jeffrey, 247 Nee, Victor, 273 Mintz, Alex, 6, 54, 432, 434 Neely, James, 159 Mirabile, Robert, 277 Nehrig, E. D., 92 Mirer, Thad, 194 Neijens, Peter, 406 Mishler, William, 250 Neil, Clayton, 464 Mislin, Alexandra, 246 Nelson, Charles, 80 Mitchell, David, 438, 440 Nelson, Kjersten, 62–63, 67, 149, 150 Mitchell, Dona-Gene, 189, 194 Nelson, Thomas, 106, 149, 150, 208, 307, 310, Mitchell, Jason, 161 509, 513, 514 Mitchell, Robert, 142, 144 Neuberg, Steven, 191 Mitchell, Thomas, 419 Neuman, W. Russell, 203, 208 Mitofsky, Warren, 262 Neyman, Jerzy, 23, 460 Mo, Cecilia, 130 Nickerson, David, 15, 30, 121, 122–123, 125, 131, Moehler, Devra, 386, 388 132, 230, 231, 232, 233, 234, 277, 279, 281, Mohan, Paula, 191 282, 392, 449, 461, 482, 484, 486 Mondak, Jeffrey, 150, 176, 210 Nie, Norman, 301, 496 Monroe, Brian, 156 Niederle, Muriel, 402 Montague, P. Read, 248 Niemi, Richard, 274 Montfort, N., 276 Nier, Jason, Mook, Douglas, 44 Nimmo, Dan, 187 Moore, Don, 406 Nisbett, Richard, 36 Moore, Ryan, 483, 486, 489–490 Nitz, Michael, 191, 313–317 Morelli, Massimo, 404, 405 Nordenmark, Mikael, 273 Morgan, Stephen, 514 Norenzaya, Ara, 54 Morgenstern, Oscar, 91 Norretranders, Tor, 155 Morrell, Michael, 264 Nosek, Brian, 161, 193 Morris, Jonathan, 151 Nutt, Steven, 205 Morton, Rebecca, 5, 6, 8, 18, 44, 58, 63, 64, 65, 66, Nuttin, J., 425 373, 376, 378, 379, 380, 381, 401, 404, 406, 408 Nystrom, Leigh, 248 Moscovici, Serge, 437 Moskowitz, David, 191, 315 O’Keefe, Daniel, 96 Moskowitz, Gordon, 193 Ochs, Jack, 403 Mottola, Gary, Ockenfels, Axel, 402, 404 Mouton, Jane, 417, 419 Odell, John, 421 Muhlberger, Peter, 263, 265–266 Okuno-Fujiwara, Masahiro, 403 Mui, Vai-Lam, 252 Olekalns, Mara, 424 Mullainathan, Sendhil, 17, 386 Olken, Benjamin, 125, 385, 391 Muller, Laurent, 342 Olson, Mancur, 339, 340, 341 Muney, Barbara, 419 Olson, Michael, 163, 331 Munshi, Kaivan, 487 Oosterbeck, Hessel, 381 Munuera, Jose, 208 Oppenheimer, Joe, 6, 94, 95, 266, 344 Mutz, Diana, 149, 207, 259, 274 Orbell, John, 341, 453 Myers, Daniel, 262, 437 Ordeshook, Peter, 93, 95, 353, 355, 356, 359, 366, Myers, David, 436 372, 373 Myerson, Robert, 375 Organ, Dennis, 417, 419 Myerson, Roger, 375, 376, 378 Orne, Martin, 77, 78 542 Name Index

Orr, Shannon, 176, 177 Plessner, Henning, 158 Osgood, Charles, 416 Plott, Charles, 44, 91–94, 354, 355, 356–357, 364, Osorio, Javier, 392 399 Ostrom, Elinor, 5, 15, 340, 341, 345, 346–347 Polichak, Jamie, 164 Ottati, Victor, 189, 190, 192, 315 Polletta, Francesca, 260 Owen, Andrew, 221, 225 Pomper, Gerald, 214, 217 Oxley, Zoe, 150, 208, 513, 514 Poole, Keith, 361 Popkin, Samuel, 171 Packel, Edward, 93 Posner, Daniel, 6, 9, 116, 348, 389, 391, 528 Page, Benjamin, 197 Posner, Michael, 159 Palfrey, Thomas, 15, 59, 73, 90, 95, 97, 98, 118, Posner, Richard, 129 138, 261, 374, 378–380, 381, 402, 527 Postmes, Tom, 205, 406 Palmer, Carl, 496 Poteete, Amy, 340 Paluck, Elizabeth, 125 Powell, Martha, 156, 159, 163 Panagopoulos, Costas, 124, 125, 236, 237, 463, Prasnikar, Vesna, 403 468, 483, 504 Pratto, Felicia, 159, 307 Pancer, Mark, 221 Preacher, Kristopher, 161 Pande, Rohini, 390 Prensky, Marc, 205 Park, Bernadette, 157, 193 Presser, Stanley, 102, 142, 144 Patterson, Samuel, 273 Preston, Thomas, 438 Patterson, Thomas, 201 Price, Jennifer, 259 Pavitt, Charles, 344 Price, Vincent, 76, 149, 207, 208, 263 Payne, John, 277 Prior, Markus, 6, 62, 63, 64, 76, 174, 175, 204 Payne, Keith, 156, 159, 161, 163 Pruitt, Dean, 418, 419, 421, 423, 425 Pearl, Judea, 514 Przeworksi, Adam, 384 Pearse, Hilary, 260 Putnam, Robert, 243, 248, 251 Pearson, Kathryn, 295 Putterman, Louis, 252 Pefﬂey, Mark, 105, 150, 311, 323, 328, 329, 501 Penaranda-Ortega,˜ M., 509 Quartz, Steven, 248 Pennington, Nancy, 195, 259 Quillian, Ross, 159 Penrod, Steven, 259 Quinones-Vidal,˜ E., 508, 509 Pentland, Alex, 408 Quirk, Paul, 7, 43, 109, 179, 180, 182, 231, 330, Peri, Pierangelo, 313 450 Pesendorfer, Wolfgang, 97, 379 Peters, Mark, 145, 150 Rabin, Matthew, 341 Peterson, Paul, 283, 453 Rahn, Wendy, 144, 145, 158, 188, 192, 194, 248 Peterson, Robert, 54 Raiffa, Howard, 406, 418 Petrie, Ragan, 249 Ramberg, Bennett, 422, 423, 424 Petrocik, John, 192, 196 Ramirez, Ricardo, 327 Petty, Richard, 96, 97, 142, 149, 150, 207, 325, Randall, David, 192 331, 516 Randazzo, Kirk, 377 Pew Research Center for the People & the Press, Randolph, Lillian, 422 137 Rapoport, Anatol, 91, 418 Peytcheva, Emilia, 54 Rapoport, Ronald, 192, 223 Phelps, Elizabeth, 29, 248 Ravallion, Martin, 386 Philpot, Tasha, 295 Ravishankar, Nirmala, 483, 486, 489–490 Piankov, Nikita, 250 Ray, Paul, 414 Piazza, Thomas, 15, 104, 105, 109, 110, 111, 307, Read, Stephen, 156 313 Redd, Steven, 54, 438–439 Pierce, John, 76 Redlawsk, David, 42, 156, 157, 179, 182, 188, 189, Piereson, James, 303 193, 194, 195, 207, 218, 224 Pietromonaco, Paual, 164 Reeves, Byron, 149, 207 Pilisuk, Marc, 416 Reeves, Keith, 316 Pinney, Neil, 195 Reid, Nancy, 468, 472 Pischke, Jorn-Steffen,¨ 515 Ren, Ting, 252 Name Index 543

Reyna, Christine, 310 Ruel, Marie, 483 Rich, Robert, 179, 180, 182 Ruiz, Salavador, 208 Richardson, Liz, 9 Russell, Peter, 105, 109 Ridout, Travis, 225, 249 Russo, Edward, 188 Riecken, Henry, 525 Rust, Mary, Rietz, Thomas, 375, 376, 378 Rustichini, Aldo, 63 Riggle, Ellen, 189, 190, 192, 315 Ryan, Lee, 248 Riker, William, 6, 73, 92, 112, 353, 355–356 Ryfe, David, 262 Rilling, James, 248 Rimm, E. B., 273 Sacerdote, Bruce, 282 Rind, Bruce, 232 Sadrieh, Abdolkarim, 249 Rips, Lance, 76 Sakhel, Khaled, 205 Rivers, Douglas, 85, 196 Sally, David, 344 Roberson, Terry, 292, 296 Salovey, Peter, 190 Robins, James, 516 Samphantharak, Krislert, 9 Robinson, Jonathan, 487 Sanbonmatsu, David, 156, 159, 163 Robinson, Victor, 425 Sanbonmatsu, Kira, 295 Rockenbach, Bettina, 251, 253 Sanbonmatsu, Lisa, 502, 505 Roddy, Brian, 221 Sandbu, Martin, 386 Roderick-Kiewiet, D., 145 Sanders, Jimy, 273 Roethlisberger, Fritz, 36 Sanders, Lynn, 165, 191, 259, 307, 308, 321 Rogers, Brian, 483 Sandroni, Alvaro, 379 Rogers, Reuel, 331 Sanfey, Alan, 248 Rogers, Robyn, 292 Sapin, Burton, 430 Rogers, Todd, 232 Sapiro, Virginia, 289, 290, 292 Romer, Daniel, 149 Sartori, Anne, 228 Romer, Thomas, 374 Satyanath, Shanker, 32 Rose, Melody, 296 Satz, Debra, 340 Rose, Richard, 250 Savage, Robert, 187 Rosenbaum, Paul, 279, 470, 474, 477, 478, 511, Savin, N. E., 403 517 Sawyer, Jack, 422 Rosenberg, Shawn, 166, 192, 260, 262 Schachter, S., 278 Rosenstone, Steven, 117–121, 141, 228, 496, 503 Schaefer, Samantha, 208 Rosenthal, Howard, 361, 378–380 Schafer, Mark, 431 Rosenthal, Robert, 98 Schaffner, Brian, 207, 209 Rosenwasser, Shirley, 292 Schechter, Laura, 66, 250 Rosenzweig, Mark, 500 Scheinkman, Jose, 244, 246 Roskos-Ewoldsen, Beverly, 81 Schellhammer, Melanie, 29 Roskos-Ewoldsen, David, 81 Schelling, Thomas, 340 Ross, H. L., 525 Schildkraut, Deborah, 261 Ross, Lee, 434 Schkade, David, 259, 277 Ross, William, 420 Schlozman, Kay, 120, 228 Rossenwasser, Shirley, 291 Schmidt, Klaus, 341, 404 Roth, Alvin, 5, 29, 44, 58, 90, 400, 403 Schmitt, Neal, 54 Rothstein, Bo, 251, 341, 350 Schmittberger, Rolf, 402, 403 Rothstein, Hannah, 21 Schneider, Carl, 130 Rouhana, Nadim, 419 Schneider, Monica, 192, 296 Rouse, Cecilia, 455 Schnyder, Ulrich, 29 Rovner, Ivy, 216, 220, 249, 250 Schochet, Peter, 462, 474 Rubenstein, Adam, 166 Schoﬁeld, Norman, 94, 399, 404 Rubin, Donald, 23, 30, 31, 121, 126, 127, 235, Schooler, Jonathan, 156, 165 448, 460, 478, 484, 497, 498, 511, 512, 513, 517 Schooler, Tonya, 155, 165 Rubin, Jeffrey, 415 Schotter, Andrew, 252 Rudd, Paul, 142, 144 Schram, Arthur, 379 Rudman, Laura, 163 Schramm, Susan, 290 544 Name Index

Schreiber, Darren, 8 Simmons, Randy, 341 Schuman, Howard, 102 Simon, Adam, 6, 77, 79, 217, 219, 220, 263, 451 Schuman, Phillip, 54 Simon, Herbert, 44, 430, 431 Schupp, Jurgen,¨ 247, 249, 250 Simpson, Brent, 249 Schwartz, Jordon, 156, 159 Sinclair, Betsy, 483, 484, 486, 487, 489 Schwartz, Norbert, 189, 190, 192, 315 Singer, Eleanor, 8 Schwartz, Richard, 107, 525 Singer, J. David, 414 Schwarze, Bernd, 402, 403 Skolnick, Paul, 416 Schweider, David, 180 Slade, Allen, 54 Schweinhart, L. J., 504 Sleeth-Keeper, David, 194 Schwieren, Christine, 158 Slonim, Robert, 246 Scioli, Frank, 74, 75 Sloof, Randolph, 381 Scott, John, 275 Slothuus, Rune, 43 Seale, Jana, 291 Smith, Dennis, 118, 120, 229 Searing, David, 261 Smith, Eliot, 34 Sears, David, 42, 45, 51, 53, 82, 143, 223, 307, Smith, James, 85 311, 314, 329 Smith, Jeffrey, 77 Sechrest, Lee, 107, 525 Smith, Jennifer, 240 Sefton, Martin, 342, 344, 346–347, 403 Smith, Natalie, 349 Seher, Rachel, 249 Smith, Patten, 176 Sekhon, Jasjeet, 460 Smith, Peter, 276 Selfton, Martin, 403 Smith, Philip, 424 Sellers, Patrick, 207, 209 Smith, Tom, 143 Selten, Reinhard, 343 Smith,V.Kerry,142, 144 Semmel, Andrew, 436 Smith, Vernon, 7, 248, 340, 371 Sened, Itati, 360, 361, 362–363, 365, 371 Smoot, Monica, 166 Sergenti, Ernest, 32 Sniderman, Paul, 6, 15, 43, 54, 105, 106, 107, Sernau, Scott, 273 109–112, 209, 307–308, 313, 330 Seying, Rasmy, 259 Snyder, Charles, 159 Shachar, Ron, 231, 496, 501, 503 Snyder, James, 118 Shadish, William, 18–19, 28, 42, 46, 48, 452 Snyder, Richard, 430 Shankar, Anisha, 344 Sokol-Hessner, Peter, 29 Shanks, J. Merrill, 496 Soldo, Beth, 85 Shapiro, Ian, 17 Sommerfeld, Ralf, 252 Shapiro, Robert, 196 Sondheimer, Rachel, 496, 500, 504 Shapley, L. S., Song, Fei, 252 Shaw, Daron, 43, 218, 225, 235, 236, 238 Sonnemans, Joep, 379 Shaw, Donald, 202 Sonner, Brenda, 54 Sheehan, Reginald, 377 Sopher, Barry, 252 Sheets, Virgil, 508 Soutter, Christine, 244, 246 Shepsle, Kenneth, 356, 360, 386, 501 Sovey, Allison, 32, 496, 499, 503 Sherman, Steven, 143, 232 Spears, Russell, 205 Shevell, Stephen, 76 Spencer, Steven, 508, 518 Shewfelt, Stephen, 277 Spira, Joan, 325, 326, 332 Shingles, Richard, 301, 331 Sporndli, Markus, 260 Shultz, Cindy, 221 Sprague, John, 259, 274, 487 Shupp, Robert, 344, 346–347 Stampfer, M. J., 273 Sicilia, Maria, 208 Stanley, Julian, 30, 144–147, 148, 261, 262, 268, Sidanius, Jim, 307, 329 525, 526 Sides, John, 223 Steed, Brian, 346 Siegel, Sidney, 415 Steekamp, Jan-Benedict, 142–143 Sigelman, Carol, 191, 192, 313–317 Steele, Claude, 175 Sigelman, Lee, 21, 77, 191, 192, 216, 220, 291, Steenbergen, Marco, 5, 6, 157, 158, 163, 188, 193, 313–317, 451 194, 218, 260 Silver-Pickens, Kayla, 292 Stehlik-Barry, Kenneth, 496 Name Index 545

Stein, Janice, 437, 438 Taylor, Michael, 339 Steinberg, Richard, 342 Taylor, Paul, 408 Steinel, Douglas, 73 Tellez-Rojo, Martha, 483, 486, 489–490 Steinel, Wolfgang, 419 Tenn, Steven, 496 Steiner, Jurg, 260 Terkildsen, Nayda, 190, 191, 291, 292, 315, 326, Steiner, Peter, 452 332 Stempel, John, 438 Tetlock, Philip, 54, 105, 109, 307, 440 Stenner, Karen, 190 Tewksbury, David, 204, 207, 208 Stevens, Charles, 54 Theiss-Morse, Elizabeth, 244, 259, 261 Stevenson, Randolph, 399 Theriault, Sean, 43, 106, 209, 330 Stewart, Brandon, 156, 159, 161, 163 Thomas, Sally, 408 Stewart, Charles, 130 Thompson, Dennis, 258, 259, 260 Stimson, James, 196 Thompson, L. L., 406 Stock, James, 503 Thoni, Christian, 247 Stoker, Laura, 108, 244, 491 Thorson, Esther, 210 Stokes, Donald, 53, 145, 190, 274, 496, 509 Thurstone, Louis, 141 Stokes, Susan, 17, 384, 387 Tingley, Dustin, 39, 512, 518 Stolle, Dietlind, 243 Tirole, Jean, 340 Stouffer, Samuel, 303 Titmuss, Richard, 64 Stranlund, John, 349 Titus, Linda, 276 Stratman, Thomas, 374, 380 Tobin, James, 451 Strauss, Aaron, 124, 125, 230 Todd, Frederick, 419 Streb, Matthew, 295, 296 Todorov, Alexandar, 193 Stroh, Patrick, 5, 6, 146, 148, 150, 189, 191, 193, Tomlin, Damon, 248 195, 315 Tomz, Michael, 15, 197, 435, 440 Stromberg, David, 388 Tooby, John, 409 Stroud, Laura, 190 Tortosa-Gil, F., 509 Sturgis, Patrick, 176 Transue, John, 54–55, 141, 150, 302, 451 Suelze, Marijane, 104 Traubes, Gary, 134 Suetens, Sigrid, 344 Traugott, Michael, 143 Sugimori, Shinkichi, 437, 438 Treyer, Valerie, 29 Suhay, Elizabeth, 150, 151, 312, 509 Trivedi, Neema, 327 Sulkin, Tracy, 6, 263 Trope, Yaacov, 155 Sullivan, John, 303 Trost, Christine, 223 Summers, David, 417 Trouard, Theodore, 248 Sundelius, Bengt, 437, 438 Truman, David, 339 Sunder, Shyam, 51, 55 Tucker, Amanda, 310 Sunstein, Cass, 259, 277 Tucker, Joshua, 514 Sunstein, R., 277 Tucker, Peter, 273 Suri, S., 276 Turner, John, 60, 300 Sutter, Matthias, 247, 252, 344 Tversky, Amos, 29, 435 Svenson, Jakob, 392 Tyran, Jean-Robert, 380 Swaab, Roderick, 406, 407, 408, 409 Swanson, Jane, 163 Udry, Christopher, 483 Sylvan, Donald, 431 Uhlaner, Carole, 379 Sztompka, Piotr, 243, 244–245 Ury, William, 418 Uslaner, Eric, 244, 247 Taber, Charles, 157, 158, 163, 164, 165, 166, 513 Tajfel, Henri, 60, 299 Valadez, Joseph, 4, 422 Takahahsi, Lois, 250 Valentino, Nicholas, 79, 80, 150, 151, 192, 219, Taniguchi, Naoko, 478 220, 312, 317, 451, 509 Taris, Toon, 146 Valley, Kathleen, 406 Tate, Henri, 302 Van Beest, Ilja, 408, 419 Tate, Katherine, 320 Van de Kragt, Alphons, 341 Taubes, Gary, 134, 138 Van de Kulien, Gijs, 381 546 Name Index

Van der Heijden, Eline, 344 Wasserman, Stanley, 275 Van Dijk, Eric, 408, 419 Watabe, Motoki, 250 Van Houweling, Robert, 197 Watson, David, 44 Van Kleef, Gerben, 408, 419 Watson, Mark, 503 Van Soest, Daan, 344 Wattenberg, Martin, 77, 190 Van Swol, Lyn, 252 Watts, Candis, 249 Vargas, Manett, 483, 486, 489–490 Webb, Eugene, 107, 525 Vavreck, Lynn, 76, 85, 123, 203, 229, 230–232, Weber, Christopher, 205 482, 483, 486 Weber, L. M., 263, 265–266 Vayanos, Dimitri, 483 Weber, Robert, 375, 376, 378 Vedlitz, Arnold, 54 Weber, Roberto, 344, 345–347 Verba, Sidney, 4, 228, 250, 251, 301, 370, Wegener, Duane, 331, 516 496 Weikart, David, 504 Verbon, Harrie, 249 Weingart, Laurie, 421 Verhulst, Brad, 165, 166 Weingast, Barry, 360 Vertzberger, Yaacov, 437 Weinstein, Jeremy, 6, 9, 116, 125, 126, 127–129, Vesterlung, Lise, 342 348, 385, 386, 387–388, 389, 391 Vicente, Pedro, 386, 394 Weiser, Phil, 259 Viscusi, W. Kip, 277 Welch, Susan, 291 Vishak, John, 204 West, Stephen, 508 Visser, Penny, 145, 147, 148, 277, 440 Wheeler, S. Christian, 130, 166 Volden, Craig, 404 White, Ismail, 150, 208, 317, 321, 324, 329, 469, Von Neumann, John, 91 478 Voss, James, 431 White, Paul, 207, 325 Vyrastekova, Jana, 344 White, Peter, 203 Whitler, Tommy, 332 Wagner, Gert, 247, 249, 250 Whitney, Simon, 130 Wagner, Lynn, 420, 425 Whittler, Tommy, 325, 326, 332 Walcott, Charles, 416, 417, 418, 419, 424 Wickens, Thomas, 8 Walker, James, 5, 15, 341, 342, 343, 344, 345, Wilkerson, John, 354 346–347 Wilking, Jennifer, 42, 51, 189, 223 Walker, Stephen, 437, 438 Wilkins, Marilyn, 419 Walker, T., 37 Willet, W. C., 273 Walkosz, Barbara, 191, 313–317 Williams, K. D., 276 Wall, James, 191 Williams, Kenneth, 6, 8, 18, 44, 58, 63, 64, 65, 66, Wallace, Bjorn, 248 372, 373, 374, 376, 377, 381 Wallace, Michael, 437 Williams, Linda, 191 Wallace, Robert, 85 Willis, Cleve, 349 Wallach, Michael, 437 Willis, Robert, 85 Walsh, Katherine, 292 Wilson, Richard, 355, 356, 357, 358, 362, Waltenburg, Eric, 324, 329 366 Walters, Pamela, 420 Wilson, Rick, 246, 247, 249, 250, 348 Walton, Eugene, 249, 418 Wilson, Timothy, 44, 155, 156, 157, 165 Walton, Hanes, 295, 307 Winden, Frans, 379 Walton, Richard, 415 Winer, Mark, 93, 359 Wang, Xiao, 325 Winograd, Morely, 205 Wantchekon, Leonard, 6, 15, 125, 387, 390, 391, Winship, Christopher, 514 392–393 Winter, Nicholas, 295, 321, 528 Ward, Andrew, 434 Wise, David, 85 Ward, Christine, Wise, Kevin, 208 Warner, Stanley, 142, 143 Wittenbrink, Bernd, 155 Warren, Mark, 259, 260 Wittman, Donald, 373 Warwick, Paul, 404 Wolf, Patrick, 283, 453 Washington, Ebonya, 17, 124, 514 Wolfe, Katherine, 402 Wason, P. C., 60, 409 Wolﬁnger, Raymond, 496 Name Index 547

Wolosin, Robert, 143 Young, Oran, 344 Wolpin, Kenneth, 500 Yukl, Gary, 416 Wong, Janelle, 327 Wood, Elizabeth, 17, 391 Zajonc, Robert, 79, 81, 158 Wood, Wendy, 296, 437 Zak, Paul, 248, 250 Woodall, Gina, 295 Zaller, John, 54, 76, 106, 157, 193, 195, 203, 204, Wooldridge, Jeffrey, 503 329 Woon, Jonathan, 344, 345–347 Zamir, Shmuel, 403 Worchel, Stephen, 150, 203 Zanna, Mark, 220, 508, 513, 518 Wright, David, 146 Zartman, I. William, 418 Wright, Oliver, 79 Zavoina, William, 92 Wu, Ximing, 54 Zebrowitz, Leslie, 192 Wyer, Robert, 189, 190, 192, 315 Zechmeister, Elizabeth, 42, 51, 189, 223 Zechmeister, Kathleen, 424 Yamada, Kyohei, 124 Zeckhauser, Richard, 247, 248 Yamagishi, Toshio, 250 Zehnder, Christian, 250 Yamamoto, Teppei, 511, 512, 518 Zeller, John, 85 Yee, Nick, 8, 80, 192 Zeng, Langche, 511, 517 Yi, Zeng, 85 Zharinova, Natasha, 97 Young, Barbara, 232 Zimbardo, Phillip, 44, 277 Young, Iris, 259 Zuckerman, Alan, 274 Young, Margaret, 143 Zwiebel, Jeffrey, 483 Subject Index

abstention, 370, 376, 378, 380 study of, 28, 29 advertising, 31, 125.Seealsomedia toward candidates, 204 attitude impact, 145 violent TV, 202 black sheep effect, 219, 220 AIDS, 325 campaign, 203, 214 American National Election Studies (ANES), 51, competing messages, 221–222 105, 120, 133, 173, 188, 217, 229, 243, 244, duration effects, 218, 224 308 experimental designs, 222–224 American Political Science Association (APSA), 3, negative, 76, 77, 78, 450–451 4 negative advertising, vote choice, candidate American Psychological Association (APA), 502, evaluation, 219, 220–221 505 observational vs. experimental studies, The American Voter, 190 216–217 AMP.SeeAffect Misattribution Paradigm self-reports, 76–77 An Inconvenient Truth, 204 survey-participant recall errors, 76 ANES.SeeAmerican National Election Studies television, 37 APA.SeeAmerican Psychological Association voting likelihood, negative advertising, APSA.SeeAmerican Political Science Association 218–220 Arrovian social choice theory, 353 affect artifactual ﬁeld experiment, 116 affect-driven dual process models, 155 Asian Americans, 326, 327–328 AMP, 156, 161 Asian Flu Experiment, 108 encoding vs. retrieval effects, 157–158 associative network model, cognition, 158 Affect Misattribution Paradigm (AMP), 156, 161 ATE.Seeaverage treatment effect African Americans, 324, 326, 329 ATT.Seeaverage treatment effect for those agendas, forward- vs. backward-moving, 356–358 treated agenda-setting, 202, 204, 205–207.Seealso attitude(s) priming effects campaign advertising effect, 145 aggression, anger, 204 change, 142, 147–148, 323–324 genetic basis, 39 crystallization, 45, 48

548 Subject Index 549

defined, 141–142 face morphing study, 80 direction, extremity, 142 foreign policy inexperience, 432 experiments, attitude measurement, 142–144 narrow margin victory, 236 explicit, 142, 156 9/11 response, 434 expressive behaviors, 143 framing conditions, 323–324 CACE.Seecomplier average causal effect implicit, 142, 156, 158–164 California, 78, 85, 327 implicit vs. explicit, 321, 330 campaign effects, 224, 229 preferences and, 142–143 campaign spending, 118–119, 121 self-report questions, 142 candidate divergence models, 373–375 unconscious cognition and, 155 candidate policy preferences, 373–374 attrition candidates downstream effects, 504 cognitive process models, 193–195 internal validity, 19, 30–31 competing messages, 221–222 selective nonresponse, 146 evaluation, 187–188 automaticity, online evaluations, 158 evaluation, vote choice, 194, 219, 220–221 average treatment effect (ATE), 17, 23, 128–129, home style, 195–196 234, 445 impressions, 187 canvassing, 234 impressions, content, 190–193 defined, 23, 445 issue ambiguity, 196–197 entire population, 128–129 issue ownership, 192, 196 random assignment, 17 name recognition, 124, 125 average treatment effect for those treated (ATT), negative advertising, evaluation, vote choice, 23, 122 219, 220–221 strategic behaviors, 195–197 bargaining process analysis (BPA), 416 traits, 190 Baron-Kenney Mediation Model, 509–511 voting likelihood, negative advertising, behavioral research 218–220 beginnings, 73 canvassing.SeealsoGet-Out-The-Vote economics, 98 difficulties, 233–234 investment game, trust, 243–244 influence on turnout, 32, 131–132 racial identity, 300–302 interpersonal influence, 125 Benin, 385, 392–393, 394 social networks, 229, 231 Berelson, Bernard, 273 types, effects, 229–230 bias capital punishment, 323–324 background conditions and, 130 causation external vs. internal validity, 27 deliberation and, 260, 261–262 gender, 290–292 experimentation, 75–78, 527–528 mediation analysis, 508 external validity, 41, 42, 44 observational studies, 121 fundamental problem, 15–16 online studies, 83 internal validity, 28, 30 publication bias, 21, 133–134 Rubin Causal model and, 24 sampling, 75, 82–83 observational studies, 188 selection bias, 36, 83, 121, 122–123, 453–455 random assignment and, 3, 16–17 subject knowledge of hypothesis, 33 social networks, 273 subject skepticism, 28 trust, social exchange and, 253 treatment effect estimates, 46, 123–124 Center for Advanced Study in the Behavioral treatment protocol and, 34 Sciences, 527 black sheep effect, 219, 220 centipede game, 98 Blair, Tony, 434 cheap talk, 343 BPA.Seebargaining process analysis Civic and Political Health of the Nation Dataset, Bradley, Tom, 314 51 Brazil, 388 Civil Rights Act of 1964, 365 Bush, George H. W., 165, 432 Clean Water Action, 237 Bush, George W. Clinton, Bill, 221–222, 238, 432 550 Subject Index

Clinton, Hillary, 144, 188, 296 common-pool resources (CPRs), 339, 340, 344, CNN, 84 345–347 coalition experiments, 353, 354, 355, 356–357, competitive solution concept, game theory, 93 365–366, 399–400.Seealsogame theory compliance, compliers, noncompliers, 24–25, 54, alternative bargaining protocols, 404–406 126–129 Baron-Ferejohn model, 99, 400–401 complier average causal effect (CACE), 128, 499 Baron-Ferejohn model, testing, 401–404 computer-assisted interviewing, 103 behavior assumptions, 92–93 question sequencing, 103–104 coalition formation, 400, 404, 405, 408–409 randomized experiments and, 104 comparative statics, 400, 401, 403 COMROD.SeeCooperative Measures to Reduce context-rich experiments, 406–410 the Depletion of the Ozone Layer continuation value, 400, 402–403 Condorcet loser, winner, 375–377 cooperative, noncooperative models, 399–400 confederates, 36 dictator game, 402–403 confounds, confounding economics vs. social psychology, 409 defined, 188 experimental economics, 402, 408 observational research vs. experiments, 317 game theory, 92–93, 94 prejudice, candidate choice, 313 laboratory experiments, 401, 404 symbolic racism, 311 minimal winning, 93 constituency opinion, legislative behavior, 125 ultimatum game, 402–403 control group, 16 Wason task/test, 409 convenience sample, 7 cognitive process models, candidate evaluation, conventional lab experiment, 116 193–195 cooperative game theory, 91–94.Seealsogame memory-based, 193, 194–195 theory online, 193, 194–195 Cooperative Measures to Reduce the Depletion of coin tosses, 178 the Ozone Layer (COMROD), 422 collective behavior, 93 coordination, three-candidate elections, 375–376 collective-action theory core outcomes, game theory, 92, 353, bargaining models, 400–401, 402 355–356 baseline CPR experiments, 344, 346–347 corruption, democracy and development, baseline PG experiments, 342–345 388 cheap talk, 343 costs, cost-benefits, 7, 74–75 collective-action problems, 339 counter-argument technique, 109 communication, 342–344 covariates, covariate balance CPRs, 339, 340, 344, 345–347 common uses, potential abuses, 460–461 ethnic diversity, PC provision mechanisms, defined, 24, 460 348 distribution, samples, 45 field experiments, 347–349 experimental use, 460 MPCR, 342 observational work, 122 observational studies, 341 precision enhancement, 460–461 PD, 340 random imbalance adjustments, 461 PG, 339, 340, 341–342 Rubin causal model, 121 political leadership, 344–345 statistics use, 18 sanctioning, 346–347 student samples, 51 social norms, cultural variability, CPRs, 347, subgroup analysis, 461 349 CPRs.Seecommon-pool resources Trust Game experiments, 341 crime, racial coverage, 78–79 Ultimatum Game, 341 crime script, 78–79 VCM, 340, 342 cross-cultural experiments voting, 346 experimental design, 90 “College Sophomores in the Laboratory” (Sears), social desirability response bias, 142 42 trust, 250–251 Color-Blind Experiment, 112, 308 web-based, 85–86 Common Knowledge (Neuman, Just, Crigler), 201, cross-cultural trust, 250–251 203 crowding out, monetary incentives, 64–65 Subject Index 551 data generating process, 46–47, 50 Uganda, 116, 125, 386, 389, 391 deception, 65–67, 68 Democrat Party, 144 deliberation, 59–60 Department for International Development causal inference, experimental control, 260, (DfID), UK, 386 261–262 Deukmejiana, George, 314 consensus, 260 DfID.SeeDepartment for International decision making, 259–260 Development, UK decision-making process, 260 discrimination vs. prejudice, 103–104 definitions, 259 Dole, Robert deliberative polls, 260, 262–263, 277 don’t know response, 144, 175–177 democratic systems and, 260 downstream effects, 7, 31, 38 effects categories, 259–260 always-takers, 497 experiment examples, 262–267 analysis in practice, 500–502 experiments’ role, 258, 260–262 attrition, 504 external validity, 268 CASE, 499 face-to-face treatments, 264–265 challenges, 502–505 framing, 263, 265–266 compliers, 497 grand treatment, 263, 266 defiers, 497 Healthcare Dialog project, 263 ethics, 495, 504–505 mediating variables, 258, 260 High/Scope Perry Experiment, 500–501 methodological issues, 266–267 internal validity, 504 online deliberation, 262, 263–264 LATE, 496–500 quasi-experiments, 259, 261, 262–263 methodological challenges, 502–503 random assignment, 261, 267 mobilization, 496, 501 social networks, 277 MTO program, 502 deliberative polls, 260, 262–263, 277 natural experiments, 501 democracy and development, 384 never-takers, 497 Benin, 385, 392–393, 394 practical challenges, 503–504 Brazil, 388 randomized experiments, 495–496, 501 clientalism, 387 RCM, 496–500 corruption, 388 voter mobilization, 501 DfID, 386 drop-in samples, 83–84 ethics, election experiments, 393–394 dual process theories, information processing, external validity, 391–393 331–332 field experiments, collaboration with NGOs, dual samples, 54 385–387 DuBois, W. E. B., 306 field experiments, collaboration with Dukakis, Michael, 76, 165 politicians, 387 duration effects, 218, 224 future directions, 394 Dynamic Process Tracing Methodology, 189, 194 gender quotas, 387–388 Dyson, James, 74 India, 386, 390, 392 Indonesia, 391 economic journals, 65 internal validity, 389–391 economics vs. psychology experiments, 29–30, 58 International Finance Corporation, 386 deception in experimental economics, 65–67 International Rescue Committee, 386 monetary incentives, 61–65 intervention, treatment, 390 preferences control, 62–63 lab, lab-in-the-field experiments, 388–389 problems, 64–65 Liberia, 386, 389, 391 rewarding accuracy/reducing noise, 62 Mexico, 386, 388, 391 scale, 63–64 natural experiments, 386, 387–388 social preferences measurement, 62, 63 Nicaragua, 392 stylization, 59–61 Nigeria, 386 subject pool, 63, 66 research protocol compliance, 388, 390 use of deception, 65–68 Rwanda, 125, 388, 389, 390, 391 effect size, 21 Sarathi Development Foundation, 390 Eighteen-Nation Disarmament Conference, 416 552 Subject Index

Eisenhower, Dwight, 432 Pre-test–Post-test Multiple Experimental Eldersveld GOTV experiment, 4 Condition Design, 147, 148 Election Day institutions, election administration, pseudoexperimental, 32 125 realism, competition, over-time designs, encoding vs. retrieval effects, affect, 157–158 330–331 encouragement designs, 17 Repeated Cross-Section Design, 145 endogenous agendas, 363–364 survey experiments, 106–108, 109 engagement, subject, 28 Twin Studies, 251–252 equilibrium, 60, 62–63.SeealsoQuantal Response weaknesses, 148–149 Equilibrium within-, between-subjects design, 17–18, 30, errors, Type I and II, 209 372 ESP. See Experimental Study of Politics The Experimental Journal of Politics, 4 “Ethical Principles of Psychologists and Code of experimental logic, 102–105 Conduct” (APA), 502, 505 experimental research, 16–17, 74–75 ethics, 21–22 racial identity, 303–305 downstream effects, 495, 504–505 experimental scholars collaboration, 6 election experiments, 393–394 Experimental Study of Politics, 4 ﬁeld experiments, 129–130 experimental treatments, political predispositions, social networks, 278–281 109–112 subject recruitment, 41 interactions, 111 exclusion restriction, 24 null hypothesis, 110 Experiential Economics, 7 experiments, 527–528 experiment norms, 7, 9 causal inference and, 3 experiment subjects, 7, 32–34 cooperative game theory and, 91–94 engagement/disengagement, 28 discrimination vs. prejudice, 103–104 participation, 7 laboratory vs. ﬁeld, 42 recruitment, ethical challenges, 41 meta-analysis, 21, 124 students, 42–44, 45–53 noncooperative game theory and, 94–99 subject pool, 63, 66 psychological vs. economic approaches, 7 experimental control, 59, 61, 66 replication, 21 experimental demand, 77 statistical power and, 21 minority populations, 330–332 theoretical development and, 3 experimental designs experiments, examples advertising, 222–224 Asian Flu Experiment, 108 between-subjects, 30 Color-Blind Experiment, 112, 308 block design, 30 CPRs, 339, 340, 344, 345–347 campaign advertising, 222–224 Eldersveld GOTV experiment, 4 candidate impressions, evaluations, 189–190 Gold Shield Coffee study, 280 cross-cultural experiments, 90 High/Scope Perry Experiment, 500–501 encouragement designs, 17 Laid-off Worker Experiment, 111 external validity and, 149 List Experiment, 107–108 framing, permissive designs, 108 Milgram obedience experiment, 35–36, 68 internal design, formal models, 90 MTO study, 283, 502 internal validity and, 28 Multi-Investigator Project, 105 longitudinal designs, 146, 147 New Haven experiment, 117–118, 119, 124 manipulative designs, survey experiments, Regardless of Race Experiment, 112, 308–309 106–107 SAT Experiment, 110 multilevel populations, 487–489, 490–491 Six Degrees of Separation, 279 observational studies, 144–147 Stanford Prison Experiment, 277 one-shot design, information, information TESS, 105 processing, 189 Trust Game, 341 permissive designs, survey experiments, Ultimatum Game, 341 107–108 Vietnam Draft Lottery, 117 Post-test-Only Control Group Design, 148 experiments, political science Pre-test–Post-test Control Group Design, 147 application diversity, 5–6, 7 Subject Index 553

benefits, 9 fairness, nonordinal considerations, 364–366 context, setting, 6 Federal Election Commission (FEC), 118 costs, cost-benefits, 7 Feinstein, Dianne, 78 downstream benefits, 7 field experiments, 278–283 experimental methods diversity, 6–8 artifactual, 116 experimental scholars collaboration, 6 bias, background conditions, 130 field experiments, 7 block design, 30 formal theory testing and, 5 challenges, 126–131 history, 4–5, 73–75, 102–106, 112–113, collaboration with NGOs, 385–387 117–121 collaboration with politicians, 387 methodological challenges, 6–7 collective-action theory, 347–349 observational research and, 5 cost, 130 policymakers and, 5–6, 9 defined, 17, 115–116 subfield examples, 6 ethics, 129–130 subjects participation, 7 external validity, 131 topics, 15 framed, 116 experiments, types.Seefield experiments; history, 117–121 laboratory experiments; natural intellectual context, 117–121 experiments; observational studies internal validity, 132–133 external interference, 36 legality, 236 external validity minority populations, 326–328 assessment, 53 natural, 116 concepts and, 43 realism and, 7, 116 concerns for researchers, 19 spillover effects, 130–131 cross-cultural experiments, 250 subject noncompliance, 32 deliberation and, 268 Field Institute, 314 democracy and development, 391–393 fMRI.Seefunctional magnetic resonance imaging downstream analysis, effects, 31, 38 foreign policy decision making.Seealso evaluation, 44–45 negotiation, mediation experimental design and, 19–20, 149 frames, expectations, 434–436 field experiments, 131 individual preferences aggregation, generalizability, 34–35 government decisions, 436–439 improving, 37–38 PD, 434 internal validity, balance, 38–39 predispositions, preferences, decisions, laboratory election experiments, 374, 381 431–433 observational studies, 146 prospect theory, 435 replication, 34–35, 37 student samples, 434, 438, 439 setting, 210 formal models.Seealsogame theory social networks, 277, 278 best practices, 89 student subjects, 42–44 debate, 89 subjects, settings, context, times, internal design, 90 operationalizations political science, 89 survey experiments and, 7 Fox News, 84 threats to, 35–37 framing eye tracking, 39 attitude change, 323–324 campaigns, 201, 203 Facebook, 278 choice behavior, environments, 61 face-to-face (FTF) interactions, 205 deliberative theory, 263, 265–266 facial appearance, 191, 192–193 field experiments, 116 attractiveness, 166 financial incentives, 62 morphing study, 80–81 first generation experiments, 106 similarities, voter preferences, 79–81 permissive designs, 108 facilitation vs. inhibition effect, 159 racial priming, 320 factor analysis, 173 study of, 207 Fahrenheit 9/11, 204 Framingham Heart Study, 275 554 Subject Index

Franken, Al, 236 field vs. lab experiments, 22 FTF.Seeface-to-face interactions issue of, 81–85 functional magnetic resonance imaging (fMRI), from a single study, 328–329 39, 248 student subjects, 41–42 Get out the Vote!, 124 game theory Get-Out-The-Vote (GOTV) bargaining models, 400–401, 402 campaign contact, 234 centipede game, 98 early field experiments, 117 coalition behavior assumptions, 92–93 external validity, 236 coalition formation, 92–94 future experiments, 331 comparative statics, 400, 401, 403 global warming, 143, 145, 147 competitive solution concept, 93 Gold Shield Coffee study, 280 contributions, 98–99 GOTV.SeeGet-Out-The-Vote conventional lab experiment vs. artifactual field graduation reciprocation in tension reduction experiment, 115, 116 (GRIT), 416 cooperative, 91 Gregoire, Christine, 236 cooperative vs. noncooperative, 91 GSS.SeeGeneral Social Survey core outcomes, 92, 353, 355–356 deception, political psychology, 67 Hawthorne effects, 36, 68 dictator game, 402–403 health care, 179–180 economics experiments, 66 Healthcare Dialog project, 263 games vs., 340 heterogeneity history, 91 canvassers, callers, 234 interpersonal interactions, 91 examining techniques for, 230 investment game, 99, 243, 245 study samples, 53–54 minimal winning coalitions, 93 heuristic trial and error (HTE), 416, 418 monetary incentive, 63, 65 heuristics, 29, 324–326 Nash Equilibrium, 94, 246 High/Scope Perry Experiment, 500–501 noncooperative, 91 home style, 195–196 payoffs, 93–94 hot cognition hypothesis, 163 PD, 91–92, 340, 434 House Rules Committee, 354 persuasion, 96 HTE.Seeheuristic trial and error preferences, 92–93 Human Relations Training Laboratory, 416 punishment in, 59–60, 251–252 repeated games, 371, 401 IAT.SeeImplicit Association Test solution concepts, 92 ideal point trust and, 245–246, 251, 253 agendas, 357–358 ultimatum game, 402–403 fairness, nonordinal points, 364–365 GATT.SeeGeneral Agreements on Tariffs and financial compensation, 354–355 Trade uncovered set, 359–362 gender, 83, 175, 191 of voters, 95, 370 bias, stereotypes, 290–292 identity.Seerace, racial prejudice elective office type, level, 292–293 IE.Seeimplicit-explicit model future research directions, 295–296 Implicit Association Test (IAT) gender quotas, 387–388 enhancing validity, 39 issue competence and, 291–292 implicit attitudes, 156 media effects, 293–294 response competition, 161 observational studies, 289–290 strengths, weaknesses, 161–163 party affiliation and, 291, 294–295 implicit-explicit (IE) model, racial priming, General Agreements on Tariffs and Trade 206–207 (GATT), 414 India, 386, 390, 392 General Social Survey (GSS), 243, 244 Indonesia, 386, 391 generalization Indonesian Kecamatan Development Program, background conditions, 130 386 external validity, 19–20, 28, 34–35, 36, 38 Induced Value Theory, 51, 371 field experiments, 116 information, information processing Subject Index 555

conscious vs. unconscious, 155–158 JQP.SeeJohn Q. Public model dual process theories, minority populations, jury decision making, 96–98 331–332 Dynamic Process Tracing Methodology, 189, Kahneman-Tversky risk aversion studies, 108 194 Kerry, John, 80 laboratory experiments, 369–370, 372–373 Knowledge Networks memory-based, 204 competing ad campaigns study, 221 one-shot design, 189 online research panel, 84 online, 204 political knowledge survey, 173, 174 online, memory-based, 150–151, 157 race, immigration study, 311, 312 selective exposure/elaboration, 149 study follow-up, 225 System 2,System1, 156 INS.SeeInter-Nation Simulation laboratory experiments Institute for Social Research (ISR), 104 abstention, 370, 376, 378, 380 institutional review board (IRB), 235–236 attitude change, 148, 149 “Institution-Free Properties of Social Choice” candidate divergence models, 373–375 (McKelvey), 363 candidate policy preferences, 373–374 institutions, 91, 94, 99 coalition experiments, 401, 404 instrumental variables, 31–32, 117, 118, 216 Condorcet loser, winner, 375–377 intent-to-treat (ITT) effect coordination, three-candidate elections, ATT vs. ATE, 126, 127–129 375–376 campaign contact, 234 defined, 17, 116 defined, 24 external validity, 374, 381 internal validity threats, 30 field experiments vs., 42 multilevel populations, 484–485 game theory, repeated game effects, 371 interactions, experimental treatments, political history, 73, 369–371 predispositions, 111 ideal point, 370 interethnic tension/prejudice reduction, 125 incentives, 90, 365, 371 internal validity, 18–19, 41, 44 Induced Value Theory, 371 attrition, 19, 30–31 information, 369–370, 372–373 downstream effects, 504 information, information processing, 369–370, experiment design and, 28 372–373 experimental economics comparisons, 29–30 lab, lab-in-the-field experiments, 388–389 experimental vs. mundane realism, 28 Mean Voter Theorem, 372–373 external validity, balance, 38–39 methodological issues, 244, 246 field experiments, 132–133 methodology, 276–278, 369–372 formal models, 90 observational studies vs., 371 improving, 34 payoffs, 376 noncompliance and, 19 randomizing, 369–370 observational studies, 145, 146 sequential voting, 376–378 threats to, 30–34 strengths/weaknesses, 380–382 Inter-Nation Simulation (INS), 419, student subjects, 6, 148 422 Swing Voter Curse, 379 International Finance Corporation, 386 treatment effects, 209–210 International Rescue Committee, 386 turnout experiments, 378–380 Internet samples, 36 utility function, 369, 370–371 Internet sampling, 324 voter preferences, 369–370, 374, 375, 376 interpersonal trust, 243, 244–245 voting rules, 374, 376 investment game, 99 Laid-off Worker Experiment, 111 ISR.SeeInstitute for Social Research Latinos issue ownership, 192, 196 California, population vs. electorate ITT.Seeintent-to-treat effect demographics, 228 candidate ethnicity importance, 326 John Q. Public (JQP) model, 159 coethnic contacts, participation, 327 journals, economics, 65 Lausanne Peace negotiations, 424 journals, political science, 74 learning, 207 556 Subject Index legislative behavior framing research and, 207 constituency opinion and, 125 gender and, 293–294 lobbying and, 125 hard vs. soft news, 204 voter knowledge, 125 interethnic tension/prejudice reduction, 125 legislative instability, field work limitations, mass media campaigns, 125 353–354 message processing, 207–208 legislative voting, cycling minimal effects, 202–203 agendas, forward- vs. backward-moving, new media, 204–205 356–358 news, 203–204 Arrovian social choice theory, 353 priming, 81, 124, 125, 151, 190, 205–207, coalitions, 353, 354, 355, 356–357, 365–366 322–323 endogenous agendas, 363–364 television news, 203, 526 fairness, nonordinal considerations, 364–366 mediation analysis House Rules Committee, 354 Baron-Kenney Mediation Model, 509–511 ideal point, 354–355, 357–358, 359–362, bias estimates, 508 364–365 bias, nonexperimental analyses, 509–512 legislative instability, field work limitations, conventional practices defenses, 517–518 353–354 experimental methods, 512–514 majority rule, 353–354 limits, 512–516 majority rule instability, preference-based nonexperimental, 516 constraints, 358–363 in political science, 509 monopoly agenda control, 358 research agenda, 516–517 negotiation, psychological literature, 364 treatment manipulations, 516–517 parallelism, 364 variables, 20 procedural rules, structure-induced MESOs.Seemultiple equivalent simultaneous equilibrium, 356 offers spatial voting models, 354–356 message processing, 207–208 uncovered set, backward-looking, 360 meta-analysis, 21, 124, 219 uncovered set, testing, 360–363 Mexico, 386, 388, 391 Liberia, 386, 389, 391 microcredit loans, 245, 246, 249, 250 List Experiment, 107–108 Milgram obedience experiments, 35–36, 68, 82 lobbying, legislative behavior, 125 minimal group paradigm, 60 local average treatment effect (LATE), minority populations.Seealsorace, racial prejudice local news coverage, racial cues, 78–79 African Americans, 324, 326, 329 The Logic of Collective Action (Olson), 339 AIDS, 325 longitudinal designs, 146, 147 Asian Americans, 326, 327–328 long-term follow-ups, 31 attitude, change, 323–324 Los Angeles, 82, 322 attitude, framing conditions, 323–324 Los Angeles Times, 314 attitudes, implicit vs. explicit, 321, 330 Lowell, A. Lawrence, 3 capital punishment, 323–324 dual process theories, information processing, majority rule, 353–354 331–332 majority rule instability, preference-based general observations, future directions, constraints, 358–363 328–332 manifest effect, 459–462 GOTV, 326–328, 331 manipulative designs, survey experiments, Internet sampling, 324 106–107 Latinos, 228, 327 marginal per capita return (MPCR), 342 media priming, 322–323 math problems, SAT scores, 178–179 pre-testing measures, 329–330 McCain, John, 144 racial cues, heuristics, 324–326 Mean Voter Theorem, 372–373 realism, competition, over-time designs, measurement, 123, 131, 133 330–331 media effects, 203 single study, generalizing, 328–329 bias, 201 survey experiments, 322–324, 326, 328, 331 entertainment, 204 theory, design integration, 331–332 Subject Index 557

treatment effect, 323, 328, 330 National Association of Latino Elected and vote choice extensions, 326 Appointed Officials (NALEO), 327 mobilization.Seealsovoters, voter mobilization National Institutes of Health, 74, 75 current research focus, 125, 126 National Opinion Research Center (NORC), direct/indirect, 449 104 experiments on, 120, 447 National Public Radio (NPR), 84 modality effect, 207, 209 National Science Foundation (NSF), 4, 74, 75, moderation, 20, 54 105 mediation vs., 45, 46 NATO.SeeNorth Atlantic Treaty Organization monetary incentives natural experiments, 22, 453 crowding out, 64–65 democracy and development, 386, economics vs. psychology experiments, 61–65 387–388 game theory, 63, 65 downstream effects, 501 laboratory election experiments, 365, 371 natural field experiment, 116 preferences control, 62–63 negotiation, mediation problems, 64–65 bargaining environment, 416–417 rewarding accuracy, reducing noise, 62 BPA, 416 scale, 63–64 COMROD, 422 social preferences measurement, 62, 63 distributive bargaining, 415–417 monopoly agenda control, 358 electronic mediation, 423–424 monotonicity, 25 experimental method, 414–415 Montreal Protocol, 414 experiments as value-added knowledge, motivations, 62–63, 64 425–426 MoveOn.org, 236 flexibility, 422–423 Moving to Opportunity (MTO) study, 283, 502 frameworks, 422 MPCR.Seemarginal per capita return GRIT, 416 MTO.SeeMoving to Opportunity study HTE, 416, 418 Multi-Investigator Project, 105 information exchange, 419 multilevel populations INS, 419, 422 campaign advertising, 483 integrative bargaining, 417–421 empirical tests, spillovers, ITT estimation, 486, integrative bargaining, strategies, 418–419 489–490 international context, 413–414 experiment design, 487–489 Kennedy experiment, 416 future research, experimental design laboratory complexity, 421–424 recommendations, 490–491 logrolling, 417, 419 instrumental variables, 483, 489, 490, 491 mediator credibility, 420 ITT, 484–485, 486, 489–490 MESOs, limits, best practices, 490 multimethods, 424–425 multilevel context identification, 485–487 NA, 423–424 spillover effects, 482–484, 486, 489–490 offer strategies, 415–416 SUTVA, 482 problem solving in situ, 420–421 treatment, 481–482 problem solving orientation, 419–420 multiple equivalent simultaneous offers (MESOs), PSS, SOS, 417, 420 mundane realism, 75, 81–82 psychological literature, 364 settlement orientation, 417, 420 NA.SeeNegotiator Assistant simulations, cases in training, 425 NAFTA.SeeNorth Atlantic Free Trade simulations vs. cases, 424–425 Agreement situational levers, 422–423 NALEO.SeeNational Association of Latino threshold adjustment, 416 Elected and Appointed Officials Negotiator Assistant (NA), 423–424 NARAL.SeeNational Abortion and Reproductive New Haven experiment Rights Action League extensions from, 231 Nash Equilibrium, 94, 246 impact of, 124, 229, 232 National Abortion and Reproductive Rights literature prior to, 117–118, 119 Action League (NARAL), 237–238 New York Times, 67, 203, 394 558 Subject Index news online information processing, 157–158 entertainment, hard vs. soft news, 204 online studies television, 203 benefits, 85 News That Matters (Iyengar, Kinder), 75 cross-national research, 85–86 NGOs.Seenongovernmental organizations drop-in samples, 83–84 Nicaragua, 392 gender, gender gaps, 83 Nigeria, 386 group dynamics, interpersonal influence, 85 noncompliance participants pool expansion, 83–84 active vs. passive, 31 sampling problem, 84–85 defined, 19, 24–25, 446 selection bias, 83 measured/unmeasured, 447–450 whack-a-pol, 83–84, 85 noncooperative game theory, open-ended recall questions, 173 experiments and, 94–99 Organization for Security and Cooperation, 414 jury decision making, 96–98 OT.Seeoxytocin Nash Equilibrium concept, 94 oxytocin (OT), 248 Nash-Harsanyi-Selten, 91 spatial voting models, 94, 95 Palin, Sarah, 296 voter confidence, 94–96 parallelism, 364 nongovernmental organizations (NGOS) Partial Nuclear Test Ban Treaty, 416 field experiments, collaboration, 385–387 partisanship NORC.SeeNational Opinion Research Center campaign exposure, 77 norms, 7, 9 candidate affiliation, 189–190, 223 North Atlantic Free Trade Agreement (NAFTA), effect on political attitudes, 124 414 face morphing study, 79, 80 North Atlantic Treaty Organization (NATO), 413 partisan political campaigns, 124 Northwestern University, 525 payoffs, 376 NPR.SeeNational Public Radio payoffs, game theory, 93–94 NSF.SeeNational Science Foundation PCL.SeePolitical Communication Laboratory Nuffield Centre for Experimental Social Sciences, PD.Seeprisoner’s dilemma 86 People’s Choice (Lazarsfeld, Berelson, Gaudet), 273 null hypothesis, experimental treatments, political permissive designs, survey experiments, 107–108 predispositions, 110 Perry, Rick, 235 persuasion, 96, 149 Obama, Barrack PG.Seepublic good attitude change studies, 142 The Philadelphia Negro (DuBois), 306 foreign policy experience, 432 physiological measures, 29, 39 implicit attitudes, 159 pivotal voter, 97 primaries, strategic concerns, 144 policymakers, 5–6, 9 observational studies, 16, 116, 117 policy-specific information, 180–181 bias, 121 Polimetrix, 85 candidate gender, 289–290 political communication collective-action theory, 341 causal inference, 75–76 design, 144–147 field experiments, 124 estimation method, 126 observational studies, 216 experimental studies vs., 3, 16–17 student samples, external validity, 42–43 external validity, 146 voter turnout and, 120 internal validity, 145, 146 Political Communication Laboratory (PCL), measurement, 123 83–84 methodological difficulties, 215–218 political documentaries, 204 shortcomings, 188–189, 202 political institutions, policy outcomes, legitimacy, social networks, 273–276 125 surveys, 76, 120, 123, 134, 299, 302 political institutions, trust, social exchange, unmeasured confounding problem, 18 251–253 online deliberation, 262, 263–264 political knowledge, 171–175, 208 online evaluations, automaticity, 158 political participation, social pressure, 125 Subject Index 559 political science black-white divide, U.S., 312–313 behavioral revolution, 73 candidate choice and, 313–317 journals, 74 causal inference, 76 Powell Amendment, 354 Color-Blind Experiment, 112, 308 power, statistical, 21 defined, 306 predisposition, survey experiments, 102 experimental studies, 303–305 preferences framing, 320 attitudes, 142–143 immigration, 312 control, 62–63 implicit bias, 161–163 game theory, 92–93 Laid-off Worker Experiment, 111–112 social preferences measurement, 62, 63 linked fate, 301 voter, 79–81 local news coverage, 78–79 prejudice.Seerace, racial prejudice media, interethnic tension/prejudice reduction, pre-testing measures, 329–330 125 pre-testing measures, minority populations, minimal group paradigm, 300 329–330 nonracial policy preferences, 311–312 pre-treatment effect, 43, 446, 450–451 political behavior research, 300–302 priming effects political ideology, 311, 315–316 agenda setting, 205–207 political science literature, 306–313 defined, 81 prejudice vs. discrimination, 103–104 entertainment media, 204 priming effects, 206–207, 304, 316–317, experimental difficulties, 202 320–323 media influence on politics, 124, 125, 190 racial cues, heuristics, 316–317, 324–326 media priming, 322–323 racial policy preferences, 307–311 media trust factor, 151 Regardless of Race Experiment, 308–309 racial priming, 206–207, 304, 316–317, skin color, 249 320–323 stereotypes, 191–192 Sequential Priming Paradigm, 159 symbolic racism, 307 principle surrogates, 31 Rajasthan, 387–388 prisoner’s dilemma (PD), 91–92, 340, 434 randomization, random assignment problem-solving style (PSS), 417, 420 balance assessment, graphs, tests, 462–464 procedural rules, structure-induced equilibrium, balance test, p values, 467–468 356 benefit, 115–116, 261 processing models.Seecognitive process models, clinical research, 117 candidate evaluation covariance, adjusted for other variables, Progresa Conditional Cash Transfer program, 471–474 386, 388 covariates, random imbalance adjustment, PSS.Seeproblem-solving style 468–477 psychological realism, 44 covariates uses, potential abuses, 461 psychology, 29, 33, 39 defined, 16–17 public good (PG), 339, 340 deliberation study, 267 public opinion, 203, 204 downstream effects and, 495–496 publication bias, 21 graphic model of assessment, 465 group selection, 122, 446 QRE.SeeQuantal Response Equilibrium internal validity, 30, 32 Quantal Response Equilibrium, 98 laboratory election experiments, 369–370 quasi-experiments, 202, 210, 453–454 linear models, adjustment, 469–471 deliberation, 259, 261, 262–263 political science experimentation, 3–4 question wording, sequencing, 103–104, 144, 308 poststratification, blocking and design, 469 don’t know response option, 144 precision enhancement strategies, before assignment, 462–464 TheRaceCard(Mendelberg), 206 public opinion experiments, 111 race, racial prejudice random sampling vs., 17–18 African Americans, 301–302 regression-based adjustment, best practices, associations, political issues, 164 477 560 Subject Index randomization, random assignment (cont.) students, statistical framework, 50–53 scientific aims, 477–478 students vs. other subjects, 50–53, 189 statistical inference and, 461–462 trade-offs, 54–55 statistical inference, linear regression tables, Sarathi Development Foundation, 390 474–477 SAT Experiment, 110 subject sampling, 46–47 school voucher programs, 453–455 RCM.SeeRubin Causal Model Schwarzenegger, Arnold, 188 realism, 77, 116 Scioli, Frank, 74 experimental vs. mundane, 28, 36, 44–45 selection, 446 mundane, 75, 81–82, 208 selective elaboration, 149 psychological, experimental, 208 selective persuasion, 149 realism, competition, over-time designs, 330–331 self-reports Regardless of Race Experiment, 112, 308–309 advertising, 76–77 relative reasoning, 108 attitudes, 142 RENEW.SeeRepublican Network to Elect measurement error, 229 Women response bias, 143 replication September 11, 2001, 434 experimental advantage, 21 Sequential Priming Paradigm, 159 external validity, 27–28, 34–35, 37 sequential voting, 376–378 publication process, 133–134 settlement-oriented style (SOS), 420 Republican Network to Elect Women (RENEW), simulations, 41, 50 294 Six Degrees of Separation, 279 Republican Party, 144 snowball surveys, social networks, 278 research paradigms SOA.SeeStimulus Onset Asynchrony AMP, 156, 161 social context, 446 IAT, 39, 156, 161–163 social identity theory, 60, 299–300 Lexical Decision Task, 161 social networks Sequential Priming Paradigm, 159 baseline attitude measurement, 280 SOA, 159–161 causation, 273 Rio Declaration on Environment and communication flow control, 282 Development, 414 conformity effect, 274, 276 risk aversion studies, Kahneman-Tversky, 108 cross-sections, 273–274 Rubin Causal Model (RCM), 23–25 deliberation, 277 average treatment effect, 445 deliberative polls, 277 downstream analysis, 497 external shocks, 278, 281 noncompliance, 31, 448–449 external validity, 277, 278 treated/untreated states, 122, 481 Facebook, 278 treatment outcome notation, 121 Framingham Heart Study, 275 Rwanda, media study, 125, 388, 389, 390, 391 Gold Shield Coffee study, 280 logistics, ethical concerns, 278–281 SALT.SeeStrategic Arms Limitations Talks MTO study, 283 samples, sampling network analysis, 275–276 campaign experiments, during vs. outside, 224 network randomization, 276–277, convenience, 7, 41 282–283 drop-in, 83–84 observation studies, 273–276 dual, 54 panel studies, 274–275 experiments, limitations, 246 role-playing, 277 Internet, 36, 324 Six Degrees of Separation, 279 online studies, 84–85 small groups, 277–278 random, 46–47 snowball surveys, 278 representativeness, 36, 37, 38, 149, 189, 224, Stanford Prison Experiment, 277 246 strategy convergence, 276 sampling bias, 36, 75, 82–83 subject attrition, 283 snowball sample, 278 social pressure, 125 student subjects, 223–224, 246 social roles, 175 Subject Index 561

SOS.Seesettlement-oriented style relative reasoning, 108 spatial voting models, 94, 95, 354–356 snowball surveys, 278 spillover effects, 482–484 spillover effects, 54 defined, 130–131 split-ballot, 102–103 survey experiments, 54 SRC, 103 treatment, control groups, 482–484 voter choices, 326 split ballot, survey experiments, 102–103 WVS, 243, 244 SRC.SeeSurvey Research Center Survey Research Center (SRC), 103 stable unit treatment value assumption (SUTVA) SUTVA.Seestable unit treatment value compliers, treatment effects, 25 assumption defined, 24, 445 Swing Voter Curse, 379 ITT effect, 484 social transmission, 482 television advertising, 37 Stanford Prison Experiment, 277 television news, 526 START.SeeStrategic Arms Reductions Talks TESS.SeeTime-Sharing Experiments in the statistics, 21 Social Sciences documenting, reporting, 20–21 theoretical development, 3 mediating variables, 20 Thomas, Clarence, 324 moderation and, 20 Time-Sharing Experiments in the Social Sciences stereotypes, 175, 191–193 (TESS), 105 gender, 290–292 treatment effects traits, 290, 291–292 ATT, 17, 122 trust, social exchange, 248–250 average, 16–17, 48 Stimulus Onset Asynchrony (SOA), 159–161 bias, 46 Stony Brook University (SUNY-Stony Brook), 74, compliance, compliers, noncompliers, 446, 194 447–450 Strategic Arms Limitations Talks (SALT), 413 convenience sample, 41 Strategic Arms Reductions Talks (START), 413 defined, 23–24 student subjects, 42 ecological heterogeneity, 453–455 external validity and, 42–44 heterogeneity, 50, 53–54, 446–447 other samples vs., 50–53 homogeneity, 46, 129 statistical framework, 45–50 internal, external validity, 455–456 stylization, 59–61 laboratory experiments, 209–210 subliminal primes.Seeunconscious stimuli manifest effect and, 459–462 subliminal tests, 39 minority populations, 323, 328, 330 survey experiments multilevel settings, 481–482 counter-argument technique, 109 noncompliance and, 25 defined, 17 pre-treatment, 43, 446, 450–451 design classification, 106–109 selection, 446 difficulties, 76, 123 self-selection, 451–453 experimentation vs., 134 social context, 446 external validity and, 7 student sample, statistics, 45 facilitative designs, 108–109 treated/untreated groups, 121–122 GSS, 243, 244 voter contact, 232 history, 102–106, 112–113 voter turnout, 120 majorities vs. counter-majorities, equilibrium trust, social exchange, 98, 99 conditions, 111–112 biological, neural mechanisms, 246, 248, 251, manipulative designs, 106–107 253 minority populations, 322–324, 326, 328, 331 causality and, 253 participant recall errors, 76 correlates, 247–248 participation causes, 120 cross-cultural, 250–251 permissive designs, 107–108 encapsulated trust, 245 political knowledge survey, 173, 174 game theory, 245–246, 251, 253 predisposition and, 102 genetics, 248 racial identification, 331 in government vs. interpersonal, 244 562 Subject Index trust, social exchange (cont.) vote choice, 194 individual characteristics, 247–248 vote margin, 117, 118–119 interpersonal, 243, 244–245 voters, turnout investment game, 245 advertising, 218 methodological issues, 246–247 campaign mobilization, 119–121 microcredit loans, 245, 246, 249, 250 instrumental variables, 32 OT hormone and, 248 negative advertising, 31, 219–220 physiological measures, 248 social pressure, 125 political institutions, 251–253 voter contact study, 117 reciprocity, 243, 245 voters, voter mobilization religion and, 247 ANES, 229 research challenges, 243 citizen lobbying, 236, 238 risk tolerance and, 247–248 civic participation, 237–238 social capital, 249, 250 downstream effects, 496, 501 social networks and, 253 extensions to first experiments, 230–232 socioeconomic status, 247 field experiments, ethical concerns, stereotypes, 248–250 235–236 twin studies, 248 field experiments, external validity, 234–235 turnout experiments, 378–380, 449 field experiments, implementation problems, 232–234 Uganda, studies field experiments, legality, 236 ethnic cooperation, 116 new technologies, 236–237 ethnic diversity, public goods, 389, 391 observational studies, 228–232 legislators’ attendance, 125 pioneering experiments, 229–230 NGOs, ex-combatant reintegration, 386 social psychological theories, 231–232 ultimatum game, 29, 341 voter choice, 237 UN Security Council, 414 voter files, 234 unconscious stimuli, 164–166 Voting (Berelson, Lazarsfeld, McPhee), uncovered set, backward-looking, 360 273 uncovered set, testing, 360–363 voting rules, 374, 376 United States–Japan Administrative Agreement, 420 Warsaw Pact, 413 unmeasured confounding, 18 Washington Post, 124, 125 U.S. Air Force Academy, 432 Washington Times, 124, 125 utility function, 369, 370–371 WashingtonPost.com, 84 WashingtonPost.com-PCL collaborative validity.Seeexternal validity; internal validity experiments, 84 VCM.SeeVoluntary Contribution Mechanism Wason selection task, 60–61 Vienna Academy of Diplomacy, 423 Wason task/test, 409 Vietnam Draft Lottery experiment, 117 WebTV, 84 Voluntary Contribution Mechanism (VCM), 340, welfare policy, 181 342 West Bengal, 387–388 vote choice extensions, 326 whack-a-pol online experiments, 83–84 Voter News Service, 313 Willie Horton advertisement, 76, 165 voters Wilson, Pete, 78 behavior, causal modeling, 187 within-subjects design, 17–18, 30, 371 competence, 177–179 women candidates, political representation, confidence, 94–96 289–290, 294.Seealsogender knowledge, legislative behavior, 125 World Bank, 386 negative advertising, voting likelihood, World Values Survey (WVS), 243, 244 218–220 WVS.SeeWorld Values Survey preferences, 369–370, 374, 375, 376 preferences, facial similarities and, 79–81 Zurich Program in the Foundations of Human uninformed, 171–175 Behavior, 86