Experimental Thinking: A Primer on Social Science Experiments by James N. Druckman [email protected] Department of Political Science Northwestern University Scott Hall 601 University Place Evanston, IL 60208 Phone: 847-491-7450 1 To Marj and Dan Druckman 2 Table of Contents List of Figures List of Tables Preface Acknowledgements Chapter 1: Why a Primer on Social Science Experiments? Chapter 2: The Scientific Process and How to Think about Experiments Chapter 3: Evaluating Experiments: Realism, Validity, and Samples Chapter 4: Innovations in Experimental Designs: Opportunities and Limitations Chapter 5: What to Do Before, During, and After an Experiment Chapter 6: Designing “Good” Experiments References 3 List of Figures Figure 1-1: American Political Science Review Experimental Articles By Decade Figure 4-1: Audit Study Logic Figure 4-2: Pager’s Results Figure 4-3: Washing Machined Profile 1 Figure 4-4: Washing Machined Profile 2 Figure 4-5: Immigrant Profile Figure 4-6: Immigrant Conjoint Results 4 List of Tables Table 2-1: Measurement Validity Concepts Table 2-2: Assumptions Underlying Solutions to the Fundamental Problem of Causal Inference Table 2-3: Experimental Approaches Table 3-1: Types of Validity Table 3-2: Generalization Questions Table 4-1: Using Experiments to Study Policy Table 5-1: ASK: Examples of How to Ask Research Questions Table 5-2: Freese and Peterson’s (2017) Forms of “Replication” 5 Preface Experiments are a central methodology in the social sciences. Scholars from every discipline regularly turn to experiments. Practitioners rely on experimental evidence in evaluating social programs, policies, institutions, and information provision. The last decade has seen a fundamental shift in experimental social science thanks not only to their emergence as a primary methodology in many disciplines, but also to technological advances and evolving sociological norms (e.g., open science). This book is about how to “think” about experiments in light of these changes. It argues that designing a good experiment is a slow moving process (given the host of considerations) which is counter to the current fast moving temptations available in the social sciences. The book includes discussion of the place of experiments in the social science process, the assumptions underlying different types of experiments, the validity of experiments, the application of different designs (such as audit field experiments and conjoint survey experiments), how to arrive at experimental questions, the role of replications in experimental research, and the steps involved in designing and conducting “good” experiments. The goal is to ensure social science research remains driven by important substantive questions and fully exploits the potential of experiments in a thoughtful manner. 6 Acknowledgements In some ways, I have been writing this book for much of my life. At an early age, my mom, Marj Druckman, taught me how to systematically address problems and think about them from multiple angles. Meanwhile, my dad, Dan Druckman, exposed me to the social sciences and taught me how to think about experiments. He even invited me to work with him on my first experiment. I thank them both and dedicate this book to them. I also thank the many students, teachers, and colleagues who have shaped my thinking on experiments. Dozens of students have challenged me to think of experiments in new ways; they also have shown me how to creatively use experiments to address a wide range of questions. Several current and former students assisted in the writing of this book including Robin Bayes, Adam Howat, Maya Novak-herzog, Richard Shafranek, Andrew Thompson, and Anna Wang. Jake Rothschild insightfully commented on the entire book and I am extremely appreciative. I also thank my early mentors – particularly Ken Janda, Skip Lupia, and Mat McCubbins – for teaching me about social science methods and experiments. Mat read the first chapter and in his typical fashion, asked “what’s your point?”, thereby forcing me to make it clearer. I am indebted to several colleagues who have talked with me about experiments and/or encouraged me to flesh out my thoughts over the years. A partial list includes Adam Berinsky, Toby Bolsen, Tabitha Bonilla, Cheryl Boudreau, John Bullock, Ethan Busby, Dennis Chong, James Chu, Fay Lomax Cook, Charles Crabtree, Catherine Eckel, David Figlio, Eli Finkel, D.J. Flynn, Jeremy Freese, Laurel Harbridge-Yong, Larry Hedges, Lenka Hrbková, Shanto Iyengar, Cindy Kam, John Kane, Martin Kifer, Samara Klar, Jim Kuklinski, Thomas Leeper, Neil Malhotra, Leslie McCall, Rose McDermott, Mary McGrath, Luke Miratrix, Dan Molden, the late Becky Morton, Kevin Mullinix, Diana Mutz, Tom Nelson, Dan O’Keefe, Mike Parkin, Erik Peterson, Dave 7 Rand, Jenn Richeson, Josh Robison, John Barry Ryan, Monica Schneider, Libby Sharrow, the late Lee Sigelman, Nick Stagnaro, Ben Tappin, Alex Theodoridis, Paul Sniderman, Sophie Trawalter, Lynn Vavreck, Jan Voelkel, Robb Willer, and Teppei Yamamoto. I am particularly indebted to colleagues who read and commented on the book. Adam Levine did so, sending along nearly ten pages of profound comments. This led to substantial revisions which Yanna Krupnikov, Matt Levendusky, and Rune Slothuus were kind enough to read – offering extreme insight and encouragement. This is nothing new as these colleagues have been vital sounding boards for years. The same is true of Don Green who played a crucial role in supporting this endeavor. I thank Robert Dreesen from Cambridge who guided me at every stage and the anonymous reviewers whose comments pressed me to think more deeply about many of the topics covered in the book. I am especially appreciative of Yoshi Ono who arranged for me to give several presentations on experimentation in Japan during the summer of 2019. He was a superb host and is a great colleague. I also thank the participants who attended a conference on advances in experimental political science at Northwestern University in May, 2019. Their presentations and conversations (and the edited volume, co-edited by myself and Don Green, to which they contributed) significantly shaped my thinking. I thank the National Science Foundation (SES-1822286), the Ford Motor Company Center for Global Citizenship at the Kellogg School of Management at Northwestern University (directed by David Austen-Smith), the Institute for Policy Research at Northwestern University, and the Department of Political Science at Northwestern University for supporting the conference. All of these experiences formed the basis for this book. Finally, my family Nikki, Jake, and Sam have been a constant source of support in all the best ways. 8 Chapter 1: Why a Primer on Social Science Experiments? In the 1909 American Political Science Association’s presidential address, A. Lawrence Lowell stated “We are limited by the impossibility of experiment. Politics is an observational, not an experimental, science…” (Lowell 1910: 7). One hundred and ten years later, the Association’s president, Rogers Smith (2020: 15), raised the question of “whether an excessive emphasis on experiments will unduly constrict the questions political scientists ask…” Clearly, much has changed in political science.1 The same can be said about many social science disciplines where experiments have evolved from a non-existent method to an accepted method to a primary method. Even psychology – where experiments have forever been a central approach – has experienced substantial changes in the last decade due to shifts in the social sciences. Specifically, massive technological advances have facilitated data access and analysis, 1 While not my focus, another contrast between Lowell and Smith concerns their perspectives on race. Lowell served as the President of Harvard from 1909 to 1933, during which time he attempted to ban African-American students from living in freshman halls (Sollors et al. 1993). In contrast, a fair amount of Smith’s work explores the incorporation of minorities into political life, such as pointing out the United States has an ascriptive tradition that involves sexism, racism, and nativism (Smith 1993). He states in his presidential address, “we are not going to be able to understand major political developments of the past, present, and future if we do not explore more deeply the politics of identity formation, using all methods that can help” (12). This comparison reveals the extent to which the discipline has changed from one that largely ignored race for the first part of its history (Blatt 2018) to one that is now recognizing the central role of studying race and identity. 9 and social scientists from all disciplines have called for more “open science” practices that involve transparency and replication. There is a concern, however, that these changes may cause experimentalists to become “methods driven,” neither asking appropriate questions nor maximizing the potential of the method (e.g., Thelen and Mahoney 2015: 19). These apprehensions accentuate the need for careful discussion of the experimental method. That is the goal of this book: the hope is to provide readers with a way to “think” about experiments, both as users and consumers. I do this from the perspective of a political scientist and thus I discuss the evolution of experiments in political science and use many examples from political science. That said, the arguments I make and the suggestions I offer apply to any social science application of an experiment – which as will be discussed in detail – I define as a study where an intervention (by a researcher or a natural event) provides the primary mechanism by which one attempts to make a casual claim. What follows can be read by those with no background and/or interest in political science. To be clear, this manuscript is not a vigorous defense of experiments, although it will become apparent that experiments have far-reaching applications. Further, the book is neither a textbook on experimental design and analyses – many such treatments exist – nor an advanced discussion of new developments, which is available in Druckman and Green’s (2021) edited volume.