The Econometrics of Randomized Experiments∗ Susan Atheyy Guido W. Imbensz Current version June 2016 Keywords: Regression Analyses, Random Assignment, Randomized Experiments, Potential Outcomes, Causality ∗We are grateful for comments by Esther Duflo. yProfessor of Economics, Graduate School of Business, Stanford University, and NBER, [email protected]. zProfessor of Economics, Graduate School of Business, Stanford University, and NBER, im- [email protected]. [1] 1 Introduction Randomized experiments have a long tradition in agricultural and biomedical settings. In eco- nomics they have a much shorter history. Although there have been notable experiments over the years, such as the RAND health care experiment (Manning, Newhouse, Duan, Keeler and Leibowitz, 1987, see the general discussion in Rothstein and von Wachter, 2016) and the Neg- ative Income Tax experiments (e.g., Robins, 1985), it is only recently that there has been a large number of randomized experiments in economics, and development economics in particu- lar. See Duflo, Glennerster, and Kremer (2006) for a survey. As digitization lowers the cost of conducting experiments, we may expect that their use may increase further in the near future. In this chapter we discuss some of the statistical methods that are important for the analysis and design of randomized experiments. Although randomized experiments avoid many of the challenges of observational studies for causal inference, there remain a number of statistical issues to address in the design and analysis of experiments. Even in the simplest case with observably homogenous, independent subjects, where the experiment is evaluated by comparing sample means for the treatment and control group, there are questions of how to conduct inference about the treatment effect. When there are observable differences in characteristics among units, questions arise about how best to design the experiment and how to account for imbalances in characteristics between the treatment and control group in analysis. In addition, it may be desirable to understand how the results of an experiment would generalize to different settings. One approach to this is to estimate heterogeneity in treatment effects; another is to reweight units according to a target distribution of characteristics. Finally, statistical issues arise when units are not independent, as when they are connected in a network. In this chapter, we discuss a variety of methods for addressing these and other issues. A major theme of the chapter is that we recommend using statistical methods that are di- rectly justified by randomization, in contrast to the more traditional sampling-based approach that is commonly used in econometrics. In essence, the sampling based approach considers the treatment assignments to be fixed, while the outomes are random. Inference is based on the idea that the subjects are a random sample from a much larger population. In contrast, the randomization-based approach takes the subject's potential outcomes (that is, the outcomes [1] they would have had in each possible treatment regime) as fixed, and considers the assignment of subjects to treatments as random. Our focus on randomization follows the spirit of Freedman (2006, p. 691), who wrote: \Experiments should be analyzed as experiments, not as observa- tional studies. A simple comparison of rates might be just the right tool, with little value added by `sophisticated' models." Young (2016) has recently applied randomization-based methods in development economics. As an example of how the randomization-based approach matters in practice, we show that methods that might seem natural to economists in the conventional sampling paradigm (such as controlling for observable heterogeneity using a regression model) require additional assumptions in order to be justified. Using the randomization-based approach suggests alternative methods, such as placing the data into strata according to covariates, analyzing the within-group exper- iments, and averaging the results. This is directly justified by randomization of the treatment assignment, and does not require any additional assumptions. Our overall goal in this chapter is to collect in one place some of the most important statis- tical methods for analyzing and designing randomized experiments. We will start by discussing some general aspects of randomized experiments, and why they are widely viewed as providing the most credible evidence on causal effects. We will then present a brief introduction to causal inference based on the potential outcome perspective. Next we discuss the analysis of the most basic of randomized experiments, what we call completely randomized experiments where, out of a population of size N, a set of Nt units are selected randomly to receive one treatment and the remaining Nc = N−Nt are assigned to the control group. We discuss estimation of, and inference for, average as well as quantile treatment effects. Throughout we stress randomization-based rather than model-based inference as the basis of understanding inference in randomized experi- ments. We discuss how randomization-based methods relate to more commonly used regression analyses, and why we think the emphasis on randomization-based inference is important. We then discuss the design of experiments, first considering power analyses and then turning to the benefits and costs of stratification and pairwise randomization, as well as the complications from re-randomization. We recommend using experimental design rather than analysis to ad- just for covariates differences in experiments. Specifically, we recommend researchers to stratify the population into small strata and then randomize within the strata and adjust the standard errors to capture the gains from the stratification. We argue that this approach is preferred [2] to model-based analyses applied after the randomization to adjust for differences in covariates. However, there are limits on how small the strata should be: we do not recommend to go as far as pairing the units, because it complicates the analysis due to the fact that variances cannot be estimated within pairs, whereas they can within strata with at least two treated and two control units. This chapter draws from a variety of literatures, including the statistical literature on the analysis and design of experiments, e.g., Wu and Hamada (2009), Cox and Reid (2000), Altman (1991), Cook and DeMets (2008), Kempthorne (1952, 1955), Cochran and Cox (1957), Davies (1954), and Hinkelman and Kempthorne (2005, 2008). We also draw on the literature on causal inference, both in experimental and observational settings, Rosenbaum (1995, 2002, 2009), Rubin (2006), Cox (1992), Morgan and Winship (2007), Morton Williams (2010) and Lee (2005), and Imbens and Rubin (2015). In the economics literature we build on recent guides to practice in randomized experiments in development economics, e.g., Duflo, Glennerster, and Kremer (2006), Glennerster (2016), and Glennerster and Takavarasha (2013) as well as the general empirical micro literature (Angrist and Pischke, 2008). There have been a variety of excellent surveys of methodology for experiments in recent years. Compared to Duflo, Glennerster and Kremer (2006), Glennerster and Takavarasha (2013) and Glennerster (2016), this chapter focuses more on formal statistical methods and less on issues of implementation in the field. Compared to the statistics literature, we restrict our discussion largely to the case with a single binary treatment. We also pay more attention to the compli- cations arising from non-compliance, clustered randomization, and the presence of interactions and spillovers. Relative to the general causal literature, e.g., Rosenbaum (1995, 2009) and Imbens and Rubin (2015), we do not discuss observational studies with unconfoundedness or selection-on-observables in depth, and focus more on complications in experimental settings. This chapter is organized as follows. In Section 8 we discuss the complications arising from cluster-level randomization. We discuss how the use of clustering required the researcher to make choices regarding the estimands. We also focus on the choice concerning the unit of analysis, clusters or lower-level units. We recommend in general to focus on cluster-level analyses as the primary analyses. Section 9 contains a discussion of non-compliance to treatment assignment and its relation to instrumental variables methods. In Section 10 we present some recent results for analyzing heterogeneity in treatment effects. Finally, Section 11 we discuss violations of the no- [3] interaction assumption, allowing outcomes for one unit to be affected by treatment assignments for other units. These interactions can take many forms, some through clusters, and some through general networks. We show that it is possible to calculate exact p-values for tests of null hypotheses of no interactions while allowing for direct effects of the treatments. Section 12 concludes. 2 Randomized Experiments and Validity In this section we discuss some general issues related to the interpretation of analyses of ran- domized experiments and their validity. Following Cochran (1972, 2015) we define randomized experiments as settings where the the assignment mechanism does not depend on characteristics of the units, either observed or unobserved, and the researcher has control over the assignments. In contrast, in observational studies (Rosenbaum, 1995; Imbens and Rubin, 2015), the researcher does not have control over the assignment