Statistical Inference and Power Analysis for Direct and Spillover Effects in Two-Stage Randomized Experiments∗ Zhichao Jiang† Kosuke Imai‡ November 16, 2020 Abstract Two-stage randomized experiments are becoming an increasingly popular experimental design for causal inference when the outcome of one unit may be affected by the treatment assignments of other units in the same cluster. In this paper, we provide a methodological framework for general tools of statistical inference and power analysis for two-stage randomized experiments. Under the randomization-based framework, we propose unbiased point estimators of direct and spillover effects, construct conservative variance estimators, develop hypothesis testing proce- dures, and derive sample size formulas. We also establish the equivalence relationships between the randomization-based and regression-based methods. We theoretically compare the two-stage randomized design with the completely randomized and cluster randomized designs, which rep- resent two limiting designs. Finally, we conduct simulation studies to evaluate the empirical performance of our sample size formulas. For empirical illustration, the proposed methodology is applied to the analysis of the data from a field experiment on a job placement assistance program. Keywords: experimental design, interference between units, partial interference, spillover effects, statistical power ∗Imai thanks the Alfred P. Sloan Foundation for partial support (Grant number 2020{13946). †Assistant Professor, Department of Biostatistics and Epidemiology, University of Massachusetts, Amherst MA 01003. ‡Professor, Department of Government and Department of Statistics, Institute for Quantitative Social Sci- ence, Harvard University, Cambridge MA 02138. Phone: 617{384{6778, Email: [email protected], URL: https://imai.fas.harvard.edu 1 Introduction Much of the early causal inference literature relied upon the assumption that the outcome of one unit cannot be affected by the treatment assignment of another unit. Over the last two decades, however, researchers have made substantial progress by developing a variety of methodological tools to relax this assumption (see e.g., Sobel, 2006; Rosenbaum, 2007; Hudgens and Halloran, 2008; Tchetgen Tchetgen and VanderWeele, 2010; Forastiere et al., 2016; Aronow and Samii, 2017; Athey et al., 2018; Basse and Feller, 2018; Imai et al., 2020, and many others). Two-stage randomized experiments, originally proposed by Hudgens and Halloran (2008), have become an increasingly popular experimental design for studying spillover effects. Under this exper- imental design, researchers first randomly assign clusters of units to different treatment assignment mechanisms, each of which has a different probability of treatment assignment. For example, one treatment assignment mechanism may randomly assign 80% of units to the treatment group whereas another mechanism may only treat 40%. Then, within each cluster, units are randomized to the treatment and control conditions according to its selected treatment assignment mechanism. By comparing units who are assigned to the same treatment conditions but belong to different clus- ters with different treatment assignment mechanisms, one can infer how the treatment conditions of other units within the same cluster affect one's outcome. Two-stage randomized experiments are now frequently used in a number of disciplines, including economics (e.g., Cr´epon et al., 2013; Angelucci and Di Maro, 2016), education (e.g., Muralidharan and Sundararaman, 2015; Rogers and Feller, 2018), political science (e.g., Sinclair et al., 2012), and public health (e.g., Benjamin-Chung et al., 2018). The increasing use of two-stage randomized experiments in applied scientific research calls for the development of general methodology for analyzing and designing such experiments. Building on the prior methodological literature (e.g., Hudgens and Halloran, 2008; Basse and Feller, 2018; Imai et al., 2020), we consider various causal quantities, representing direct and spillover effects, and develop their unbiased point estimators and conservative variance estimators under the nonparamet- ric randomization-based framework. We also show how to conduct hypothesis tests and derive the sample size formulas for the estimation of these causal effects. The resulting formulas can be used to conduct power analysis when designing two-stage randomized experiments. Finally, we theoretically compare the two-stage randomized design with its two limiting designs, the completely randomized and cluster randomized designs. Through this comparison, we analyze the potential efficiency loss of the two-stage randomized design when no spillover effect exists. 1 We make several methodological contributions to the literature. First, the proposed causal quantities generalize those of Hudgens and Halloran (2008) to more than two treatment assignment mechanisms. We consider the joint estimation of the average direct and spillover effects to charac- terize the causal heterogeneity across different treatment assignment mechanisms. We also propose the average marginal direct effect as a scalar summary of several average direct effects. Second, our variance estimators are guaranteed to be conservative while those of Hudgens and Halloran (2008) are not when generalized to our setting. Third, while Baird et al. (2018) develops power analysis under similar settings, they adopt linear regression models and focus on the randomized saturation design, in which the proportion of treated units for each cluster is considered as a parameter to be op- timized. In contrast, we study the standard two-stage randomized design and use the nonparametric randomization-based framework without making modeling assumptions. In addition, we prove the equivalence relationships between the proposed randomization-based estimators and the popular least squares estimators. This extends the result of Basse and Feller (2018) to more general two-stage randomized experiments. Our result can also be viewed as a generalization of Samii and Aronow (2012) in the presence of interference. We conduct simulation studies to evaluate the sample size formulas and use data from an experiment on a job placement assistance program to illustrate the proposed methodology. The remainder of the paper is organized as follows. Section 2 introduces our motivating study, which uses the two-stage randomized design to evaluate the efficacy of a job placement assistance pro- gram (Cr´epon et al., 2013). In Section 3, we formally present the two-stage randomized design and define the three causal quantities of interest. In Section 4, we propose a general methodology for sta- tistical inference and power analysis. While Sections 5 presents simulation studies, Section 6 revisits the evaluation study of the job placement assistance program and apply the proposed methodol- ogy. Finally, Section 7 compares the two-stage randomized design with the cluster and individual randomized designs before providing concluding remarks in Section 8. 2 Randomized Evaluation of a Job Placement Assistance Program In this section, we describe the randomized evaluation of a job placement assistance program (Cr´epon et al., 2013), which serves as a motivating application. The goal of the study is to assess the impacts of the job placement assistance program on the labor market outcomes of young, educated job seekers in France. The experiment took place in a total of 235 areas (e.g., cities), each of which is covered by one of the French public unemployment agency offices in 10 administrative regions. Each office 2 Treatment assignment mechanisms I II III IV V Treatment assignment probability 0% 25% 50% 75% 100% Number of clusters 47 47 47 47 47 Number of job seekers 4,467 4,839 4,899 4,598 4,517 Table 1: The Two-stage Randomized Design for the Evaluation of the Job Placement Assistance Program. represents a small labor market. The program eligibility criteria are based on age (30 years old or younger), education (at least a two-year college degree), and unemployment status (having spent either 12 out of the last 18 months or 6 months continuously unemployed). The evaluation was conducted through the two-stage randomized design shown in Table 1. In the first stage, 235 areas were randomly assigned to one of five treatment assignment mechanisms, which correspond to different levels of treatment assignment probabilities: 0, 25, 50, 75, and 100%. In the second stage, job seekers were assigned to the treatment within each area according to its treatment assignment probability chosen in the first stage. Those assigned to the treatment group were offered an opportunity to enroll in the job placement program, whereas those in the control group received the standard placement assistance. Job seekers participated in the experiment as 14 monthly cohorts, starting in September 2007. The study focused on cohorts 3{11, which consists of 11,806 unemployed individuals. It targeted two binary labor market outcomes after eight months of the assignment: fixed-term contract of six months or more (LTFC) and permanent contract (PC). Four follow-up surveys were conducted 8 months, 12 months, 16 months, and 20 months after the treatment assignment, which collected the information on labor market outcomes. Both direct and spillover effects are of interest in this evaluation. While the direct effect charac- terizes how much the job seekers would benefit from the program, the spillover effect corresponds to the displacement effect, representing the possibility that job