Statistics in the Biosciences manuscript No. (will be inserted by the editor) Defining the Study Population for an Observational Study to Ensure Sufficient Overlap: A Tree Approach Mikhail Traskin · Dylan S. Small Received: date / Accepted: date Abstract The internal validity of an observational study is enhanced by only comparing sets of treated and control subjects which have sufficient overlap in their covariate distributions. Methods have been developed for defining the study population using propensity scores to ensure sufficient overlap. However, a study population defined by propensity scores is difficult for other investi- gators to understand. We develop a method of defining a study population in terms of a tree which is easy to understand and display, and that has similar internal validity as that of the study population defined by propensity scores. Keywords Observational study · Propensity score · Overlap 1 Introduction An observational study attempts to draw inferences about the effects caused by a treatment when subjects are not randomly assigned to treatment or control as they would be in a randomized trial. A typical approach to an observational study is to attempt to measure the covariates that affect the outcome, and then adjust for differences between the treatment and control groups in these covariates via propensity score methods [25,16], matching methods [21,13] or regression; see [18], [22] and [27] for surveys. Mikhail Traskin Department of Statistics, The Wharton School, University of Pennsylvania, Philadelphia, PA 19104, USA Tel.: +215-898-8231 Fax: +215-898-1280 E-mail: [email protected] Dylan S. Small Department of Statistics, The Wharton School, University of Pennsylvania, Philadelphia, PA 19104, USA E-mail: [email protected] 2 A quantity that is often of central interest in an observational study is the average treatment effect, the average effect of treatment over the whole population. However, in many observational studies, there is a lack of overlap, meaning that parts of the treatment and control group’s covariate distribu- tions do not overlap. For example, in studies of the comparative effectiveness of medical treatments, virtually all patients of certain types may receive a par- ticular treatment and virtually no patients of certain other types may receive this treatment, but there may be a marginal group of patients who may or may not receive the treatment depending upon circumstances such as where the patient lives, patient preference or the physician’s opinion [24]. When there is a lack of overlap, inferences for the average treatment effect rely on extrap- olation. This is because extrapolation is needed to estimate the treatment effect for those subjects in the treatment group whose covariates differ sub- stantially from any subjects in the control group (or vice versa). Rather than rely on extrapolation, it is common practice to limit the study population to those subjects with covariates that lie in the overlap between the treatment and control groups. In comparative effectiveness studies, these subjects in the overlap are the marginal patients for whom there is some chance that they would receive either treatment or control based on their covariates. Focusing on the average treatment effect for the marginal patients rather than all pa- tients enhances the internal validity of the study and is more informative for deciding how to treat patients for whom there is currently no definitive stan- dard of care. Knowing the average treatment effect on the currently marginal patients may also shift the margin and over a period of time, a sequence of studies may gradually shift the consensus of which patients should be treated [24]. A number of approaches have been developed for defining a study popula- tion that has overlap between the treated and control groups. An often used approach is to discard subjects whose propensity score values fall outside the range of propensity scores in the subsample with the opposite treatment [9, 27]; the propensity score is the probability of receiving treatment given the measured covariates [25]. This approach seeks to make the study population as large as possible while maintaining overlap. However, if there are areas of limited overlap, that is parts of the covariate space where there are lim- ited numbers of observations for the treatment group compared to the control group or vice versa, the average treatment effect estimate on this study popu- lation may have large variance. It may be better to consider a more restrictive study population for which there is sufficient overlap. The goal is to define a study population which is as inclusive as possible but for which there is enough overlap that, not only is extrapolation not needed, but also the aver- age treatment effect can be well estimated. Crump, Hotz, Imbens and Mitnik [8] and Rosenbaum [24] develop criterion by which to compare different choices of study populations according to this goal and then choose the study pop- ulation to optimize these criteria. Crump et al.’s criterion is the variance of the estimated average treatment effect on the study population. They show that, under some conditions, the optimal study population by this criterion is 3 those subjects whose propensity scores lie in an interval [α, 1 − α] with the op- timal interval determined by the marginal distribution of the propensity score and usually well approximated by [0.1, 0.9]. Rosenbaum’s criterion involves balancing (i) the sum of distances between the propensity scores (or related quantities) of matched treated and control subjects in the study population and (ii) the size of the study population; the criterion will be explained in detail in Section 2.3. Rosenbaum develops an algorithm that uses the optimal assignment algorithm for choosing the optimal study population according to his criterion. All of the above approaches define the study population in terms of the propensity score and related quantities. A difficulty with these approaches is that it is hard to have a clear understanding of a study population defined in terms of propensity scores. Rosenbaum, in his book Design of Observational Studies [23], states, “Rather than delete individuals one at a time based on extreme propensity scores, it is usuallybettertogobacktothecovariates themselves, perhaps redefining the population under study to be a subpopu- lation of the original population of subjects. A population defined in terms of [the propensity score] is likely to have little meaning to other investigators, whereas a population defined in terms of one or two familiar covariates will have a clear meaning.” Our goal in this paper is to develop an approach to defining a study pop- ulation that has sufficient overlap and is good by Crump et al.’s [8] criterion or Rosenbaum’s [24] criterion for judging study populations, but that is also easily described. Our approach is to use a classification tree [4] to define the study population in a way that approximates a propensity score based rule for defining the study population. The resulting study population is easily de- scribed by a tree diagram. Figure 2 provides an example of a study population that is defined in terms of a classification tree. Our paper is organized as follows. Section 2 provides the framework and assumptions that we will use as well as reviewing Crump et al.’s (Section 2.2) and Rosenbaum’s (Section 2.3) approaches to defining the study population. Section 3 describes our method. Sections 4 and 5 presents examples. Section 6 provides discussion. 2 Framework 2.1 Assumptions and Notation The framework we use is that of [25]. We have a random sample of size N from a large population. For each subject i in the sample, let Di denote whether or not the treatment of interest was received, with Di =1ifsubjecti receives 1 the treatment of interest and Di = 0 if subject i receives the control. Let Yi 0 be the outcome that subject i would have if she received the treatment and Yi be the outcome that subject i would have if she received the control; these are 1 0 called potential outcomes. The treatment effect for subject i is τi = Yi − Yi . 4 Let τ(X) denote the average treatment effect for subjects with covariates X, 1 0 τ(X)=E[Yi − Yi |Xi = X]. Di We observe Di and Yi,whereYi = Yi . In addition, we observe a vector of pre-treatment covariates denoted by Xi, where the support of the covariates is X. The propensity score e(X) for a subject with covariates X is the probability of selection into the treatment given X, e(X)=P (Di =1|Xi = X). We make the assumption that all of the confounders have been measured [25]. This means that, conditional on the covariates X, the treatment indicator is independent of the potential outcomes: 0 1 Assumption 1:Di⊥⊥(Yi ,Yi )|Xi. Assumption 1 is called the strongly ignorable treatment assignment assump- tion [25] or the unconfoundedness assumption [18]. 2.2 Minimum Variance Approach to Defining the Study Population Crump et al. [8] seek to choose the study population that allows for the most precise estimation of the average treatment effect within the study population. They show that, under some conditions, this leads to discarding observations with propensity scores outside an interval [α, 1 − α] with the optimal cut-off value determined by the marginal distribution of the propensity score. Their approach is consistent with the common practice of dropping subjects with extreme values of the propensity score with two differences. First, the role of the propensity score in the selection rule is not imposed a priori, but emerges as a consequence of the criterion, and second, there is a principled way of choosing the cutoff value α. They show that the precision gain from their approach can be substantial with most of the gain captured by using a rule of thumb to discard observations with estimated propensity score outside the range [0.1, 0.9].