External Validity: from Do-Calculus to Transportability Across Populations
Total Page：16
File Type：pdf, Size：1020Kb
Statistical Science 2014, Vol. 29, No. 4, 579–595 DOI: 10.1214/14-STS486 c Institute of Mathematical Statistics, 2014 External Validity: From Do-Calculus to Transportability Across Populations Judea Pearl and Elias Bareinboim Abstract. The generalizability of empirical findings to new environ- ments, settings or populations, often called “external validity,” is es- sential in most scientific explorations. This paper treats a particular problem of generalizability, called “transportability,” defined as a li- cense to transfer causal effects learned in experimental studies to a new population, in which only observational studies can be conducted. We introduce a formal representation called “selection diagrams” for ex- pressing knowledge about differences and commonalities between pop- ulations of interest and, using this representation, we reduce questions of transportability to symbolic derivations in the do-calculus. This re- duction yields graph-based procedures for deciding, prior to observing any data, whether causal effects in the target population can be in- ferred from experimental findings in the study population. When the answer is affirmative, the procedures identify what experimental and observational findings need be obtained from the two populations, and how they can be combined to ensure bias-free transport. Key words and phrases: Experimental design, generalizability, causal effects, external validity. 1. INTRODUCTION: THREATS VS. most studies are conducted with the intention of ap- ASSUMPTIONS plying the results elsewhere means that we usually deem the target environment sufficiently similar to Science is about generalization, and generalization the study environment to justify the transport of requires that conclusions obtained in the laboratory experimental results or their ramifications. be transported and applied elsewhere, in an envi- Remarkably, the conditions that permit such arXiv:1503.01603v1 [stat.ME] 5 Mar 2015 ronment that differs in many aspects from that of transport have not received systematic formal treat- the laboratory. ment. In statistical practice, problems related to Clearly, if the target environment is arbitrary, combining and generalizing from diverse studies are or drastically different from the study environment handled by methods of meta analysis (Glass (1976); nothing can be transferred and scientific progress Hedges and Olkin (1985); Owen (2009)), or hierar- will come to a standstill. However, the fact that chical models (Gelman and Hill (2007)), in which re- sults of diverse studies are pooled together by stan- Judea Pearl is Professor and Elias Bareinboim is Ph.D. dard statistical procedures (e.g., inverse-variance Candidate, Computer Science Department, University reweighting in meta-analysis, partial pooling in hi- of California, Los Angeles, California 90095-1596, USA erarchical modeling) and rarely make explicit dis- e-mail: [email protected]; [email protected]. tinction between experimental and observational This is an electronic reprint of the original article regimes; performance is evaluated primarily by sim- published by the Institute of Mathematical Statistics in ulation. Statistical Science, 2014, Vol. 29, No. 4, 579–595. This To supplement these methodologies, our paper reprint differs from the original in pagination and provides theoretical guidance in the form of limits on typographic detail. what can be achieved in practice, what problems are 1 2 J. PEARL AND E. BAREINBOIM likely to be encountered when populations differ sig- language within which the notion “environment” (or nificantly from each other, what population differ- “population”) is given precise characterization, and ences can be circumvented by clever design and what differences among environments can be encoded and differences constitute theoretical impediments, pro- analyzed. hibiting generalization by any means whatsoever. The advent of causal diagrams (Wright (1921); On the theoretical front, the standard literature Heise (1975); Davis (1984); Verma and Pearl (1988); on this topic, falling under rubrics such as “exter- Spirtes, Glymour and Scheines (1993); Pearl (1995)) nal validity” (Campbell and Stanley (1963), Man- together with models of interventions (Haavelmo ski (2007)), “heterogeneity” (H¨ofler, Gloster and (1943); Strotz and Wold (1960)) and counterfactu- Hoyer (2010)), “quasi-experiments” (Shadish, Cook als (Neyman (1923); Rubin (1974); Robins (1986); and Campbell (2002), Chapter 3; Adelman (1991)),1 Balke and Pearl (1995)) provides such a language consists primarily of “threats,” namely, explanations and renders the formalization of transportability of what may go wrong when we try to transport re- possible. sults from one study to another while ignoring their Armed with this language, this paper departs differences. Rarely do we find an analysis of “licens- from the tradition of communicating “threats” and ing assumptions,” namely, formal conditions under embarks instead on the task of formulating “licenses which the transport of results across differing envi- to transport,” namely, assumptions that, if they held ronments or populations is licensed from first prin- true, would permit us to transport results across ciples.2 studies. The reasons for this asymmetry are several. First, In addition, the paper uses the inferential ma- threats are safer to cite than assumptions. He chinery of the do-calculus (Pearl (1995); Koller and who cites “threats” appears prudent, cautious and Friedman (2009); Huang and Valtorta (2006); Sh- thoughtful, whereas he who seeks licensing assump- pitser and Pearl (2006)) to derive algorithms for de- tions risks suspicions of attempting to endorse those ciding whether transportability is feasible and how assumptions. experimental and observational findings can be com- Second, assumptions are self-destructive in their bined to yield unbiased estimates of causal effects in honesty. The more explicit the assumption, the more the target population. criticism it invites, for it tends to trigger a richer The paper is organized as follows. In Section 2, space of alternative scenarios in which the assump- we review the foundations of structural equations tion may fail. Researchers prefer therefore to declare modeling (SEM), the question of identifiability and threats in public and make assumptions in private. the do-calculus that emerges from these founda- Third, whereas threats can be communicated in tions. (This section can be skipped by readers fa- plain English, supported by anecdotal pointers to miliar with these concepts and tools.) In Section 3, familiar experiences, assumptions require a formal we motivate the question of transportability through simple examples, and illustrate how the solution de- 1Manski (2007) defines “external validity” as follows: “An pends on the causal story behind the problem. In experiment is said to have “external validity” if the distribu- Section 4, we formally define the notion of trans- tion of outcomes realized by a treatment group is the same as portability and reduce it to a problem of symbolic the distribution of outcome that would be realized in an ac- transformations in do-calculus. In Section 5, we pro- tual program.” Campbell and Stanley (1963), page 5, take a vide a graphical criterion for deciding transporta- slightly broader view: ““External validity” asks the question of generalizability: to what populations, settings, treatment bility and estimating transported causal effects. We variables, and measurement variables can this effect be gen- conclude in Section 6 with brief discussions of re- eralized?” lated problems of external validity, these include sta- 2Hern´an and VanderWeele (2011) studied such conditions tistical transportability, and meta-analysis. in the context of compound treatments, where we seek to predict the effect of one version of a treatment from experi- ments with a different version. Their analysis is a special case 2. PRELIMINARIES: THE LOGICAL of the theory developed in this paper (Petersen (2011)). A FOUNDATIONS OF CAUSAL INFERENCE related application is reported in Robins, Orellana and Rot- nitzky (2008) where a treatment strategy is extrapolated be- The tools presented in this paper were developed tween two biological similar populations under different ob- in the context of nonparametric Structural Equa- servational regimes. tions Models (SEM), which is one among several EXTERNAL VALIDITY: FROM DO-CALCULUS TO TRANSPORTABILITY 3 approaches to causal inference, and goes back to In principle, each query Qi ∈ Q should be “well (Haavelmo (1943); Strotz and Wold (1960)). Other defined,” that is, computable from any fully approaches include, for example, potential-outcomes specified model M compatible with A. (See Def- (Rubin (1974)), Structured Tree Graphs (Robins inition 1 for formal characterization of a model, (1986)), decision analytic (Dawid (2002)), Causal and also Section 2.4 for the problem of identifi- Bayesian Networks (Spirtes, Glymour and Scheines cation in partially specified models.) (2000); Pearl (2000), Chapter 1; Bareinboim, Brito I-3. A set D of experimental or non-experimental and Pearl (2012)), and Settable Systems (White and data, governed by a joint probability distribu- Chalak (2009)). We will first describe the generic tion presumably consistent with A. features common to all such approaches, and then summarize how these features are represented in The outputs are: 3 SEM. O-1. A set A∗ of statements which are the logical 2.1 Causal Models as Inference Engines implications of A, separate from the data at hand. For