Combining Surveys From A Meta-analysis Perspective Sally C. Morton The RAND Corporation, Statistics Group 1700 Main Street Santa Monica, CA 90401 USA [email protected] 1. Introduction Several issues faced in combining surveys, specifically dealing with survey heterogeneity, prospectively planning data collection, and pooling data across surveys, may benefit from insights gained from meta-analysis, or “the practice of using statistical methods to combine the outcomes of a series of different experiments or investigations” (Laird and Mosteller 1990, p. 5). The goal of this talk is to draw such parallels. I will focus on study-level meta-analysis, in which study effect sizes (summary statistics) are combined. As an extended example, I will discuss estimating the risk of gastrointestinal (GI) complications in the general population, using data from a meta-analysis of nonsteroidal antiinflammatory drug (NSAID) side effects. Meta-analysis has four steps: identify all relevant studies; assess study quality; deal with study heterogeneity; and summarize the results. Many references describe these steps and how to assess their correctness (Cook et al. 1995). The Cochrane Collaboration (www.cochrane.org) is a leader in providing techniques for comprehensive, unbiased, and reproducible searches that locate all the evidence, including unpublished studies to avoid publication bias. Assessing study quality involves considering both the strength of the study design and its execution; methods have primarily focused on randomized controlled trials (Moher et al. 1996). 2. Dealing With Survey and Study Heterogeneity I will discuss heterogeneity and illustrate the benefit of design variation (RCT, case- control, cohort) in the NSAIDs meta-analysis studies in terms of expanding the generalizability of the GI complication risk estimate. Kish (1994) discusses the effect of heterogeneity on combining surveys and emphasizes “deliberate” survey design, “not the mere post hoc or ad hoc utilization of survey results” (p. 1). He stresses standardization of survey elements (definitions, methods, measurements, models), but suggests that sampling designs and sizes may vary to utilize resources efficiently. In meta-analysis, “heterogeneity characterizes the situation in which differences in study outcomes are not readily accounted for by sampling variation” (Colditz et al. 1995); heterogeneity is assessed from substantive and statistical perspectives. A random effects model may be used to incorporate between-study variation (Sec. 3 below). Ideally, heterogeneity is an opportunity to understand study inconsistencies, for example, different treatment effects in subpopulations, perhaps via meta-regression. Rubin (1992) argues that study effect sizes should be modeled as a response surface varying along scientifically important factors, for example population or setting characteristics, and design variables. New studies should be proposed and performed to fill in poorly estimated areas. A related idea is prospective meta-analysis, in which studies are designed jointly so they may be combined later via meta-analysis. 3. Combining Data Across Surveys and Across Studies Kish (1998), in the context of constructing an average birthrate for a continent using separate country birthrates, proposes three options for pooling multinational samples that are directly comparable to the three main meta-analytic models for combining study effect sizes: equal, fixed and random effects. Kish’s population weights approach is akin to equal effects; his suggestion to give each country’s estimate an equal weight is fixed effects; and his use of country weights to optimize least-square loss is in the spirit of random effects. In meta-analysis, the latter two models are the most common (Laird and Mosteller 1990). The equal effects ˆ population proportion estimate Pe treats all subjects as independent and of equal importance, ˆ while the fixed and random effects estimates Pw are weighted averages of the study proportions: K n k K ∑∑xik ∑ w k pk == = (1) Pˆ = k 1i 1 and Pˆ = k 1 . e K w K ∑ n k ∑ w k k =1 k =1 = In (1), xik 1 if patient i (i=1,…, n k ) from study k (k=1,…,K) has the outcome (a GI complication), and equals 0 if not; and the proportion of patients in study k with GI ˆ = complications is pk . The fixed effects estimate Pw treats all studies equally ( w k 1for all k). The random effects approach sets w k equal to the inverse of the sum of the within-study and between-study variances. Meta-analysis has proved a useful technique in combining information across studies. Preliminary evidence suggests that statisticians should explore its application in the context of combining surveys. REFERENCES Colditz G. A., Burdick E. and Mosteller F. (1995). Heterogeneity in Meta-analysis of Data from Epidemiologic Studies: A Commentary. American J. of Epidemiology 142, 371-382. Cook D. J., Sackett D. L. and Spitzer W. O. (1995). Methodologic Guidelines for Systematic Reviews of Randomized Control Trials. J. of Clinical Epidemiology 48, 167-171. Kish L. (1994). Multipopulation Survey Designs: Five Types with Seven Shared Aspects. International Statistical Review 62, 167-186. Kish, L. (1998). Combining Multipopulation Statistics. Institute for Social Research, University of Michigan, Ann Arbor, MI. Laird N. M. and Mosteller F. (1990). Some Statistical Methods for Combining Experimental Results. International J. of Technology Assessment 6, 5-30. Moher D., Jadad A. R. and Tugwell P. (1996). Assessing the Quality of Randomized Controlled Trials. International J. of Technology Assessment 12, 195-208. Rubin D. B. (1992). Meta-analysis: Literature Synthesis or Effect-Size Surface Estimation? J. of Educational Statistics 17, 363-374. RÉSUMÉ Les questions posées par le groupement des données d’enquêtes à sondage – en particulier la question d’hétérégénéité, la planification prospective des enquêtes de sondage, et les méthodes pour grouper les données de plusieurs enquêtes – peuvent bénéficier des connaissances gagnées dans la “meta-analysis”, dans laquelle les resultats de différentes études sont groupées. Le but de cette présentation est de démontrer ce parralléle..