A FRAMEWORK FOR THE META-ANALYSIS OF SURVEY DATA by Karla Michelle Phillips Cooper Fox A thesis submitted to the Department of Mathematics and Statistics In conformity with the requirements for the degree of Doctor of Philosophy Queen’s University Kingston, Ontario, Canada (December, 2011) Copyright ©Karla Michelle Phillips Cooper Fox, 2011 Abstract The research outlined in this thesis covers various different statistical issues relating to meta- analysis of survey data. These issues include the creation of an original comprehensive methodological framework for combining survey data, a comparison of this framework with the traditional one proposed by Cochran for the combination of experiments, a proposal for a new weighting method that takes into account the differences in variability due to the sampling plan, an examination of the convergence of meta-analytic estimators, and a discussion on the numerous implicit assumptions researchers make when they are using meta-analysis methods with survey data along with guidelines for completing and reporting reviews when the data come from surveys. ii Acknowledgements It is a pleasure to thank those who made this thesis possible. First, I wish to express my gratitude to my supervisor, Dr. David Steinsaltz who was instrumental in starting this research; he was helpful, offered significant assistance, support and guidance throughout. Gratitude is also due to the members of the supervisory committee, and staff at Queens University, in particular Jennifer Read for her timely help in all of my queries. Additionally, I would like to thank Statistics Canada for supporting this research. I am indebted to my many colleagues at Statistics Canada who assisted me and in particular I would like to thank Georgia Roberts, Sander Post and Ioana Schiopu-Kratina for their insights and consultations. I owe my deepest gratitude to three extraordinary women, Isobel Barnes, Gaye Taylor and Maxine Morrison, who helped in many different ways throughout this process and without whose support, editing, and counsel this thesis would not be possible. Finally, I would also like to express my deepest love and gratitude to my husband Michael Fox for his understanding, patience and endless belief in me through the long duration of my studies. For my father Jack Phillips Cooper iii Statement of Originality I hereby certify that all of the work described within this thesis is the original work of the author. Any published (or unpublished) ideas and/or techniques from the work of others are fully acknowledged in accordance with the standard referencing practices. (Karla Michelle Phillips Cooper Fox) (December, 2011) iv Table of Contents Abstract............................................................................................................................................ii Acknowledgements.........................................................................................................................iii Statement of Originality..................................................................................................................iv Table of Contents.............................................................................................................................v List of Figures...............................................................................................................................viii Chapter 1 Introduction.....................................................................................................................1 1.1 Existing Meta-Analysis Techniques ......................................................................................6 1.1.1 Hypothesis Tests .............................................................................................................7 1.1.2 Fixed Effect Models......................................................................................................10 1.1.3 Random Effect Models .................................................................................................14 1.1.4 Regression Methods......................................................................................................19 1.1.5 Likelihood Based Methods ...........................................................................................22 1.2 Probability Sampling ...........................................................................................................24 1.3 Superpopulations..................................................................................................................25 1.4 Combining Surveys..............................................................................................................26 Chapter 2 Design and Model Based Methods in Sampling ...........................................................38 2.1 Design Based Sampling.......................................................................................................39 2.1.1 Simple Random Sampling .....................................................................................43 2.1.2 Horvitz-Thompson Estimation...............................................................................46 2.1.3 Stratified Sampling ................................................................................................48 2.1.4 Cluster Sampling....................................................................................................50 2.1.5 Two-phase Sampling .............................................................................................55 2.1.6 Complex Sampling.......................................................................................................57 2.2 Models and the Superpopulation....................................................................................60 2.3 Inference ........................................................................................................................61 Definitions .................................................................................................................................61 2.4 Estimation ......................................................................................................................67 Ignorability and Informativeness ...............................................................................................73 2.5 Design Effect .................................................................................................................76 2.6 Mathematical Examples.................................................................................................79 Some Theory of Sampling: W. Deming.....................................................................................79 v Answers to Criticisms of Using Design-Based Methods for Inference: D. Binder ...................81 Chapter 3 The Framework .............................................................................................................85 3.1 The Framework for the Meta-Analysis of Survey Data.................................................85 Experiments as Generated in a Superpopulation Survey Context .............................................86 Case 1: Combining Finite Population Estimates....................................................................89 Variances ...................................................................................................................................99 Case 2: Combining Finite Parameters from more than one Finite Population.....................104 Case 3: Combining Finite Parameters from Multiple Superpopulations .............................108 3.2 Asymptotic Techniques and Definitions......................................................................112 3.3 Convergence of Combined Estimates under the Sampling Framework ......................116 3.4 Testing Heterogeneity..................................................................................................125 3.5 Adjusted Weights.........................................................................................................128 Survey Design Adjusted Weights ............................................................................................129 Chapter 4 Simulations..................................................................................................................132 Notation ...................................................................................................................................132 Simulations ..............................................................................................................................136 4.1 Cases 1a and 1b: Combining several estimates from different surveys of the same finite population. ...............................................................................................................................138 4.1.1 Case 1a: Mixing Identical Studies .......................................................................139 Simple Random Sampling .......................................................................................................140 Stratified Sampling ..................................................................................................................144 Cluster Sampling......................................................................................................................146 4.1.2 Case 1a Mixing Surveys of Different Types........................................................148 Ignoring the Design Weights ...................................................................................................151