University of Mannheim Representativeness and Response Quality of Survey Data Inaugural dissertation submitted in partial fulfillment of the requirements for the degree Doctor of Social Sciences in the Graduate School of Economic and Social Sciences at the University of Mannheim Submitted by Carina Cornesse Mannheim, 10/24/2017 Dekan Prof. Dr. Michael Diehl Erstbetreuerin Prof. Annelies Blom, Ph.D. Zweitbetreuer Prof. Joseph Sakshaug, Ph.D. Erstgutachter Prof. Joseph Sakshaug, Ph.D. Zweitgutachter Prof. Dr. Christof Wolf Tag der Disputation 24.01.2018 Acknowledgements There are a number of people who have contributed to this dissertation in different ways and whose help, advice, or general support I would like to acknowledge. First of all, I would like to thank Annelies Blom for her excellent supervision of my thesis and for generally being a great mentor. I am very grateful to her for always providing me with outstanding opportunities and support. I would also like to thank Joe Sakshaug for being a great second dissertation advisor. I am especially grateful for the valuable spot-on feedback that he provided on my early paper drafts. In addition, I would like to thank Christof Wolf for reviewing this thesis and agreeing to chair my dissertation committee. I greatly appreciate the time and effort that my supervisors and reviewers give. Furthermore, I would like to thank my co-authors: I am grateful to Michael Bosnjak, Tobias Enderle, and Annelies Blom for providing valuable input to our papers. I appreciate and highly value all of my co-authors’ contributions to my dissertation. I benefited greatly from their ideas, patience, and expertise. In general, I would like to express my gratitude to the SFB 884 “Political Economy of Reforms” and GESIS – Leibniz Institute for the Social Sciences for providing an excellent and motivating working environment. I am particularly grateful to my colleagues at the SFB and GESIS. I would especially like to thank Daniela Ackermann-Piek and Suse Helmschrott. I greatly enjoyed working, discussing, laughing, and sometimes moaning with them. I would also like to thank Christian Bruch, Barbara Felderer, Franziska Gebhard, Jessica Herzing, Marina Jesse, Uli Krieger, Rita Maklakova, and Dayana Sieger. I highly appreciate the valuable and inspiring discussions we had at work and I am grateful for their moral support. At GESIS, I would like to thank all former and current members of the GESIS Panel team. Finally, I am grateful to my family, especially my boyfriend Sebastian Marx and my parents Alfred and Elvira Cornesse for their encouragement, support, and patience. 3 Table of contents General introduction ............................................................................................................................... 9 1. The relevance of survey data quality .............................................................................................. 9 2. The Total Survey Error (TSE) framework ....................................................................................... 11 3. The concept of survey representativeness ................................................................................... 13 4. The concept of survey response quality ........................................................................................ 15 5. Summary of dissertation papers ................................................................................................... 16 Literature ........................................................................................................................................... 21 Is there an association between survey characteristics and representativeness? A meta-analysis ..... 27 Abstract ............................................................................................................................................. 27 Keywords ........................................................................................................................................... 27 Acknowledgements ........................................................................................................................... 27 1. Background and aims .................................................................................................................... 29 2. Survey representativeness concepts ............................................................................................. 30 3. Conceptual development of research question and expectations ............................................... 32 3.1 Probability surveys versus nonprobability surveys ................................................................. 32 3.2 Response rates ........................................................................................................................ 34 3.3 Mixed-mode surveys versus single-mode surveys .................................................................. 35 3.4 Web surveys versus other single-mode surveys ..................................................................... 36 3.5 Auxiliary variables .................................................................................................................... 37 4.Method ........................................................................................................................................... 38 4.1 Literature search, study identification and data extraction .................................................... 38 4.2 Meta-analytic procedure ......................................................................................................... 39 4.3 Effect size measures ................................................................................................................ 40 5.Results ............................................................................................................................................ 42 5.1 General findings....................................................................................................................... 43 5.2 Moderator analyses ................................................................................................................. 43 6. Summary and conclusion .............................................................................................................. 49 Literature ........................................................................................................................................... 52 Online Appendix A: R-Indicator articles in the meta-analysis ........................................................... 60 Online Appendix B: Benchmark comparison articles in the meta-analysis ....................................... 63 Is it in the method? Testing five measures of survey representativeness ............................................ 71 Abstract ............................................................................................................................................. 71 Keywords ........................................................................................................................................... 71 4 Acknowledgements ........................................................................................................................... 72 1. Introduction ................................................................................................................................... 73 2. Measures of survey representativeness ....................................................................................... 75 2.1 Representativeness measures at the aggregate level ............................................................. 79 2.2 Representativeness measures at the variable level ................................................................ 83 2.3 Representativeness measures at the category level ............................................................... 85 3. Data ............................................................................................................................................... 86 3.1 The survey designs of the GIP 2012, the GESIS Panel, and the GIP 2014: differences and similarities ..................................................................................................................................... 86 3.2 Auxiliary data for modelling representativeness .................................................................... 89 4. Results ........................................................................................................................................... 92 4.1 Response rates ........................................................................................................................ 93 4.2 R-Indicators ............................................................................................................................. 94 4.3 Fraction of Missing Information (FMI) .................................................................................... 95 4.4 Subgroup response rates ......................................................................................................... 97 4.5 Benchmark comparison ......................................................................................................... 101 5. Discussion .................................................................................................................................... 103 Literature ........................................................................................................................................