1 AAPOR/WAPOR Task Force Report on Quality in Comparative Surveys April, 2021 Chairing Committee: Lars Lyberg, Demoskop, AAPOR T
Total Page：16
File Type：pdf, Size：1020Kb
AAPOR/WAPOR Task Force Report on Quality in Comparative Surveys April, 2021 Chairing Committee: Lars Lyberg, Demoskop, AAPOR Task Force Chair Beth-Ellen Pennell, University of Michigan, WAPOR Task Force Chair Kristen Cibelli Hibben, University of Michigan, Co-Chair Julie de Jong, University of Michigan, Co-Chair Contributors: Dorothée Behr, GESIS – Leibniz Institute for the Social Sciences Jamie Burnett, Kantar Public Rory Fitzgerald, City, University of London Peter Granda, University of Michigan Linda Luz Guerrero, Social Weather Stations Hayk Gyuzalyan, Conflict Management Consulting Tim Johnson, University of Illinois, Chicago Jibum Kim, Sungkyunkwan University, South Korea Zeina Mneimneh, University of Michigan Patrick Moynihan, Pew Research Center Michael Robbins, Princeton University Alisú Schoua-Glusberg, Research Support Services Mandy Sha, www.mandysha.com Tom W. Smith, NORC University of Chicago Ineke Stoop, The Netherlands Irina Tomescu-Dubrow, Institute of Philosophy and Sociology, Polish Academy of Sciences (PAN) and CONSIRT at Ohio State University and PAN Diana Zavala-Rojas, Universitat Pompeu Fabra, Barcelona Elizabeth J. Zechmeister, Vanderbilt University, LAPOP This report was commissioned by the AAPOR and WAPOR Executive Councils as a service to the profession. The report was reviewed and accepted by AAPOR and WAPOR Executive Councils. The opinions expressed in this report are those of the authors and do not necessarily reflect the views of either council. The authors, who retain the copyright to this report, grant AAPOR a non-exclusive perpetual license to the version on the AAPOR website and the right to link to any published versions. 1 This report is dedicated to the memory of Lars Lyberg, who has had a profound and lasting influence on our field. He was a generous collaborator, colleague, and mentor, and a great friend. 2 Table of Contents Abbreviations used in the report 5 Executive Summary 7 Background 7 Priority areas for future research 12 1. Introduction 16 2. Background 19 2.1. History of 3MC surveys 19 2.2 3MC surveys in practice 20 2.3 The fundamental challenges of 3MC surveys 24 3. Quality and comparability in 3MC surveys 32 4. Prevailing operational and design challenges 37 4.1 Organizational structure 37 Introduction and key operational and design challenges 37 Current best practices 38 Recent innovations 39 Suggested future directions 40 4.2 Sampling 40 Introduction and key operational and design challenges 40 Current best practices 41 Recent innovations 47 Suggested future directions 50 4.3 Questionnaire design 51 Introduction and key operational and design challenges 51 Current best practices 54 Recent innovations 55 Suggested future directions 57 4.4 Translation and adaptation 58 Introduction and key operational and design challenges 58 Current best practices 60 Recent innovations 63 Suggested future directions 63 4.5 Questionnaire pretesting 65 Introduction and key operational and design challenges 65 3 Current best practices 66 Recent innovations 67 Suggested future directions 69 4.6 Field implementation 70 Introduction and key operational and design challenges 70 Current best practices 71 Recent innovations 77 Suggested future directions 81 4.7 Documentation in 3MC surveys 83 Introduction and key operational challenges 83 Current best practices 86 Recent innovations 89 Suggested future directions 90 5. The changing survey landscape 94 6. Summary and recommendations 97 Appendix 1 – Task Force Charge 101 Appendix 2 – Table 2 References 106 Appendix 3 – Smith’s 2011 TSE and comparison error figure 110 Appendix 4 – Pennell et al. 2017 TSE framework adapted for 3MC surveys 112 Appendix 5 – Bauer’s random route alternatives 114 True Random Route (TRR) 114 Street Section Sampling (SSS) 115 Appendix 6 – 3MC Survey documentation standards for study-level and variable-level metadata and auxiliary data 117 References 121 4 Abbreviations used in the report 3MC Multicultural, Multinational, and Multiregional AAPOR American Association for Public Opinion Research ADQ Asking different questions ASQ Ask-the-same question ASQT Ask-the-same questions and translating AT Advance translation CAMCES Computer-Assisted Measurement and Coding of Educational Qualifications in Surveys CAPI Computer-assisted Personal Interviews/interviewing CCSG Cross-cultural Survey Guidelines CESSDA Consortium of European Social Science Data Archives CIRF Cognitive Interviewing Reporting Framework CRONOS Cross-National Online Survey CSDI Comparative Survey Design and Implementation CSES Comparative Study of Electoral Systems DDI Data Documentation Initiative DHS Demographic and Health Surveys EES European Election Studies EFTA European Free Trade Association EQLS European Quality of Life Survey ESRA European Survey Research Association ESS European Social Survey EU-LFS EU-Labour Force Survey EU-SILC EU-Statistics on Income and Living Conditions EVS European Values Surveys EWCS European Working Conditions Survey fMoW Functional Map of the World GAP Pew Global Attitudes Project GDPR General Data Protection Regulation GEM Global Entrepreneurship Monitor GGS Generations and Gender Survey HETUS Harmonised European Time Use Surveys IALS International Adult Literacy Survey ICT Information and Communications Technology ISCED International Standard Classification of Education ISCO International Standard Classification of Occupations ISSP International Social Survey Programme LAPOP Latin American Public Opinion Project LSMS World Bank Living Standards Measurement Survey 5 MCA Multiple correspondence analysis NCES National Center for Education Statistics ODNI Office for the Director for National Intelligence OECD Organization for Economic and Co-operation and Development OMB Office of Management and Budget PAPI Paper and Pencil Interviews/interviewing PCA Principal Component Analysis PIAAC Programme for the International Assessment of Adult Competencies PIRLS Progress in International Reading Literacy Study PISA Programme for International Student Assessment PPeS Probability Proportional to Estimated Size QAS Quality Appraisal System QDDT Questionnaire Design and Documentation Tool QUAID Question Understanding Aid SERISS Synergies for Europe’s Research Infrastructure in the Social Sciences SHARE Survey of Health, Aging and Retirement in Europe SQP Survey Quality Predictor SRC Survey Research Center SSHOC Social Sciences and Humanities Open Cloud SSS Street Section Sampling SUSTAIN Sustainable Tailored Integrated Care for Older People in Europe TA Translatability Assessment TIMSS Trends in International Mathematics and Science Study TMT Translation Management Tool TRAPD Translation, Review, Adjudication, Pretest, and Documentation TRR True Random Routes TSE Total Survey Error TTT Train-the-trainer WAPOR World Association for Public Opinion Research WMHS World Mental Health Survey WVS World Values Survey 6 Executive Summary Background Comparative surveys are surveys that study more than one population with the purpose of comparing various characteristics of the populations. The purpose of these types of surveys is to facilitate research of social phenomena across populations, and, frequently, over time. Researchers often refer to comparative surveys that take place in multinational, multiregional, and multicultural contexts as “3MC” surveys (Mneimneh et al., forthcoming).1 To achieve comparability, these surveys need to be carefully designed according to state-of-the-art principles and standards. There are many 3MC surveys conducted within official statistics, and the academic and private sectors. They have become increasingly important to global and regional decision-making as well as theory-building. At the same time these surveys display considerable variation regarding methodological and administrative resources available, organizational infrastructure, awareness of error sources and error structures, level of standardized implementation across populations, as well as user involvement. These circumstances make 3MC surveys vulnerable from a quality perspective. Quality problems present in single-population surveys are therefore magnified in 3MC surveys. In addition, there are quality problems specific to 3MC surveys such as translation processes. The wealth of output from such surveys is usually not accompanied by a corresponding interest in informing researchers, decision-makers, and other users about quality shortcomings. This can lead to understated margins of error and estimates that therefore appear more precise than they actually are. There are also cases where researchers are informed about quality shortcomings but opt to ignore those in their research reports. There are of course many possible explanations for this state of affairs. One is that 3MC surveys are very expensive and the formidable planning and implementation leaves relatively little room for a comprehensive treatment of quality issues. Another explanation is that the survey-taking cultures among survey professionals vary considerably across nations as manifested by varying degrees of methodological capacity, risk assessment, and willingness to adhere to specifications that are not normally applied. The literature on data quality in 3MC surveys is scarce compared to the substantive literature. There are exceptions, though, including the Cross-Cultural Survey Guidelines developed by the University of Michigan and members of the International Workshop on Comparative Survey Design and Implementation (CSDI). AAPOR has created a cross-cultural and multilingual research