Master thesis Department of Statistics Masteruppsats, Statistiska institutionen Quality Assurance and Quality Control in Multinational and Multicultural Surveys Gelaye Worku HaileMichael Masteruppsats 30 högskolepoäng, vt 2013 Supervisor: Lars Lyberg Quality Assurance and Quality Control in Multinational and Multicultural Surveys by Gelaye worku HaileMichael Submitted in partial fulfillment of the requirements For the Master in Survey Methodology and Official Statistics Supervisor: Prof. Lars Lyberg Faculty of Social Science, Department of Statistics Stockholm University, Department of Statistics Stockholm University May 2013 Abstract Maintaining a high quality survey is difficult for any kind of survey and it is even more complicated and difficult to perform quality assurance and quality control in multicultural and multinational surveys. This thesis will try to determine a minimum required quality assurance (QA) and quality control (QC) framework for multicultural and multinational surveys. In order to achieve this goal we will start explaining what makes multicultural and multinational surveys so special, followed by some examples of such surveys (3M). Then the survey lifecycle for high quality surveys from the currently available guidelines for best practice in cross-cultural surveys (CCSG) is then introduced. Furthermore we investigate the efforts made by some selected surveys to acquire survey quality. Before concluding the thesis we will try to state what lessons can be learned from the selected surveys concerning QA and QC by going through what is in place and what is not. Finally following the guidelines by the Comparative Survey Design and Implementation (CSDI) group and the survey quality experience from the selected surveys, we try to define a basic minimum QA and QC framework needed for a quality multicultural and/or multinational survey. Key words: 3M surveys; Survey quality; translation; sampling design; pretesting; data collection; quality assurance; quality control. Acknowledgment I would like to express my greatest gratitude to the people who have helped and supported me. I am grateful to my advisor Prof. Lars Lyberg for the continuous support, his patience, motivation, and immense knowledge. A special thank goes to my friends, Fredrik Holmér, Hamelmal Mesganaw, Meseret Haile and more others , for all their help, motivation and appreciation. I also wish to thank my family for their unlimited support, for inspiring me and encourage me , without them I would not have make it this far. And finally to God who made all the things possible. Contents Introduction ................................................................................................................................................... 1 1. What is so special about multinational and multicultural surveys? ...................................................... 3 2. Examples of Multinational, Multicultural and Multiregional (3M) surveys ......................................... 8 2.1 Global surveys .............................................................................................................................. 8 2.2 Regional surveys ........................................................................................................................... 9 3. Survey lifecycle for a high quality survey .......................................................................................... 10 4. Quality assurance and quality control in a 3M setting ........................................................................ 18 5. Selected Surveys ................................................................................................................................. 23 5.1. European Social Survey (ESS) ................................................................................................... 24 5.2. Trends in International Mathematics and Science Study (TIMSS)............................................. 37 5.3. World Values Survey (WVS) ..................................................................................................... 43 5.4. Program for the International Assessment of Adult Competencies (PIAAC) ............................ 45 5.5. International Adult Literacy Survey (IALS) ............................................................................... 49 5.6. Survey of Health, Ageing and Retirement in Europe (SHARE) ................................................. 53 5.7. International Social Survey Program (ISSP)............................................................................... 57 5.8. European Working Conditions Survey (EWCS)......................................................................... 61 5.9. Comparative Study of Electoral System (CSES) ........................................................................ 68 5.10. Gallup World Poll (GWP) ........................................................................................................... 71 5.11. Eurobarometer (EB) .................................................................................................................... 73 6. The QC and QA system in selected surveys ....................................................................................... 76 7. A basic QA and QC framework .......................................................................................................... 80 8. Conclusion .......................................................................................................................................... 83 References .................................................................................................................................................. 85 Appendix ..................................................................................................................................................... 92 Introduction Multicultural, multinational, and multiregional surveys (3M) started more than 40 years ago and since then they have grown and become very useful. But despite this development there is still not so much literature on how to obtain high quality comparable data in such surveys. And also the more languages, cultures and nations that are included in the surveys, the more complicated it becomes to design the survey instruments and to actually implement the survey and control the survey quality (Pennell et al., 2010). The main goal of this thesis is to contribute to the basic quality assurance (QA) and quality control (QC) framework for multinational and multicultural surveys (cross-national surveys). Throughout this thesis the term “cross- national survey” will have the definition by Lynn et al.(2006), which is “all types of surveys where efforts are made to achieve comparability across countries”. The steps that are assumed to lead to the goal of this thesis are to first state what make cross-national surveys so special referring to what currently available guidelines (the CSDI guidelines) and subject matter expertise have to say about it for example Kish (1994), Lynn, Japec, and Lyberg (2006), Harknesset et al. (2010), Smith (2010), and Tortora et al.(2010). Then we follow up with some examples of global and regional 3M surveys. After that the survey lifecycle for 3M surveys is introduced. Some important references studied included are, Smith (2010), Pennell et al. (2010), Harkness et al. (2010), Lyberg and Biemer (2003), Lyberg et al. (2006), Juran and Gryna (1980), Lyberg and Stukel (2010), Lyberg and Biemer (2008), Kreuter et al. (2010), Couper and Lyberg, (2005), Couper (1998), and Vehovar (2007). After stating the QA and QC in 3M settings using available 3M books, papers, and quality standards it is time to study the efforts made by a few selected surveys to achieve comparative survey quality and to see what is in place and what is not and what can be complemented or criticized based on the survey lifecycle. Finally we conclude by proposing a basic QA and QC framework for multinational and multicultural surveys based on Cross-Cultural Survey guidelines, 3M books and papers, experiences from relatively successful cross-national surveys and various quality standards. 1 In general in section (1) we try to see what makes multinational and multicultural surveys special. Then in section (2) type of 3M surveys with example will be stated. Section (3) will be all about survey lifecycle for a high quality surveys. In section (4) quality assurance and quality control in 3M surveys will be discuss. Following that section (5) is about the selected surveys. In section (6) we will talk about the QA and QC system in the selected surveys. Then finally section (7) will be about the basic QA and QC needs followed by conclusion in section (8). 2 1. What is so special about multinational and multicultural surveys? A survey is called multinational /multicultural, when it concerns comparisons between different nationalities/ cultures/populations within- country or cross-country. What makes it special is the presence of different nationalities, different languages, different cultures, in a multipopulation setting. A multipopulation survey as Kish, (1994) described it can include: Periodic surveys, multidomain designs, multinational survey designs, cumulated and combined samples, and controlled observations. As Kish explains it, the classical single population survey sampling need to be extended to include multiple populations, which is very difficult to design and control. Also a lot of effort is needed to achieve comparability across or within-country.