The European Social Survey1: One Survey in Two Dozen Countries Ineke Stoop2, Roger Jowell3, Peter Mohler4 Paper to be presented at the International Conference on Improving Surveys Copenhagen, 25-28 August 2002 Summary In September 2002 the fieldwork of the first round of the new European Social Survey (ESS) will start. This sur- vey is jointly funded by the European Commission, the European Science Foundation and the National Science Foundations within 24 European countries. The study will focus on changes in attitudes, values and behavioural patterns in the context of a changing Europe. It aims at achieving the highest standards of cross-national survey research, attempting to match the quality standards of the best national surveys. With the help of experts from different fields, a Central Coordinating Team has developed the survey design, the questionnaire and strict guidelines on issues such as random sampling, translation, response rates and fieldwork documentation. In each participating country a national coordinator has been appointed and a survey organisation that will conduct the fieldwork. The first half of 2002 has been largely devoted to the construction of the questionnaire and the meth- odological preparation of the survey ensuring high quality and optimal comparability despite local differences. The paper will present the methodological issues that will have come up during preparation of the ESS. 1 Introduction The European Social Survey is a new, conceptually well-anchored and methodologically rigorous survey that aims to pioneer and ‘prove’ a standard of methodology for cross-national attitude surveys that only the best na- tional studies usually aspire to. The preparation of the survey has involved a wide range of international experts on methodological and substantive issues. Funding by the EU 5th Framework Programme and the European Science Foundation covers the preparation, co-ordination, consultation and the exchange of information. The number of European countries that decided to participate in this new venture has far exceeded the original ex- pectations and now extends to twenty-four. National funders have committed themselves to following a centrally designed specification for the fieldwork and have appointed survey organisations that they believe can achieve the required exacting standards. The study will subscribe to the Declaration on Ethics of the International Statistical Institute, to which all national teams will also be asked to adhere (see Jowell, 1986). While all teams are committed to the highest quality standards for the ESS, these standards are, of course, by no means easy to achieve. Obviously there is never enough money and never enough time, even for a well-founded project such as the ESS. More important than monetary and time constraints, however, is the fact that survey quality is a multi-faceted phenomenon (see, for instance, Lyberg, 2001; Lyberg et al. 2001; Fellegi, 2001). Various quality criteria matter, such as relevance or content, accuracy, timeliness, accessibility, interpretability and coherence. But the problem with multi-faceted criteria such as these is that they always involve a degree of trade-off. This paper discusses these quality criteria and their implementation in the European Social Survey with particular reference to such trade-offs. In particular, the need for optimal cross-national comparability - discussed here under the heading of coherence - is often in direct conflict with the need to adhere closely to the other criteria (Jowell, 1998). 2 Content The ESS intends to measure changing social attitudes and values in (now) 24 European nations. Around one half of the hour-long questionnaire is a core element comprising key repeat questions to measure change and persis- tence in a range of social and demographic characteristics, attitudes and behaviour patterns. This core contains questions on occupation and social structure, social exclusion, religious affiliation and identity, ethnic and na- tional identity, political trust, party affiliation, multilevel governance and voting behaviour, media consumption and value orientations. The other half of the questionnaire – the rotating element – consists of two topic-specific modules per round to measure particular academic and policy concerns and debates that require examination in depth. These modules are selected via an international competition. In 2002, the selected modules are on ‘citi- zenship, involvement and democracy', and ‘immigration’. In addition to the hour-long face-to-face interview questionnaire, a short self-completion questionnaire (though in some countries it will be an extension of the face- to-face interview) will provide room for a scale on ‘basic human values’, plus a number of methodological test questions designed to quantify the reliability and validity of certain measures in the interview. 1 Information on all aspects of the survey is available at: www.europeansocialsurvey.org. This site will be regularly updated. 2 Social and Cultural Planning Office, The Hague, The Netherlands, [email protected], www.scp.nl 3 National Centre for Social Research, London, UK, 4 Zentrum für Umfragen, Methoden und Analysen, Mannheim, Germany 1 There were, of course, substantial trade-offs involved in constructing a core 30-minute questionnaire that is de- signed to measure changes in attitudes, norms, values and social structure over time. Considerations such as sustainability, scope and how to deal with diversity continually arose. Such questions arise in national surveys too, where certain topics or phraseologies may be more relevant for certain parts of the population than for oth- ers, but they are greatly magnified when contemplating a diverse 24-nation survey. Can the same question on, say, social exclusion be asked of a small farmer in a relatively poor country and of a stockbroker in a rich coun- try? Do the same questions about confidence in multi-level governance have the same meaning to inhabitants of EU countries as to people from Switzerland, Hungary and Turkey? The principle is, of course, all too clear. The task of our survey measurements is to discover and calibrate cross- cultural and cross-national differences in people’s responses, and to achieve that we must try to keep the stimu- lus of the questions as constant as possible between respondents. The problem, however, is that there is no easy way of guaranteeing such equivalence of meaning, especially in cross-national surveys where between-country variance is large. But the word “especially” in that sentence is important, because – as we have noted - these problems are by no means confined to cross-national surveys. Almost all surveys are, to varying degrees, cross- cultural; cross-national surveys tend merely to be more so. The conflict arises acutely when the ‘best’ cross-national measure nationally for a particular concept seems to be different from the ‘best’ national measure of the same concept. We are not here referring to the added problem of lexical equivalence between different languages, just to the problem of different cultural constructions of the same basic concept. Suffice it to say that our approach in the ESS is, wherever possible, to rely in such circum- stances on the best cross-national measure we can come up with, for fear otherwise of creating measurement anarchy. Another variation, both between countries and over time, that we will be trying to mitigate within the ESS, is the influence of context on responses. Part of the ESS data set will thus consist of an ‘event data bank’ which briefly documents the major political, social and economic factors just before and during the fieldwork period that are likely to have a substantial bearing on a particular country’s - or a group of countries’ - response patterns (see also section 5.3). 3 Accuracy The accuracy of surveys is generally affected by: • coverage errors, which occur because each member of the target population does not have a known, non- zero chance of inclusion in the sample; • sampling errors, which occur because only a subset of the target population is selected; • measurement errors; which occur because the data collection mechanisms (such as the questionnaires, inter- viewers or coders) introduce faulty answers; • nonresponse errors, which occur because respondents and non-respondents turn out to have different char- acteristics that relate to the survey’s purpose. 3.1 Population and coverage errors The ESS aims to be representative of the residential population of each participating nation aged 15 years and above (with no upper age limit), regardless of their nationality, citizenship or legal status. A ‘resident’ is defined as anyone who has been living in the country for at least a year and who has no immediate concrete plans to return to his or her country of origin. Although, of course, the population of a country also includes the ‘home- less’ and people living in institutions (such as hospitals and prisons), these groups will be excluded from the target population, at least in Round 1 of the ESS. In any country where a minority language is spoken as a first language by at least 5% of the resident population, the questionnaire will be translated into that language too and appropriate interviewers will be deployed to administer the interviews. Given the small sizes of the groups above, the costs of interviewing them did not seem justified by their expected yield. Nonetheless, their exclusion introduces coverage errors in a survey designed to represent the resident populations of all participating countries. And not only are these errors greater in some countries than in others, but they are also often difficult to quantify. 3.2 Sampling and sampling errors Strict probability methods (systematic random sampling) will be deployed at every stage in all countries, such that the relative selection probabilities of every sample member will be known and recorded in the data set. Nei- ther quota sampling nor any from of substitution of non-responding households or individuals (whether ‘refus- als’ or ‘non-contacts’) will be allowed at any stage.