Evaluation Report
Total Page：16
File Type：pdf, Size：1020Kb
Perception Survey on the Quality of Life in European Cities 2019 Evaluation Report Contract n° 2017CE160AT133 Prepared by: Ipsos Date: 9 March 2020 Regional and Urban Policy EUROPEAN COMMISSION Produced by on behalf of the Directorate-General for Regional and Urban Policy Unit B1 - Policy Development and Economic Analysis E-mail: [email protected] European Commission B-1000 Brussels EUROPEAN COMMISSION Perception Survey on the Quality of Life in European Cities 2019 Evaluation Report Directorate-General for Regional and Urban Policy 2020 3 EUROPEAN COMMISSION Directorate-General for Regional and Urban Policy 2020 5 TABLE OF CONTENTS 1 INTRODUCTION ..................................................................................................... 5 2 PROJECT OVERVIEW .............................................................................................. 6 2.1 Timing ............................................................. Error! Bookmark not defined. 2.2 Sample design.............................................................................................. 6 2.2.1 Sample size and associated margin of error ........................................ 6 2.2.2 Sample methodology ........................................................................ 7 2.3 Questionnaire design ................................................................................... 14 2.3.1 Screening questions ....................................................................... 14 2.3.2 Socio-demographic background questions ......................................... 16 2.3.3 Weighting questions ....................................................................... 16 2.3.4 Changes related to GDPR compliance ............................................... 17 2.4 Pilot testing................................................................................................ 18 2.5 Main fieldwork ............................................................................................ 18 2.5.1 Timing .......................................................................................... 18 2.5.2 Fieldwork follow-up ........................................................................ 18 2.6 Reporting and data delivery ......................................................................... 19 2.6.1 Pre-fieldwork reporting ......................... Error! Bookmark not defined. 2.6.2 Data delivery ................................................................................ 19 3 WEIGHTING ........................................................................................................ 20 3.1 Weighting procedure ................................................................................... 20 3.1.1 Post-stratification & design weighting ............................................... 20 3.1.2 Weight trimming ............................................................................ 22 3.2 Weighting benchmarks ................................................................................ 23 3.2.1 Age and gender ............................................................................. 23 3.2.2 Phone ownership ........................................................................... 24 3.3 Design effects and weighting efficiency per city .............................................. 26 4 SAMPLE PERFORMANCE ANALYSIS ......................................................................... 28 4.1 Target population versus achieved distribution ............................................... 28 4.1.1 Age .............................................................................................. 28 4.1.2 Gender ......................................................................................... 33 4.1.3 Phone ownership ........................................................................... 35 4.1.4 Eligibility ....................................................................................... 37 5 FIELDWORK PERFORMANCE ANALYSIS ................................................................... 40 5.1 Interview validation .................................................................................... 40 5.2 Interview breakoffs ..................................................................................... 41 5.3 Item non-response ..................................................................................... 41 5.4 Response rates ........................................................................................... 43 6 DATA COMPARISON 2019-2015 ............................................................................. 46 7 RECOMMENDATIONS FOR FUTURE WAVES .............................................................. 47 ANNEX 1. FINAL QUESTIONNAIRE ................................................................................. 49 1 Introduction This evaluation report is one of two parts of the final report for the 2019 Perception Survey on the Quality of Life in European Cities. It presents an overview of the design, preparation and execution of the Perception Survey. It also discusses the survey’s performance in terms of sampling, fieldwork quality and accuracy of the collected data. Finally, this report also lays out some recommendations for possible changes to the survey design that could improve the performance of future waves of the Perception Survey. This evaluation report is accompanied by a technical report, which forms the second part of the final report. The technical report lists, per city, the most important sample performance data (amount of sample used, eligibility rate, refusals, response rate, average interview length, etc.) 2 Project overview This chapter gives a concise overview of the different steps of the 2019 Perception Survey, from the questionnaire design until the final data deliveries. 2.1 Sample design The Perception Survey targets citizens of all (greater) cities within the scope of the survey – covering a total of 83 cities. The target population includes all people aged 15 and above, who satisfy the requirements outlined below: 1. Being a resident of the city surveyed; 2. Having sufficient command of (one of) the respective national/regional language(s) or English, which allows them to comfortably answer the questionnaire; 3. Living in a private household, which means that the target population will exclude prisoners, residents of retirement homes, etc. who are difficult to reach via a telephone survey. Regarding the first requirement, the scope is technically defined for each city in terms of a set of Local Area Units (LAUs) that together comprise the area of the city under scope. The residence of a given respondent in one of these LAUs determines their eligibility for the survey. The list of LAUs in scope per city is added to this Evaluation Report as Annex 2. Regarding the second requirement, the language command was assessed by the interviewer at the start of the survey. In case it was clear that a respondent is not able to answer questions in one of the official languages, they were offered to conduct the interview in English. Regarding the third requirement, the survey in practice targeted all residents aged 15+ with private access to a telephone, which is de facto confirmed by a given respondent being reachable by phone during the fieldwork. 2.1.1 Sample size and associated margin of error The target sample size was 700 complete interviews in each city surveyed. This means that interviews were gathered from 58 100 respondents in total, all of which are citizens who are resident in one of the (greater) cities under scope The following chart depicts how the margin of error associated with survey estimates can vary as a function of sample size, assuming a confidence level of 95%. Quality of Life in European Cities Survey 2019 ±9.8% ±6.9% ±5.7% ±4.9% ±4.4% ±4.0% Marginof Error ±3.7% ±3.5% ±3.3% ±3.1% ±3.0% ±2.8% ±2.7% ±2.6% ±2.5% 100 200 300 400 500 600 700 800 900 1000 1100 1200 1300 1400 1500 Required Sample Size 2.1.2 Sample methodology Telephone samples require a specific design in order to cover the entire target population and to reduce a) potential coverage bias and b) non-response bias. Some aspects are country-specific, such as prefixes, operators, overall telephone penetration, penetration of mobile phones and penetration of mobile only. As a growing share of the population is becoming “mobile only” (i.e. persons who only have access to a mobile phone), the optimal composition of telephone samples should take into account the incidence of households that are reachable only via mobile numbers. Each telephone mode (fixed line or mobile) also covers a specific profile with parameters such as age and urbanization degree. According to the 2017 Eurobarometer on E-communications, omitting “mobile only” persons implies 37% of the EU households are not included in the sample frame.1 In order to ensure maximum population coverage resulting in a representative sample, a mixed (or “dual frame”) approach was taken for this Perception Survey, which takes into account the respective distributions of persons who only have access to a mobile phone (i.e. “mobile only”), persons who only have access to a fixed line phone (i.e. “fixed only”) and persons who have access to both mobile and fixed line phones (i.e. “mixed”). Based on these data, the necessary distribution of mobile phone and landline sample units needed in the sample frame is calculated. By utilizing two separate, overlapping sample frames to interview a population