ISSN 1977-0375 Methodologies and Working papers Handbook on precision requirements and variance estimation for ESS households surveys 2013 edition Methodologies and Working papers Handbook on precision requirements and variance estimation for ESS household surveys 2013 edition Europe Direct is a service to help you find answers to your questions about the European Union. Freephone number (*): 00 800 6 7 8 9 10 11 (*)0H The information given is free, as are most calls (though some operators, phone boxes or hotels may charge you). More information on the European Union is available on the Internet (http://europa.eu). Cataloguing data can be found at the end of this publication. Luxembourg: Publications Office of the European Union, 2013 ISBN 978-92-79-31197-0 ISSN 1977-0375 doi:10.2785/13579 Cat. No: KS-RA-13-029-EN-N Theme: General and regional statistics Collection: Methodologies & Working papers © European Union, 2013 Reproduction is authorised provided the source is acknowledged. Acknowledgments Acknowledgments The European Commission expresses its gratitude and appreciation to the following members of the ESS1 Task Force, for their work on Precision Requirements and Variance Estimation for Household Surveys: Experts from the European Statistical System (ESS): Martin Axelson Sweden — Statistics Sweden Loredana Di Consiglio Italy — ISTAT Kari Djerf Finland — Statistics Finland Stefano Falorsi Italy — ISTAT Alexander Kowarik Austria — Statistics Austria Mārtiņš Liberts Latvia — CSB Ioannis Nikolaidis Greece — EL.STAT Experts from European universities: Yves Berger United Kingdom — University of Southampton Ralf Münnich Germany — University of Trier Coordinators of the Task Force from Eurostat unit ‘Quality, methodology and research’ (formerly ‘Methodology and research’): Jean-Marc Museux Chairman Denisa Camelia Florescu Coordinator Methodologists from elsewhere: Nicola Massarelli Eurostat unit ‘Labour market’ Albrecht Wirthmann Eurostat unit ‘Innovation and information society’ (formerly ‘Information society; tourism’) Onno Hoffmeister FAO2 The handbook received significant contributions from Guillaume Osier (Luxembourg — STATEC), working for SOGETI S.A., and specific contributions from Michele D’Alò (Italy — ISTAT) and from AGILIS S.A. experts. The handbook received useful comments from the reviewers Julia Aru (Estonia — Statistics Estonia), Harm Jan Boonstra (Netherlands — CBS), László Mihályffy (Hungary — Hungarian Central Statistical Office), Karim Moussallam (France — INSEE) and Paul Smith (UK — Office for National Statistics), and from the DIME3 members. 1 European Statistical System. 2 Initially attached to Eurostat unit ‘Government and sector accounts; Financial indicators’, member of the Eurostat network of methodologists. 3 Directors of Methodology of the National Statistical Institutes (NSIs) of the European Statistical System (ESS). Handbook on precision requirements and variance estimation for ESS household surveys 2 Foreword Foreword The quality of European statistics is essential for users. European legislation requires that a certain level of quality of statistics is guaranteed, and this quality has to be assessed. One important dimension of quality is accuracy. The accuracy is in the general statistical sense the degree of closeness of estimates to the true values and its components are variance and bias. The scope of this initiative is the variance requirements and estimation. Different statistical domains have been confronted with similar needs related to the variance of estimates. These needs range from setting up precision (variance) requirements and a process to assess compliance to the requirements (in EU Labour Force Survey – LFS), to developing procedures to streamline the production of standard errors of European statistics on the basis of the information provided by National Statistical Institutes (NSIs) of the European Statistical System - ESS (in the Community Survey on ICT4 Usage in Households and by Individuals). The initiatives launched by different statistical domains on similar issues called for a harmonisation of the methods in the ESS. In agreement with the ESS Directors of Methodology (DIME), the Eurostat Directorate ‘Methodology, corporate statistical and IT services’ (in charge of the methodological coordination and support both at Eurostat and ESS level) set up a Task Force (TF) with a generic mandate to issue general recommendations on variance requirements and estimation for ESS household surveys. The implementation of the general recommendations and the specific agreements at stake for LFS and ICT are decided by the domain specialists. Actually, a LFS domain specialist in Eurostat set up a domain specific TF which run in parallel with the DIME TF, discussed the general recommendations and provided valuable feedback to the DIME TF. The coordination between the two TFs was ensured by the Eurostat methodologists. With respect to the ICT, domain specialists in Eurostat are currently assessing the use of methods to estimate standard errors centrally in Eurostat. For efficiency reasons, the DIME TF was a think tank composed of a limited number of high profile experts in Member States, two high profile academic experts and the Eurostat methodologists involved in their respective projects. The handbook prepared under the auspices of the DIME TF was additionally submitted for review by experts designated from other Member States than those who participated to the DIME TF and was subsequently developed on specific issues. We expect that the general recommendations issued within this coordinated approach will provide a basis for a more harmonised/standardised approach of similar problems in other surveys. Antonio Baigorri Daniel Defays Head of Unit Director 4 Information and Communication Technology. Handbook on precision requirements and variance estimation for ESS household surveys 3 Table of contents Table of contents 1. Introduction ....................................................................................................................... 5 2. Precision requirements ...................................................................................................... 6 2.1 The two approaches for specifying precision requirements .......................................... 6 2.2 Precision measures in relation to type of indicators ....................................................12 2.3 Precision requirements and reporting domains ...........................................................16 2.4 Examples of precision thresholds/sizes ......................................................................18 2.5 Recommendations for a standard formulation of precision requirements ....................20 3. Best practices on variance estimation...............................................................................27 3.1 Overview of sampling designs in household surveys ..................................................27 3.2 Sources of variability of an estimator ..........................................................................31 3.3 Variance estimation methods: description, evaluation criteria and recommendations .37 3.4 Some recommended variance estimation methods to account for different sources of variability ..........................................................................................................................53 3.5 Software tools for variance estimation: presentation ...................................................65 3.6 Some examples of methods and tools used for variance estimation ...........................70 3.7 Sampling over time and sample coordination..............................................................71 3.7.1 Variance estimation for annual averages .............................................................76 3.7.2 Variance estimation for estimators of net change .................................................78 3.7.3 Estimation of gross change ..................................................................................85 4. Computing standard errors for national and European statistics .......................................89 4.1 Recommendations for improving computation of standard errors for national and European statistics ...........................................................................................................89 4.2 Possible methods for implementing the integrated approach of variance estimation ...95 5. Possible ways of assessing compliance with precision requirements ............................. 104 6. References ..................................................................................................................... 109 7. Appendix ........................................................................................................................ 120 7.1 Glossary of statistical terms ...................................................................................... 120 7.2 Design effect............................................................................................................. 130 7.3 Metadata template for variance estimation ............................................................... 138 7.4 Suitability of variance estimation methods for sampling designs and types of statistics ....................................................................................................................................... 151 7.5 Suitability of software tools for sampling designs and related issues on variance estimation ......................................................................................................................