Catalogue no. 12-587-X Survey Methods and Practices How to obtain more information For information about this product or the wide range of services and data available from Statistics Canada, visit our website at www.statcan.gc.ca, e-mail us at [email protected], or telephone us, Monday to Friday from 8:30 a.m. to 4:30 p.m., at the following numbers: Statistics Canada’s National Contact Centre Toll-free telephone (Canada and United States): Inquiries line 1-800-263-1136 National telecommunications device for the hearing impaired 1-800-363-7629 Fax line 1-877-287-4369 Local or international calls: Inquiries line 1-613-951-8116 Fax line 1-613-951-0581 Depository Services Program Inquiries line 1-800-635-7943 Fax line 1-800-565-7757 To access this product This product, Catalogue no. 12-587-X, is available free in electronic format. To obtain a single issue, visit our website at www.statcan.gc.ca and browse by “Key resource” > “Publications.” Standards of service to the public Statistics Canada is committed to serving its clients in a prompt, reliable and courteous manner. To this end, Statistics Canada has developed standards of service that its employees observe. To obtain a copy of these service standards, please contact Statistics Canada toll-free at 1-800-263-1136. The service standards are also published on www.statcan.gc.ca under “About us” > “The agency” > “Providing services to Canadians.” Statistics Canada Survey Methods and Practices Published by authority of the Minister responsible for Statistics Canada © Minister of Industry, 2010 All rights reserved. The content of this electronic publication may be reproduced, in whole or in part, and by any means, without further permission from Statistics Canada, subject to the following conditions: that it be done solely for the purposes of private study, research, criticism, review or newspaper summary, and/or for non-commercial purposes; and that Statistics Canada be fully acknowledged as follows: Source (or “Adapted from”, if appropriate): Statistics Canada, year of publication, name of product, catalogue number, volume and issue numbers, reference period and page(s). Otherwise, no part of this publication may be reproduced, stored in a retrieval system or transmitted in any form, by any means—electronic, mechanical or photocopy—or for any purposes without prior written permission of Licensing Services, Client Services Division, Statistics Canada, Ottawa, Ontario, Canada K1A 0T6. Originally published in October 2003 Catalogue no. 12-587-X ISBN 978-1-100-16410-6 Frequency: Occasional Ottawa Cette publication est également disponible en français. Note of appreciation Canada owes the success of its statistical system to a long-standing partnership between Statistics Canada, the citizens of Canada, its businesses, governments and other institutions. Accurate and timely statistical information could not be produced without their continued cooperation and goodwill. National Library of Canada Cataloguing in Publication Data Main entry under title: Survey methods and practices Issued also in French under title: Méthodes et pratiques d’enquête. ISBN 0-660-19050-8 CS12-587-XPE 1. Surveys – Methodology. 2. Household Surveys – Methodology. 3. Questionnaires – Design. 4. Sampling (Statistics) – Methodology. I. Statistics Canada. II. Statistics Canada. Social Survey Methods Division. III. Title. HA37.C3 S87 2003 001.4’33 C2003-988000-1 Preface I am very proud of the Statistics Canada publication Survey Methods and Practices. It is a real achievement and marks the culmination of the efforts of a large number of staff at Statistics Canada, particularly in the Survey Methodology Divisions, and I should like to express my sincere thanks and appreciation to them all. This publication has benefited from courses given to Statistics Canada staff, workshops given to clients and courses given on censuses and surveys to African and Latin American statisticians. Of particular note is Statistics Canada’s unique and innovative Survey Skills Development Course, which has been given on more than 80 occasions to over 2,000 staff at Statistics Canada, as well as to staff from other national statistical agencies. It was given particular impetus by the production of the Survey Skills Development Manual for the National Bureau of Statistics of China under the auspices of the Canada – China Statistical Co-operation Program. The main use of this publication will be in support of the Survey Skills Development Course and I feel that it will become required reading and a reference for all employees at Statistics Canada involved in survey-related work. I hope it will also be useful to statisticians in other agencies, as well as to students of survey design and methods courses for whom it could serve as a source of insights into survey practices. Ottawa Dr. Ivan P. Fellegi October 2003 Chief Statistician of Canada Foreword This manual is primarily a practical guide to survey planning, design and implementation. It covers many of the issues related to survey taking and many of the basic methods that can be usefully incorporated into the design and implementation of a survey. The manual does not replace informed expertise and sound judgement, but rather is intended to help build these by providing insight on what is required to build efficient and high quality surveys, and on the effective and appropriate use of survey data in analysis. It originated as part of the Canada – China Statistical Co-operation Program, funded by the Canadian International Development Agency. The manual developed for that program was designed to assist the National Bureau of Statistics of China as part of its national statistical training program. It was accompanied by a Case Study that covered the main points of the manual through the use of a hypothetical survey. The China manual and Case Study were revised and modified to yield this manual for use at Statistics Canada, particularly as a companion reference and tool for the Statistics Canada Survey Skills Development Course. Although the main focus of the manual is the basic survey concepts useful to all readers, some chapters are more technical than others. The general reader may selectively study the sections of these technical chapters by choosing to skip the more advanced material noted below. The first five chapters cover the general aspects of survey design including: - an introduction to survey concepts and the planning of a survey (Chapter 1); - how to formulate the survey objectives (Chapter 2); - general considerations in the survey design (Chapter 3), such as: - whether to conduct a sample survey or a census; - how to define the population to be surveyed; - different types of survey frames; - sources of error in a survey; - methods of collecting survey data (Chapter 4), such as: - self-enumeration, personal interview or telephone interview; - computer-assisted versus paper based questionnaires; and - how to design a questionnaire (Chapter 5). Chapters 6, 7 and 8 cover more technical aspects of the design of a sample survey: - how to select a sample (Chapter 6); - how to estimate characteristics of the population (Chapter 7); - how to determine the sample size and allocate the sample across strata (Chapter 8). In Chapter 7, the more advanced technical material begins with section 7.3 Estimating the Sampling Error of Survey Estimates. In Chapter 8, the formulas used to determine sample size, requiring more technical understanding, begin with section 8.1.3 Sample Size Formulas. Chapter 9 covers the main operations involved in data collection and how data collection operations can be organised. Chapter 10 discusses how responses to a questionnaire are processed to obtain a complete survey data file. The more technically advanced material begins with section 10.4.1 Methods of Imputation. Chapter 11 covers data analysis. The more technically advanced material in this chapter is covered in section 11.4 Testing Hypotheses About a Population: Continuous Variables. Chapter 12 deals with how data are disseminated to users and avoiding disclosure of individual data or data for a particular group of individuals. Chapter 13 treats the issues involved in planning and managing a survey. This chapter is a non- technical chapter, aimed at potential survey managers or those who would be interested in or who are involved in planning and managing a survey. In addition to these 13 chapters, there are two appendices. Appendix A addresses the use of administrative data – data that have been collected by government agencies, hospitals, schools, etc., for administrative rather than statistical purposes. Appendix B covers quality control and quality assurance, two methods that can be applied to various survey steps in order to minimise and control errors. Acknowledgements Thanks are due to the many Statistics Canada employees who contributed to the preparation of Survey Methods and Practices, in particular: Editors: Sarah Franklin and Charlene Walker. Reviewers: Jean-René Boudreau, Richard Burgess, David Dolson, Jean Dumais, Allen Gower, Michel Hidiroglou, Claude Julien, Frances Laffey, Pierre Lavallée, Andrew Maw, Jean-Pierre Morin, Walter Mudryk, Christian Nadeau, Steven Rathwell, Georgia Roberts, Linda Standish, Jean-Louis Tambay. Reviewer of the French translation: Jean Dumais. Thanks are also due to everyone who contributed to the preparation of the original China Survey Skills Manual, in particular: Project Team: Richard Burgess, Jean Dumais, Sarah Franklin, Hew Gough, Charlene Walker.