12-539-XIE Statistics Canada Quality Guidelines Third Edition - October 1998 Statistics Statistique Canada Canada Data in many forms Statistics Canada disseminates data in a variety of forms. In addition to publications, both standard and special tabulations are offered. Data are available on the Internet, compact disc, diskette, computer printouts, microfiche and microfilm, and magnetic tape. Maps and other geographic reference materials are available for some types of data. Direct online access to aggregated information is possible through CANSIM, Statistics Canada’s machine-readable database and retrieval system. How to obtain more information Inquiries about this product and related statistics or services should be directed to: Methodology Branch, Statistics Canada, Ottawa, Ontario, K1A 0T6 (telephone: (613) 951-8615), fax: (613) 951-5711, e-mail: [email protected] or to the Statistics Canada Regional Reference Centre in: Halifax (902) 426-5331 Regina (306) 780-5405 Montréal (514) 283-5725 Edmonton (403) 495-3027 Ottawa (613) 951-8116 Calgary (403) 292-6717 Toronto (416) 973-6586 Vancouver (604) 666-3691 Winnipeg (204) 983-4020 You can also visit our World Wide Web site: http://www.statcan.ca Toll-free access is provided for all users who reside outside the local dialling area of any of the Regional Reference Centres. National enquiries line 1 800 263-1136 National telecommunications device for the hearing impaired 1 800 363-7629 Order-only line (Canada and United States) 1 800 267-6677 Ordering Please order by mail, at Statistics Canada, Dissemination Division, Circulation Management, 120 Parkdale Avenue, Ottawa, Ontario, K1A 0T6; by phone, at (613) 951-7277, or 1 800 770-1033; by fax, at 613 951-1584 or 1800 889-9734; or by Internet, at [email protected]. Statistics Canada products may also be purchased from authorized agents, bookstores and local Statistics Canada offices. Standards of service to the public Statistics Canada is committed to serving its clients in a prompt, reliable and courteous manner and in the official language of their choice. To this end, the agency has developed standards of service which its employees observe in serving its clients. To obtain a copy of these service standards, please contact your nearest Statistics Canada Regional Reference Centre. Statistics Canada Methodology Branch Statistics Canada Quality Guidelines Third Edition - October 1998 Published by authority of the Minister responsible for Statistics Canada © Minister of Industry, 1998 All rights reserved. No part of this publication may be reproduced, stored in a retrieval system or transmitted in any form or by any means, electronic, mechanical, photocopying, recording or otherwise without prior written permission from Licence Services, Marketing Division, Statistics Canada, Ottawa, Ontario, Canada K1A 0T6. November 1998 Catalogue no. 12-539-XIE Frequency: Occasional Ottawa La version française de cette publication est disponible no 12-539-XIF au catalogue. Note of appreciation Canada owes the success of its statistical system to a long-standing partnership between Statistics Canada, the citizens of Canada, its businesses and governments. Accurate and timely statistical information could not be produced without their continued cooperation and goodwill. TABLE OF CONTENTS PREFACE......................................................................2 TABLE OF CONTENTS ..........................................................3 1. INTRODUCTION .........................................................4 2. SURVEY STEPS ..........................................................7 2.1 Objectives, Uses and Users.........................................9 2.2 Concepts, Definitions and Classifications .............................12 2.3 Coverage and Frames............................................16 2.4 Sampling .....................................................20 2.5 Questionnaire Design............................................24 2.6 Response and Nonresponse .......................................28 2.7 Data Collection Operations........................................31 2.8 Editing.......................................................35 2.9 Imputation....................................................38 2.10 Estimation ....................................................41 2.11 Seasonal Adjustment and Trend-cycle Estimation.......................46 2.12 Data Quality Evaluation..........................................50 2.13 Disclosure Control..............................................54 2.14 Data Dissemination..............................................59 2.15 Data Analysis (and Presentation) ...................................62 2.16 Documentation.................................................66 2.17 Administrative Data Use..........................................69 3. THE MANAGEMENT CONTEXT...........................................74 3.1 Quality at Statistics Canada .......................................74 3.2 Organization and Infrastructure for Management of Quality...............75 3.3 The Quality Assurance Framework..................................76 3.4 Conclusion....................................................79 4. BIBLIOGRAPHY.........................................................81 5. APPENDICES............................................................92 5.1 Appendix 1: Policy on the Development, Testing and Evaluation of Questionnaires ............................................................92 5.2 Appendix 2: Policy on Informing Users of Data Quality and Methodology....95 6. INDEX OF IMPORTANT TERMS...........................................98 3 1. INTRODUCTION This document brings together guidelines and checklists of issues to be considered in the pursuit of quality objectives in the execution of statistical activities. It draws on the collective experience of many Statistics Canada employees. It should be useful to staff engaged in the planning and design of surveys as well as those who evaluate and analyze the results. Since the publication of the earlier versions of Quality Guidelines, the use of the term quality has broadened, largely because of the Total Quality Management (TQM) movement. We retain the term "quality" to signify the fitness for use of statistical data. It encompasses not only the statistical quality concepts of variance and bias, but also relevance in the sense of measuring the right things and of being available in time to be useful. While broad, this definition is narrower than that of quality in the TQM sense. To reconcile the two, it is worth considering what TQM means for a statistical agency. To oversimplify, there seem to be three elements that comprise TQM: knowing and understanding the clients’ (i.e., users’) needs; involving employees in decision-making associated with meeting these needs; and continuously reviewing business processes for reengineering potential. That attention to these three activities will lead to quality improvement is as true for a statistical agency as it is for any other organization. These Quality Guidelines are related to all three of these elements and support Statistics Canada's long-standing efforts to ensure the quality of its products and services. The following paragraphs expand on the various dimensions of quality with which a statistical agency needs to be concerned (Statistics Canada, 1997a). The quality of data must be defined and assured in the context of being “fit for use.” Whether data and statistical information are fit for use will depend on the intended uses and on their fundamental characteristics of quality, as well as on the expectations of users for what is acceptable for these characteristics of quality. Among statistical agencies there is no standard definition of quality for official statistics. There is, nevertheless, a generally accepted but evolving range of quality issues underlying “fitness for use.” The particular issues of quality or fitness for use that must be addressed by Statistics Canada can be summarized as relevance, accuracy, timeliness, accessibility, interpretability and coherence. These elements of quality need to be considered and balanced in the design and implementation of the agency’s statistical programs. The relevance of data or of statistical information is a qualitative assessment of the value contributed by these data. Value is characterized by the degree to which the data or information serve to address the purposes for which they are produced and sought by users. Value is further characterized by the merit of those purposes, in terms of the mandate of the agency, legislated requirements and the opportunity cost to produce the data or information. Accuracy of data or statistical information is the degree to which those data correctly estimate 4 or describe the quantities or characteristics that the statistical activity was designed to measure. Accuracy has many attributes, and in practical terms there is no single aggregate or overall measure of it. Of necessity these attributes are typically measured or described in terms of the error, or the potential significance of error, introduced through individual major sources of error - e.g., coverage, sampling, nonresponse, response, processing and dissemination. Timeliness of information reflects the length of time between its availability and the event or phenomenon it describes, but considered in the context of the time period that permits the information to be of value and still acted upon. It is typically involved in a trade-off with