MEMBERS OF THE FEDERAL COMMITTEE ON STATISTICAL METHODOLOGY (April 1990) Maria E. Gonzalez (Chair) office of Management and Budget Yvonne M. Bishop Daniel Kasprzyk Energy Information Bureau of the Census Administration Daniel Melnick Warren L. Buckler National Science Foundation Social Security Administration Robert P. Parker Charles E. Caudill Bureau of Economic Analysis National Agricultural Statistical Service David A. Pierce Federal Reserve Board John E. Cremeans Office of Business Analysis Thomas J. Plewes Bureau of Labor Statistics Zahava D. Doering Smithsonian Institution Wesley L. Schaible Bureau of Labor Statistics Joseph K. Garrett Bureau of the Census Fritz J. Scheuren Internal Revenue service Robert M. Groves Bureau of the Census Monroe G. Sirken National Center for Health C. Terry Ireland Statistics National Computer Security Center Robert D. Tortora Bureau of the Census Charles D. Jones Bureau of the Census PREFACE The Federal Committee on Statistical Methodology was organized by the Office of Management and Budget (OMB) in 1975 to investigate methodological issues in Federal statistics. Members of the committee, selected by OMB on the basis of their individual expertise and interest in statistical methods, serve in their personal capacity rather than as agency representatives. The committee conducts its work through subcommittees that are organized to study particular issues and that are open to any Federal employee who wishes to participate in the studies. working papers are prepared by the subcommittee members and reflect only their individual and collective ideas. The Subcommittee on Survey Coverage studied the survey errors that can seriously bias sample survey data because of undercoverage of certain subpopulations or because of overcoverage of other subpopulations. The purpose of this report is to heighten the awareness of survey planners and data users regarding the existence and effects of coverage error, and to provide survey researchers with information to evaluate the trade-offs between coverage error and survey costs. The report profiles selected methods for controlling and measuring the effects of coverage errors using examples from Federal sampling frames and surveys. The report includes seven case studies based on Federal surveys that illustrate selected aspects of coverage errors. The Subcommittee on Survey Coverage was cochaired by Cathryn S. Dippo of the Bureau of Labor Statistics, Department of Labor, and Gary M. Shapiro of the Bureau of the Census, Department of Commerce. MEMBERS OF THE SUBCOMMITTEE ON SURVEY COVERAGE Cathryn S. Dippo (Co-chair) Bureau of Labor Statistics (Labor) Gary M. Shapiro (Co-chair) Bureau of the Census (Commerce) Raymond R. Bosecker National Agricultural Statistics Service (Agriculture) Vicki Huggins Bureau of the Census (Commerce) Roy Kass Energy Information Administration (Energy) Gary L. Kusch Bureau of the Census (Commerce) Melanie Martindale Defense Manpower Data Center (Defense) D.E.B. Potter Agency for Health Care Policy and Research (Health and Human Services) ACKNOWLEDGMENTS This report is the result of the collective work and many meetings of the Subcommittee on Survey Coverage. All of the subcommittee members made significant contributions to the text of the report, taking responsibility for various sections of the report during the long period of preparation. All of the members of the Federal Committee on Statistical Methodology reviewed several drafts and made many important suggestions. The subcommittee wishes to recognize in particular the valuable contributions made by the following committee members: Yvonne Bishop, Joseph Garrett, Charles Jones, Daniel Kasprzyk, Fritz Scheuren Monroe Sirken, and Robert Tortora. The subcommittee also benefitted significantly from an outside review of the final draft by Steven Heeringa and Benjamin Tepping. The subcommittee also thanks the following persons: John Paletta and Richard Pratt for preparing the Current Population Survey and Producer Price Index case studies, respectively; Robert Casady and Charles Cowan for contributing to the section on sample design strategies; and Rosalie Epstein of the Bureau of Labor Statistics for editing the report. TABLE OF CONTENTS Page LIST OF TABLES. vii LIST OF FIGURES . .viii EXECUTIVE SUMMARY . 1 CHAPTER 1. Coverage errors occurring before sample selection. 3 1.1 Conceptual or relevance error . 4 1.2 Frame construction and maintenance. 8 1.2.1. Classification of frame errors. .13 Missing elements; clusters of elements appearing on list; blanks or foreign elements; duplicate elements; incorrect auxiliary information 1.2.2. Frame maintenance . .15 New frame elements; inactive frame elements; misclassified elements; out-of-scope elements; split-out or combined frame elements 1.2.3. Match-merging of independent source lists . .21 1.3. Sample design strategies to minimize coverage error . .22 1.3.1. Defining target population to equal frame population. .23 1.3.2. Random-digit dialing sampling . .23 1.3.3. Multiple frame sampling . .24 1.3.4. Sampling rare populations . .25 1.3.5. Estimation procedures . .27 1.4. Evaluation methods. .28 1.4.1. Macro-level analysis. .28 1.4.2. Micro-level analysis. .29 CHAPTER 2. Coverage errors occurring after initial sample selection. .31 2.1. Incorrect association of frame with reporting unit(s) .31 2.1.1. Location errors . .31 2.1.2. Classification errors . .33 2.1.3. Temporal errors . .36 2.2. Listing errors. .38 2.2.1. Area segment listing errors . .39 Studies measuring error; an alternative to area listing 2.2.2. Household listing errors. .43 Motivational causes; lack of correspondence between survey designer's and respondent's residency concepts; effect of household listing errors; methods for reducing household listing errors 2.2.3. Nonhousehold listing errors . .47 2.3. Other nonsampling errors. .47 2.3.1. Recording errors. .47 2.3.2. Responses from nonsampled units . .49 2.3.3. Coverage errors resulting from nonresponse. .50 CONCLUSION. .53 APPENDIX A. CASE STUDIES Introduction . .55 A.1. Annual Survey of Manufactures (ASM) . .56 A.2. National Long-term Care Survey (NLTCS). .61 A.3. National Master Facility Inventory (NMFI) . .65 A.4. Producer Price Index (PPI). .71 A.5. Quarterly Agricultural Surveys (QAS). .77 A.6. Monthly Report of Industrial Natural Gas Deliveries . .83 A.7. Current Population Survey (CPS) . .89 APPENDIX B. GLOSSARY OF ACRONYMS . .96 APPENDIX C. GLOSSARY OF TERMS. .97 REFERENCES. 106 LIST OF TABLES Number Title Page 1. Selected sampling frames used for Federal surveys. .10 2. Scope of frame versus population of interest for selected surveys. .33 3. Reinterview classification of units originally classified as noninterview: October 1966 . .34 4. Reinterview classification of units originally classified as noninterview: April to September 1966. .35 5. Reinterview classification of units originally classified as noninterview: 1987 35 6. Type B rates for the Survey of Income and Program Participation and the Current Population Survey, 1985-87 (percent). .35 7. Selected surveys in which the frame sampling unit and the final sampling unit are the same . .38 8. Selected surveys in which the frame sampling unit and the final sampling unit differ . .38 9. Examples of surveys requiring field listing 39 10. Comparison of A.C. Nielsen 1982 field canvass of housing units with 1980 census housing unit counts by block group or enumeration district (National Nielsen Television Index Survey segments only) . .40 11. Number of listing errors found in Labor Force Survey study (Statistics Canada). .41 12. Reasons units were added and deleted during reinterview, as determined by reconciliation--area segments only: October 1966 . .41 13. Estimates of percent net CPS within-household undercoverage relative to the 1980 census for males aged 25 and over by their household status (standard errors in parentheses). .45 14. 1986 average coverage ratios by age, sex, and race for CPS .92 15. 1986 average coverage ratios for Hispanics by age and sex for CPS. .92 LIST OF FIGURES Number Title Page 1. Typical physical flow of natural gas from gas well to industrial customer (custody relationship) . .84 2. Possible financial flows (ownership) from gas well to industrial customer (equity relationship). .84 3. Industrial gas estimates from Form EIA-857 submissions: Total United States. .87 viii EXECUTIVE SUMMARY Coverage errors can cause serious biases in estimates based upon sample survey data. Undercoverage may be substantial in many surveys, especially of selected subpopulations. For example, the estimated undercoverage of Hispanic males aged 14 and over is 23 percent in the Current Population Survey (see appendix A.7). In economic surveys, new businesses may be missed at a higher rate than older ones. If the characteristics of the missed portion of the population are very different from those of the covered portion, serious biases in the survey estimates for the total population will result. The purpose of this report is to heighten the awareness of survey program planners and data users concerning the existence and effects of coverage error and to provide survey researchers with information and guidance on how to assess and improve coverage in sample surveys. The report outlines the possible sources and effects of coverage error by documenting current knowledge of coverage errors in Federal surveys. It also profiles selected methods for controlling, measuring, determining the effects of, and reducing coverage errors using examples from Federal