11. MEASURES OF NONSAMPLING ERROR All survey estimates are subject to both sampling and newly-constructed housing units; the estimates would nonsampling error. In the section on “Understanding suffer from errors because of housing undercoverage. Error and Determining Statistical Significance,” the topic of sampling error and the various measures avail- The final ACS estimates are adjusted for undercover- able for understanding the uncertainty in the estimates age and overcoverage by controlling county-level esti- due to their being based on estimates derived from a mates to independent total housing unit controls and sample, rather than from an entire population, are dis- to independent population controls by sex, age, race, cussed. The margins of error published with American and Hispanic origin, produced by the Census Bureau’s Community Survey (ACS) estimates measure only the Population Estimates Program. However, it is impor- effect of sampling error. Other errors that affect the tant to measure the extent of coverage adjustment overall accuracy of the survey estimates may occur in by comparing the precontrolled ACS estimates to the the course of collecting and processing the ACS and final controlled estimates. If the extent of coverage are referred to collectively as nonsampling errors. adjustments is large, there is a greater chance that dif- ferences in characteristics of undercovered or over- Broadly speaking, nonsampling error refers to any covered housing units or individuals differ from those error affecting a survey estimate outside of sam- eligible to be selected. When this occurs, the ACS pling error. Nonsampling error can occur in complete may not provide an accurate picture of the population censuses as well as in sample surveys, and is com- before the coverage adjustment, and the population monly recognized as including coverage error, unit controls may not eliminate or minimize that coverage nonresponse, item nonresponse, response error, and error. For more information about coverage error visit processing error. The U.S. Census Bureau has many the Census Bureau’s Coverage Rates Definitions Web procedures in place designed to reduce these sources page.78 of nonsampling error and thus improve the quality of the data. More information about these procedures is However, the process of controlling ACS estimates to available in the section on “Improving Data Quality by the official population estimates may lead to addi- Reducing Nonsampling Error” on the Census Bureau’s tional errors in the ACS data. The population controls Design and Methodology Report Web page.77 used in the ACS are midyear, point-in-time population estimates based on the decennial census, which has different residence rules than the monthly samples Types of Nonsampling Error on which ACS estimates are based. See the section Nonsampling error can result in both random errors on “Residence Rules and Reference Periods” for more and systematic errors. Of greatest concern are system- information. atic errors. Random errors are less critical since they tend to cancel out at higher geographic levels in large Unit nonresponse is the failure to obtain the minimum samples such as the ACS. required information from a housing unit or a resident of group quarters in order for it to be considered a On the other hand, systematic errors tend to accu- completed interview. Unit nonresponse means that no mulate over the entire sample. For example, if there survey data are available for a particular sampled unit is an error in the questionnaire design that negatively or person. For example, if no one in a sampled housing affects the accurate capture of respondents’ answers, unit is available during the time frame for data collec- processing errors are created. Systematic errors often tion, unit nonresponse will result. lead to a bias in the final results. Unlike sampling error and random error resulting from nonsampling error, It is important to measure unit nonresponse because bias caused by systematic errors cannot be reduced by it has a direct effect on the quality of the data. If the increasing the sample size. unit nonresponse rate is high, it increases the chance that the final survey estimates may contain bias, even Coverage error occurs when a housing unit or person though the ACS estimation methodology includes a does not have a chance of selection in the sample nonresponse adjustment intended to control potential (undercoverage), or when a housing unit or person unit nonresponse bias. This will happen if the charac- has more than one chance of selection in the sample teristics of nonresponding units differ from the char- or is included in the sample when they should not acteristics of responding units. For more information have been (overcoverage). For example, if the frame used for the ACS did not allow the selection of
77 U.S. Census Bureau, “Chapter 15: Improving Data Coverage by 78 U.S. Census Bureau, American Community Survey (ACS), Reducing Non-Sampling Error,” American Community Survey Design Coverage Rate Definitions,
Understanding and Using American Community Survey Data 67 U.S. Census Bureau What All Data Users Need to Know 67 about unit nonresponse, visit the Census Bureau’s ACS Quality Measures Response Rates Definitions Web page.79 Nonsampling error is extremely difficult, if not impos- Item nonresponse occurs when a respondent fails sible, to measure directly. However, the Census Bureau to provide an answer to a required item or when the has developed a number of indirect measures of answer given is inconsistent with other information. nonsampling error to help inform users of the qual- With item nonresponse, while some responses to ity of the ACS estimates: sample size, coverage rates, the survey questionnaire for the unit are provided, unit response rates and nonresponse rates by reason, responses to other questions are not obtained. For and item allocation rates. These measures are available example, a respondent may be unwilling to respond on the Census Bureau’s Sample Size and Data Quality 81 to a question about income, resulting in item nonre- Web page. sponse for that question. Another reason for item non- response may be a lack of understanding of a particu- Sample size measures for the ACS summarize infor- lar question by a respondent. mation for the housing unit and group quarters (GQ) samples. The measures available at the state level are:82 Information on item nonresponse allows users to judge the completeness of the data on which the survey • Housing Units estimates are based. Final estimates can be adversely • Number of initial addresses selected impacted when item nonresponse is high, because • Number of final survey interviews bias can be introduced if the actual characteristics of • Group Quarters People (beginning with the the people who do not respond to a question differ 2006 ACS) from those of people who do respond to it. The ACS • Number of initial persons selected estimation methodology includes imputations for item • Number of final survey interviews nonresponse, intended to reduce the potential for item nonresponse bias. For more information about item Sample size measures may be useful in special cir- nonresponse, visit the Census Bureau’s Item Allocation cumstances when determining whether to use 1-year Rates Definitions Web page.80 or 5-year estimates in conjunction with estimates of the population of interest. While the coefficient of Response error occurs when data are reported or variation (CV) can be used to determine usability, as recorded incorrectly. Response errors may be due to explained in the section on “Understanding Error and the respondent, the interviewer, the questionnaire, or Determining Statistical Significance,” there may be the survey process itself. For example, if an interviewer some situations where the CV is small but the user conducting a telephone interview incorrectly records has reason to believe the sample size for a subgroup a respondent’s answer, response error results. In the is very small and the robustness of the estimate is in same way, if the respondent fails to provide a correct question. response to a question, response error results. Another potential source of response error is a survey process Coverage rates measure the ratio of ACS population that allows proxy responses to be obtained, wherein a or housing estimates of geographic areas or groups to knowledgeable person within the household provides the independent estimates for those areas or groups, responses for another person within the household multiplied by 100. National coverage rates are available who is unavailable for the interview. Even more error for the total population by six race/ethnicity catego- prone is allowing neighbors to respond. ries and the GQ population. Coverage rates are also available for housing units and total population by Processing error can occur during the preparation of sex at both the state and national level. Low coverage the final data files. For example, errors may occur if rates are an indication of greater potential for cover- data entry of questionnaire information is incomplete age error in the estimates. or inaccurate. Coding of responses incorrectly also results in processing error. Critical reviews of edits and Unit response and nonresponse rates for housing tabulations by subject matter experts are conducted units are available at the county, state, and national to keep errors of this kind to a minimum.
81 U.S. Census Bureau, American Community Survey, Sample Size and Data Quality,
68 Understanding and Using American Community Survey Data 68 What All Data Users Need to Know U.S. Census Bureau level by reason for nonresponse: refusal, unable to Additional Background Information locate, no one home, temporarily absent, language problem, insufficient data, maximum contact attempts Sample Size and Data Quality reached, and other.
Item allocation rates differ by state, so data users are advised to examine the allocation rates for character- istics of interest before drawing conclusions from the published estimates.
Understanding and Using American Community Survey Data 69 U.S. Census Bureau What All Data Users Need to Know 69