Research at the United States Bureau of the Census Carmen E

Home , Survey data collection, Survey methodology

Research at the United States Bureau of the Census Carmen E. Hoy and Cynthia Z.F. Clark U.S. Bureau of the Census, Washington, D.C.

1. Introduction

Statistical and survey methodology research at the U.S. Bureau of the Census (Census Bureau or Bureau) is conducted by a number of researchers throughout the agency, but most of them are concentrated within the Methodology and Standards Directorate. The directorate has three divisions/offices, each of which contains groups of persons with expertise in specific areas of research. Additionally, the directorate oversees sampling and survey methodology and applied research; establishes best practices; and sets standards for the decennial census, for demographic surveys, and for economic census and survey programs of the Bureau.

The Statistical Research Division (SRD) works in three main research areas: statistical computing, survey statistics, and survey methodology. • The statistical computing staff specializes in statistical computing, computing applications, and technology research. Statistical computing deals with data editing and imputation, record linkage, and variance estimation. Technology research deals with various technology applications, including graphics and metadata. • The survey statistics staff focuses on sampling, estimation, and small area/domain estimation research; statistical estimation and analysis; disclosure limitation research; and time series and seasonal adjustment research. • The survey methodology staff have expertise in social science research, i.e., questionnaire design and measurement research, usability, questionnaire pretesting, and human factors research.

The Planning, Research, and Evaluation Division (PRED) includes groups with interest in administrative records research, survey and census evaluations, and planning for the year 2010 census. The Computer Assisted Survey Research Office (CASRO) coordinates research, develops prototypes, and conducts pretests on computer-assisted survey data collection.

2. Types of Interaction with Academic and Other Researchers

On any given research project there may be various opportunities for Bureau staff to interact with academic researchers and researchers at other agencies. Some researchers are funded under contracts with the Census Bureau to conduct research off-site (i.e., at their academic institutions) on the Bureau's priority projects. Some of these contracts with researchers provide for shared expenses and benefits between the contractor and the Census Bureau. For other research contracts the Census Bureau awards a large multi- year and general research contract and pays the entire cost for services rendered. For the general research contract many of the prime contractors are teamed with one or more organizations or have arrangements with outside experts/consultants to broaden their ability to meet all of the potential needs of the Bureau. These contracts allow divisions and offices to quickly and easily obtain outside advisory and assistance services to support their research and development efforts. These contracts undergo public competitive bidding every five years. Winners of the contract work on specific research task orders submitted by the Census Bureau throughout the period of the contract. The Bureau is currently in the second round for these competitive research contracts. Since 1978 the Census Bureau has partnered with the National Science Foundation and the American Statistical Association to solicit research proposals from academics and other researchers to conduct research at the Census Bureau (on-site) through the ASA/NSF/Census Bureau Research Fellow Program. For these research projects, experienced researchers are encouraged to submit research proposals whose results would benefit Census Bureau programs. This program allows researchers with Special Sworn Status to use Census Bureau confidential data that would not be made available to them at their academic sites. This research arrangement facilitates interaction and exchange of knowledge between the senior researcher and the Census Bureau staff. It also exposes the researchers to interesting problems confronting the Bureau. The researchers often work on these problems after returning to their academic institutions, and may interest their graduate students in them — continuing a knowledge exchange over many years.

The Census Bureau has several other forums in which staff interact with academic researchers not funded by the agency. Such interactions occur at Bureau-sponsored conferences, workshops, expert review panels, professional advisory committee meetings, and professional meetings. For 12 years (1985 - 1996) the Census Bureau sponsored an Annual Research Conference that brought together researchers from academia and from organizations producing official statistics. This conference was successful in interesting researchers in the work of official statisticians. During 1997-98 the Census Bureau hosted three one-day conferences on, respectively, Censuses of Population and Housing, the American Community Survey, and Small Area Estimation. These conferences also attracted a mix of individuals from academia and from official statistical organizations. In November 1999 the Office of Management and Budget's Federal Committee on Statistical Methodology (FCSM) will, with significant support from the Census Bureau, host the FCSM Research Conference. In June 2000 the Census Bureau will join with other official statistical agencies in sponsoring the Second International Conference on Establishment Surveys.

Two ongoing forums are particularly valuable to the Census Bureau. The Bureau has a professional advisory committee, meeting twice yearly for two days, made up of representatives of the American Statistical Association, the Population Association of America, the American Economic Association, and the American Marketing Association. On occasions when in-depth review of a specific survey topic is needed, the Bureau contracts with the National Academy of Sciences’ Committee on National Statistics to create a panel of academic researchers to review the area and make recommendations. Several panels of note have been convened during the past few years, particularly to review census methodology. The three panels were: 1) Panel on Census Requirements in the Year 2000 and Beyond; 2) Panel to Evaluate Alternative Census Methods; and 3) Panel on Alternative Census Methodologies. Two new census panels were initiated recently: Panel to Review the 2000 Census, and Panel on the Census in the Twenty-first Century.

Additionally, the Department of Education commissioned a Panel on the Small Area Estimates of Income and Poverty (SAIPE) [estimates that the Census Bureau was producing] providing the basis for distributing $8 billion in funds to local school districts. More recently, a workshop on the newly-initiated American Community Survey (ACS) was held. This was the American Community Survey Workshop - Building a Research Agenda.

3. Examples of Research Projects 3

We have selected several projects in the areas of research mentioned above to illustrate the impact of academic interactions on the statistical and survey research of the Census Bureau.

3.1. Mathematical Statistics Research

Disclosure Limitation Methods. The purpose of this research is to develop disclosure limitation methods to be used for Census Bureau data products that are to be made publicly available. Emphasis will be placed on techniques to implement disclosure limitation at the stage of processing. On this project, researchers from academia, other countries, and other agencies have interacted.

The Census Bureau has recently funded contracts with Steve Samuels at Purdue University and, through Westat, Inc., with Stephen Fienberg at Carnegie Mellon University. Samuels’ research focused on a measure of disclosure risk for microdata. Fienberg’s research focused on disclosure limitation methods for demographic tabular data. The Bureau plans to fund a contract for the disclosure review of American FactFinder, the computer system that will be used to facilitate user access to Census Bureau data products and user-defined data products.

With academics and federal workers from nine different European countries, the Bureau also funded a recent workshop on confidentiality and data access. The focus of the workshop was to coordinate disclosure-limitation research, encourage collaborations, and prioritize research. Attendees will now be seeking funding for this collaborative work. With Census Bureau leadership, the United States will host a follow-up conference in the fall of 2000.

In addition, the Census Bureau recently made free cell-suppression and auditing software available to other statistical agencies. All staff in the disclosure limitation research group within the Statistical Research Division are members of the Interagency Confidentiality and Data Access Group (ICDAG), an interest group of the FCSM. ICDAG has done a series of workshops on Privacy, Confidentiality, and the Protection of Data--A Statistical Perspective. ICDAG has prepared a Checklist on Disclosure Potential of Proposed Data Releases which aids people in reviewing data for potential disclosure problems. Other ICDAG products under development include five-dimensional cell-suppression auditing software, a general brochure on confidentiality and data access issues, a review of data licensing procedures, and a manual on restricted access options.

Time Series and Seasonal Adjustment Research. For many years there has been significant interaction between SRD’s time series staff and academic time series researchers. This interaction has taken many forms. There have been numerous ASA/NSF/Census Bureau Research Fellows working in time series, starting with Arthur Dempster (Harvard University), William Cleveland (now at the Federal Reserve Board), and Steve Hillmer (University of Kansas) in the first two years (1978-80) of the program, all of whom worked on seasonal adjustment. This has continued: recent Fellows include James Durbin (retired from the London School of Economics, working on non-Gaussian seasonal time series models) and Peter Thomson (Victoria University of New Zealand, working on trend filters). The Bureau has expanded its interactions to include researchers from outside the United States.

Time series research has also been funded through joint statistical agreements, including one with Wayne Fuller (Iowa State University, working on a variety of time series and sample survey topics). Substantial informal interaction with researchers has resulted in a number of joint publications between Census Bureau researchers and academics.

The Census Bureau has organized four conferences on time series topics, with concentration on seasonal adjustment. The first two (in 1976 and 1981) were rather formal, had substantial academic participation, and resulted in published proceedings. The third (1992) concentrated on attracting participants from statistical offices, with more limited academic participation. The fourth was organized as a follow-on to the 1995 Annual Research Conference, and as a successor to the 1992 conference with somewhat broader academic participation.

The Bureau’s time series research has focused on seasonal adjustment and on a multi-platform computer program for seasonal adjustment, trend estimation, and calendar-effect estimation extending beyond the capabilities of current seasonal adjustment programs. This computer program would provide more effective diagnostics. Versions of the X-12-ARIMA seasonal adjustment program were distributed worldwide with user documentation, accompanied by requests for user identification of improvements. One such identified improvement was the automatic time series model identification procedure, used by Eurostat, of the Gomez-Maravall TRAMO program. Four short courses on the use of the X-12-ARIMA and X-12 Graph, presented in Washington, Tokyo, Seoul, and Rio de Janeiro, provided additional opportunities for identifying improvements.

Small Area Income and Poverty Estimates Research. The Small Area Income and Poverty Estimates (SAIPE) project has involved a unique relationship between the Census Bureau and academic researchers. The primary goal of the project has been to produce biennial estimates of poverty among school-aged (5-17) children in U.S. states, counties, and “local education areas” (LEAs, more commonly known as school districts). This approach uses small area models constructed from data taken from the U.S. Current Population Survey (CPS) March Income Supplement, from the previous (1990) decennial census, and from administrative records (tax return data obtained from the U.S. Internal Revenue Service, and food stamp participant data obtained from the U.S. Department of Agriculture).

Because of the importance of developing estimates of children in poverty in connection with allocating funds to school districts, the Department of Education contracted with the National Academy of Sciences’ Committee on National Statistics to fund the Panel on Estimates of Poverty for Small Geographic Areas. This panel was directed to review the work of the Census Bureau SAIPE project team. The panel was composed of U.S. and Canadian academic and private sector researchers knowledgeable about the problems and issues. They came from the disciplines of statistics, economics, sociology, and education. What distinguished this arrangement from those of other NAS panels touching on Census Bureau work was the fact that this panel was funded by another agency (Department of Education) to make recommendations to both the Secretary of Commerce and the Secretary of Education respecting the question whether the Census Bureau’s estimates should be used for allocation of education funds. This set

Sample Design and Estimation Research. Members of the Statistical Research Division conduct joint research with Iowa State University faculty through an interagency contract and a cooperative agreement. Problems for research are motivated by Census Bureau needs. Personnel working under the agreement interact constantly via phone, fax, mail, and meetings. The joint research has been presented at major statistical conferences, with selected papers accepted for publication in statistical journals. Descriptions of two research projects follow.

• Developing a method to construct a census data file in the presence of sampling and estimation where such effects are transparent to the data user. Data files of interest include both short-and long-form data.

• Developing a method to weight census long-form sample data with desirable properties. This research investigates methodology for obtaining a set of common household weights in a situation where person and household constraints are stipulated. Currently, in the decennial census, two separate weights are developed. In addition, there is a general investigation of a general robust weighting procedure that “borrows strength” for efficient estimation.

Survey of Income and Program Participation (SIPP) Research. Research is under way on methods of modeling, measuring, and adjusting for attritional bias, also on the relative effectiveness and interaction between missing data and post-stratification adjustments for selected survey items in a longitudinal survey. In addition, the Bureau is developing and/or adapting appropriate survey methodology that adequately accounts for survey design complexities in the estimation and analysis of SIPP data.

A standing committee of the Survey Research Methods Section of the American Statistical Association reviews the SIPP research activities annually and offers valuable recommendations regarding the nature and direction of future work on that survey. The Census Bureau engages in regular, informal, individual, and small group discussions with committee members. It also sponsors formal interchanges of ideas and results relating to SIPP with a variety of researchers from academia and from Statistics Canada.

Latent Class Models. Paul Biemer, Research Triangle Institute (RTI), has developed expertise in latent class model applications to survey problems. The Census Bureau has had a contract with RTI for Biemer to conduct research on measurement error for the employment status question of the CPS and on person coverage for the census/ICM. These models use latent variables in the analysis of measurement errors on demographic surveys.

A member of the SRD staff worked with Biemer to become familiar with latent class models and their usefulness in estimating measurement error. This staff member's expertise should be sufficient to advise

3.2. Statistical Computing Research

Increased use of the best computational methods in statistics, computer science, and operations research makes possible, in relation to Federal statistical data, both improved accuracy and significant cost-savings. The statistical computing research group has the responsibility for developing generalized software systems that can be used in many surveys or censuses. The Bureau has experienced some success in developing statistical methods (and software) touching on editing and imputation, record linkage, variance estimation, and automated coding. The new software is being integrated into production programs. One major recent success story has been the development of a generalized economic editing system, SPEER, that can be used in most economic surveys and censuses. Other examples are the VPLX variance estimation program and various automated coding systems.

General Edit/Imputation Support. The purpose of this project is to provide advice, to develop computer edit/imputation systems in support of demographic and economic projects, to implement prototype production systems, and to investigate edit/imputation methods. In addition to the SPEER project mentioned above, efforts are currently directed toward the creation of an edit/imputation system for decennial census and demographic survey data using the DISCRETE prototype edit system and a statistically valid item and unit imputation methods. The edit part of the project is directed toward creating valid code and sufficiently fast algorithms for editing, and translating traditional decennial edit rules into the Felligi-Holt framework in a technically feasible manner. The objective of the imputation component is to impute for missing and contradictory data using statistically valid methods.

In collaboration with Brian Greenberg of the Census Bureau, a precursor of the DISCRETE edit system was developed by Professor Robert Garfinkel and his associates when he was at the University of Tennessee. The Census Bureau staff has distributed this system and has interacted with interested staff from Statistics Canada, Statistics Netherlands, ISTAT (Italy), Statistics Sweden, and the Office of National Statistics (United Kingdom).

General Record Linkage Support. The purpose of this project is to provide advice, develop computer matching systems in support of Census Bureau record linkage projects, and develop analytic methods for adjusting statistical analyses for computer matching error. The Census Bureau funded a contract with Don Rubin and Mike Larsen (Harvard University) to conduct record linkage research on error rate estimation and model evaluation. Larsen and Rubin recently completed a paper that generalizes 1993 work by William Winkler of Census Bureau. Using ideas of Larsen and Rubin, Winkler is expanding his 1993 work.

National Science Foundation Funded Research in Digital Government. Recently, the Census Bureau and other Federal statistical agencies have initiated collaborative efforts with academics from university computer science and statistical computing departments. A new grant program of the National Science Foundation (NSF), the Digital Government Grant Program, is targeted at partnering Federal agencies with research groups at universities and in industry in order to improve the adoption of new information technologies by Federal agencies. A working group of representatives from U.S. statistical agencies, including the Bureau of Labor Statistics, the Census Bureau, the Bureau of Justice Statistics, the Energy Information Agency, and the National Center for Health Statistics, has encouraged academic researchers in statistics-related areas to form partnerships and apply for the NSF Digital Government Grants. In the first year of the program, four grants for research that would benefit official statistics programs were awarded.

• Center for Applied Research and Development in Government Services (CARDGIS) “Survey Authoring and Administration Testbed,” Information Sciences Institute (ISI) of the University of Southern California.

• “A Web-based Query System for Disclosure-Limited Statistical Analysis of Statistical Data,” National Institute of Statistical Sciences and Carnegie Mellon University.

• “Citizen Access to Government Statistical Data: A Pilot Collaboratory,” Universities of Maryland, Syracuse, and North Carolina

• “Center for Applied Research and Development in Government Services (CARDGIS),” Information Sciences Institute (ISI) at USC/Columbia University. This program is an umbrella for several related projects focused on improving access to statistical data. The first project has to do with automatically generating ontologies (collections of terms).

3.3. Center for Survey Methods Research

Decennial Census Forms Design and Mail Strategy. After the 1990 census revealed generic problems, both the forms for the 2000 census and the mail strategy for delivering those forms have been tested extensively. Working with Don Dillman (Washington State University), Census Bureau staff conducted a series of experiments involving use of an advance letter, a mail reminder card, a replacement questionnaire, personalization of mailing pieces, first class stamps, and other factors to test the effectiveness of expanding the individual components of the mail design strategy. Don Dillman and Eleanor Singer (University of Michigan) planned and conducted an experiment testing the effectiveness of messages stressing the fact that response to the census is mandated, compared with messages stressing that participation in the census is objectively beneficial.

Research on the questionnaire itself involved a variety of stages. Design work was done both in-house and by 212 Associates, a private design firm. Cognitive testing was conducted both in-house and by outside contractors, including Don Dillman, Robert Belli, and Roger Tourangeau (University of Michigan). Questionnaire testing focused on respondents’ reactions to the questionnaire, envelope, and letter from the

Director of the Census Bureau. Motivational factors touching on participation were examined, and alternative colors and questionnaire formats were evaluated.

The 2000 census will include a series of experimental questionnaires focused on testing skip instructions and residency rules (for determining who should be included as household members). The development of these forms has been accomplished both by Census Bureau staff and others working under contract. Dillman has worked with Bureau staff in developing and pretesting the skip instruction versions; innovative research was also contracted out to Erica, Inc., to study respondents’ eye movements as they completed questionnaires with different skip instructions. The Census Bureau contracted with Westat, Inc., a private research organization, to develop and cognitively pretest alternative versions of the residency rules.

In a related project, the Census Bureau has worked with university researchers to apply linguistic analysis algorithms to the diagnosis of potential cognitive problems associated with survey questionnaires. Arthur C. Graesser (University of Memphis) is the lead academic researcher with responsibility for modeling and software development. The Bureau will lead an iterative evaluation of the prototype products.

Usability Laboratory. The Census Bureau has recently established a usability laboratory to conduct tests of the human-computer software and hardware interfaces involved in data collection, data dissemination, and internal operations. The goals are to promote user-centered design through research and collaboration that seeks to understand both general usability principles and their expression in specific software applications developed at the Census Bureau. Current research focuses on the design of navigational menus and on the visualization of results from searches. Ben Shneiderman (University of Maryland) consults with Bureau staff and conducts seminars to inform staff of latest developments. This program also relies heavily on graduate student interns from local universities to staff its applied research and service programs. Graduate student input has helped us to develop expertise in this emerging field that has application to processes used for producing official statistics.

Ethnographic Research. The Census Bureau has sponsored ethnographic research focused on issues related to undercounted populations for a number of years. In 1971, the Census Bureau contracted with anthropologists Charles and Betty Lou Valentine to produce alternate enumerations of households in the Current Population Survey sample. The results showed that the survey rosters omitted a number of young black men who had marginal attachments to the sample households. This method was used extensively in the 1990 census, when the Census Bureau sponsored 29 contracts with ethnographers associated with universities and non-profit community organizations. Each ethnographer worked with a specific target population, among them Blacks, Hispanics, Native Americans, recent immigrants from Asia, and undocumented immigrants (primarily from Latin America and Haiti). The work of the ethnographers focused on the behavioral causes and correlates of undercount in the decennial census among their target population.

The Center for Survey Methods Research has recently sponsored ethnographic research in several areas. The Cognitive Study of Living Situations examined respondents' understandings of residence concepts related to the decennial census. The research identified terminology and concepts used in the census

An on-going ethnographic project, entitled Protecting Privacy: The Ethnography of Personal Information Management, is investigating respondents’ beliefs about privacy as they relate to household surveys. It focuses on respondents’ decision-making when they are asked to reveal information about themselves in a variety of governmental and non-governmental contexts. This project is expected to be useful in understanding nonresponse, question sensitivity, and respondent reactions to collections. The Census Bureau has funded contracts with academic ethnographers to work on this research. Currently, four anthropologists are collaborating in the project to study privacy beliefs. They include Susan Trencher (George Mason University), and three experienced contract ethnographers, Alisu Shoua-Glusberg, Betsy Strick, and Jessica Skolnikoff.

3.4. Administrative Records Research Program

Administrative records offer potential solutions to those who seek to generate timely statistics — and to do so at lower costs and with reduced respondent burden. The Census Bureau’s authorizing legislation explicitly directs: “to the maximum extent possible . . . the Secretary shall acquire and use information available from any source . . . instead of conducting direct inquiries.” The Census Bureau established a corporate research staff to investigate expanded uses of administrative records for various programs, including inter-census population estimates, evaluation and enhancement of survey data, address list development, and execution of the decennial census.

A simulation of an administrative records census is planned for Census 2000. For this experiment, an administrative record census is defined as a census that uses existing records from other administrative agencies as the primary data collection method and that provides content and geographic detail which meet reapportionment and redistricting requirements. The Bureau plans to compare the administrative records census results with Census 2000 results in order to provide critical measures and a basis for 2010 Census planning.

A number of national-level administrative records files will be assembled, unduplicated using the Social Security Numbers (SSN), and assigned block level geographic codes. Only then will the results be produced. Two sites have been selected [with a total of approximately one million housing units and a population of three million persons]. The areas selected represent a mixture of difficulties in conducting an administrative records census. These include variations in type of housing and mobility of population. Files being considered for inclusion in the experiment are from the Internal Revenue Service, Social Security Administration, Selective Service, Federal Housing Programs, Indian Health Service, and the Medicare Program.

In order to facilitate both this large test and other research and operational applications of administrative records, the Census Bureau has been expanding collaboration with federal agencies (and some state

This paper reports the results of research and analysis undertaken by Census Bureau staff. It has undergone a more limited review than official Census Bureau publications. This report is released to inform interested parties of research and to encourage discussion. 10 agencies) over the last several years. Our relationship with the Internal Revenue Service goes back over 20 years because their data, by statute, provides a basis for much of our economic statistics program and for inter-census estimates of the population. Our relationship with the Social Security Administration (SSA) is based in large part on collaborative research projects that link SSA summary earnings to Census Bureau demographic survey data. The Census Bureau has arrangements that provide Special Sworn Status to some of the SSA’s researchers so that the linked data sets reside at both agencies. We have also contracted with retired employees of statistical units in administrative record agencies to increase our understanding of the ways in which administrative records are used and enhance our vision of how these records might be used for official statistics programs. In particular, the Census Bureau has received services from Frederick Scheuren (formerly of IRS), and Barry Bye (formerly of SSA). The Census Bureau uses administrative data to evaluate and enhance the survey data; the administrative record agencies use aggregate survey data to conduct program and policy analysis. 4. Summary

The Census Bureau has taken advantage of opportunities for staff interaction with academic researchers and other researchers, both from the private sector and from other government statistical agencies. Both individual researchers and the research program at the Census Bureau have clearly benefitted from such interactions. The Bureau plans to continue to initiate and nurture interactions with outside researchers It will do this through collaboration with other government statistical organizations, through the funding of contracts, and through the support of conferences, workshops, advisory panels, and meetings. The Census Bureau will continue to seek out those individuals who can assist in advancing the Bureau’s research agenda.

5. References

Citro, Constance F., Michael L. Cohen, Graham Kalton, and Kirsten K. West, Small-Area Estimates of School-Age Children in Poverty, Interim Report 1, Evaluation of 1993 County Estimates for Title 1 Allocations, 1997; Interim Report 2, “Evaluation of Revised 1993 County Estimates for Title 1 Allocations,” 1998; Interim Report 3, “Evaluation of 1995 County and School District Estimates for Title 1 Allocations,” 1999.

Cohen, Michael L. and Andrew A. White (1999), Measuring a Changing Nation: Modern Methods for the 2000 Census. National Academy Press, Washington, DC.

Edmonston, Barry and Charles Schultze (1995). Modernizing the U.S. Census. National Academy Press, Washington, DC.

Steffey, Duane L. and Norman M. Bradburn (1994). Counting People in the Information Age. National Academy Press, Washington, DC.

Annual Report of the Statistical Research Division (Fiscal Year 1998). Bureau of the Census.

Program of General Research & Support (FY 2000-FY 2004) (July 1999). Statistical Research Division, Bureau of the Census.

Other contributors to this paper; Leroy Bailey, William Bell, Robert Creecy, Theresa DeMaio, Eleanor Gerber, Cary Isaki, Charlene Leggieri, Kent Marquis, William Winkler, Laura Zayatz.