
MEMBERS OF THE FEDERAL COMMITTEE ON STATISTICAL METHODOLOGY (November 1983) Maria Elena Gonzalez (Chair) Office of Information and Regulatory Affairs (OMB) Barbara A. Bailar Bureau of the Census (Commerce) Norman O. Beller National Center for Education Statistics (Education) Yvonne M. Bishop Energy Information Administration (Energy) Edwin J. Coleman Bureau of Economic Analysis (Commerce) John E. Cremeans Bureau of Industrial Economics (Commerce) Zahava D. Doering Defense Manpower Data Center (Defense) Maria D. Eldridge National Center for Education Statistics (Education) Daniel H. Garnick Bureau of Economic Analysis (Commerce) Charles D. ]ones Bureau of the Census (Commerce) Daniel Kasprzyk Bureau of the Census (Commerce) William E. Kibler Statistical Reporting Service (Agriculture) Thomas Plewes Bureau of Labor Statistics (Labor) Fritz J. Scheuren Internal Revenue Service (Treasury) Monroe G. Sirken National Center for Health Statistics (Health and Human Services) Thomas G. Staples Social Security Administration (Health and Human Services) Statistical Policy Working Paper 10 Approaches to Developing Questionnaires Prepared by Subcommittee on Questionnaire Design Federal Committee on Statistical Methodology Edited by Theresa J. DeMaio Bureau of the Census MEMBERS OF THE SUBCOMMITTEE ON QUESTIONNAIRE DESIGN Dawn D. Nelson (Chair) Maria E. Gonzalez* (ex officio) Bureau of the Census (Commerce) Office of Information and Regulatory Affairs (OMB) Deborah H. Bercini Janice Olson National Center for Health Social Security Administration Statistics (HHS) (HHS) Theresa J. DeMaio Anitra Rustemeyer Streett Bureau of the Census (Commerce) Energy Information Administration (Energy) Richard W. Dodge Ronny Schaul Bureau of the Census (Commerce) Bureau of Labor Statistics (Labor) Gemma M. Furno Margaret Weidenhamer Bureau of the Census (Commerce) Statistical Reporting Service (Agriculture) Additional Contributors to the Report on Approaches to Developing Questionnaires Catherine J. Baca Carol M. Utter Bureau of the Census (Commerce) Bureau of Labor Statistics (Labor) *Member, Federal Committee on Statistical Methodology ii OFFICE OF INFORMATION AND REGULATORY AFFAIRS Christopher DeMuth, Administrator Thomas D. Hopkins, Deputy Administrator for Regulatory and Statistical Analysis Dorothy M. Tella, Chief Statistician Maria E. Gonzalez, Chairperson Federal Committee on Statistical Methodology PREFACE The Subcommittee on Questionnaire Design was formed by the Federal Committee on Statistical Methodology to address the general topic of questionnaire design. The Subcommittee focused on a review of methods used in developing questionnaires. The working paper discusses approaches to devising questionnaires in three broad areas: tools for developing questions, procedures for testing the questionnaire draft, and techniques for evaluating the ques- tionnaire. While the report is intended primarily to be useful to Federal agencies that develop questionnaires, a broader audience may also find the report of interest. Seminars and meetings will be organized to discuss the topics addressed by this subcommittee with Federal agency personnel. The Subcommittee was chaired by Dawn D. Nelson, Bureau of the Census, Department of Commerce. As a subcommittee report, this document does not necessarily represent the views of the Office of Management and Budget. iii ACKNOWLEDGMENTS This report represents the collective effort of the Subcommittee on Questionnaire Design. Although all members of the subcommittee reviewed and commented on the entire report, individual members were responsible for preparing various chapters. Chapters 4 and 11, however, were prepared by persons who were not members of the subcommittee. The names of the authors of the respective chapters appear below. Chapter Author 1 Theresa J. DeMaio 2 Anitra Rustemeyer Streett 3 Margaret Weidenhamer 4 Catherine J. Baca 5 Dawn D. Nelson 6, Section I Gemma M. Furno 6, Section II Janice Olson 7 Anitra Rustemeyer Streett 8 Deborah H. Bercini 9 Theresa J. DeMaio 10 Richard W. Dodge 11 Carol M. Utter The following persons also deserve special recognition for their role in assisting the work of the subcommittee. Our work was initially guided by Naomi Rothwell who served as the chair from November 1980 until her retirement from government service in March 1981. Maria Gonzalez worked with the subcommittee throughout the development of the report and provided a link with the Federal Committee on Statistical Methodology (FCSM). Various members of the FCSM provided advice on our work at different stages, and Barbara Bailar and Zahava Doering supplied comments on the complete report draft. Much of the report material was also reviewed by Thomas Jabine who provided encouragement for our work. We especially appreciate Theresa DeMaio's contribution in sharpening the focus of the report by effectively organizing and editing the material. Also, we are grateful to Laura Taylor, Vicki Horton, Debbie Barnett, and Cathleen Tyson of the Bureau of the Census for typing and assembling the many drafts of this report. The Bureau of the Census also provided the funding for publication preparation and printing. iv CONTENTS Part I Chapter 1: Overview. 3 Part II: Tools for Developing Questions. .11 Chapter 2: Unstructured Individual Interviewing. .13 Chapter 3: Qualitative Group Interviews. .21 Chapter 4: Participant Observation29 Part III: Procedures for Testing the Questionnaire Draft. .41 Chapter 5: Informal Testing. .45 Chapter 6: Formal Testing. .57 Section I. Pilot Studies. .57 Section II. Split Sample Testing . .70 Part IV: Techniques for Evaluating the Questionnaire Draft . .89 Chapter 7: Investigating Respondent's Interpretations of Survey Questions. .93 Chapter 8: Observation and Monitoring of Interviews. 101 Chapter 9: Learning From Interviewers. 119 Section I. Interviewer Debriefing . 119 Section II. Structured Post-Interview Evaluation . 124 Chapter 10: Using Record Checks . 137 Chapter 11: Response Analysis Surveys . 151 References. 159 v Part I 1 Chapter 1 Overview I. INTRODUCTION Formulating a series of questions to obtain the answers to a set of data needs may appear to be a relatively simple task; however, constructing a questionnaire that will elicit accurate information from most respondents interviewed is more complicated than it may seem. For example, a seemingly simple question concerning vehicle ownership--How many cars do you own?--may appear to convey all the information necessary for respondents to answer it and to mean the same thing to respondents, survey designers, and data users alike.1 However, upon reflection, such a question is not as clear as it seems. The word "car" may or may not be intended to include such vehicles as vans, campers, motorcycles, tractors, and snowmobiles; "you" may or may not refer to household or family members as well; "own" may or may not include vehicles which are leased or are in the process of being bought. Questionnaire designers need to consider many factors during the process of creating a questionnaire. For example, will every question be interpreted in the same way by most respondents? If not, the data might not provide the information required by the questionnaire designer. Or, for another example, can respondents remember whether or not events of interest to the questionnaire designer have occurred within a given time frame, and if so, can they recall the details of those events accurately? Some generally accepted rules exist for wording, sequencing, and formatting questionnaires and can be used to guide a questionnaire designer in constructing an initial draft of a questionnaire. Yet the development of any particular questionnaire is unique. Refinement is necessary to ensure that any questionnaire used in the field will produce sufficiently accurate results. In the example described above, for instance, testing of the question would reveal the ambiguities inherent in it and lead to the development of a question more likely to meet the data requirements. The purpose of this report is to present a series of tools and tests which are useful in the initial drafting and subsequent refinement of a survey questionnaire, to explain their applicability to questionnaire design, and to describe the mechanics of implementing them. Numerous examples of these techniques are also provided to illustrate the points made. Although the _________________________ 1This example was adapted from one described by Biderman et al. (1982). Principal Contributor: Theresa J. DeMaio 3 4 focus is on survey questionnaires, many of the techniques are applicable to the development of data collection forms for administrative and other purposes. Many of these techniques are relatively simple, inexpensive ways to improve the quality of a questionnaire. For the most part they are appropriate for developing survey questionnaires regardless of the type of information being collected (e.g., factual, behavioral, opinion, or knowledge), the method used to obtain it (e.g., mail, telephone, personal visit, or a combination), or the type of reporting unit (e.g., households, individuals, farms, or estab- lishments). Used appropriately, these techniques should result in more efficient use of resources, reduced respondent burden and nonsampling error, and better realization of a survey's objectives. Maximum effort is justified during the developmental stage, because once a questionnaire is in use, problems are costly or impossible to correct. The time and money spent in developing a questionnaire should be
Details
-
File Typepdf
-
Upload Time-
-
Content LanguagesEnglish
-
Upload UserAnonymous/Not logged-in
-
File Pages461 Page
-
File Size-