Statistical Policy Working Paper 47 Evaluating Survey Questions: An Inventory of Methods Statistical and Science Policy Office Office of Information and Regulatory Affairs Office of Management and Budget January 2016 THE FEDERAL COMMITTEE ON STATISTICAL METHODOLOGY January 2016 MEMBERS Jonaki Bose (Chair) Peter Meyer (CDAC Chair) Substance Abuse and Mental Health Services National Center for Health Statistics Administration Peter Miller Nancy Bates U.S. Census Bureau U.S. Census Bureau Renee Miller Patrick Cantwell Energy Information Administration U.S. Census Bureau Jeri Mulrow Chris Chapman National Science Foundation National Center for Education Statistics Amy O’Hara John Eltinge U.S. Census Bureau Bureau of Labor Statistics Polly Phipps Dennis Fixler Bureau of Labor Statistics Bureau of Economic Analysis Nathaniel Schenker Barry Graubard National Center for Health Statistics National Cancer Institute Rolf Schmitt Arthur B. Kennickell Bureau of Transportation Statistics Federal Reserve Board Marilyn Seastrom Jennifer Madans National Center for Education Statistics National Center for Health Statistics Joy Sharp Rochelle (Shelly) Martinez Bureau of Transportation Statistics Office of Management and Budget Katherine K. Wallman (Champion) Jaki McCarthy Statistical and Science Policy National Agriculture Statistic Services Office of Management and Budget Pam McGovern (Webmaster) G. David Williamson National Agriculture Statistic Services Agency for Toxic Substances and Disease Registry Grace Medley (Secretary) Substance Abuse and Mental Health Services Gordon Willis Administration National Cancer Institute 1 Statistical Policy Working Paper 47 Evaluating Survey Questions: An Inventory of Methods Prepared by Subcommittee on Questionnaire Evaluation Methods Statistical and Science Policy Office Office of Information and Regulatory Affairs Office of Management and Budget January 2016 Evaluating Survey Questions: An Inventory of Methods A Report of the Subcommittee on Questionnaire Evaluation Methods January 2016 Washington, D.C. Members of the Subcommittee on Questionnaire Evaluation Methods Jennifer Madans, Chair Kristen Miller National Center for Health Statistics National Center for Health Statistics Lionel Deang Rebecca L. Morrison Social Security Administration National Science Foundation Jennifer Edgar Cleo Redline Bureau of Labor Statistics National Center for Education Statistics Scott Fricker Paul Scanlon Bureau of Labor Statistics National Center for Health Statistics Patricia Goerman Kristin Stettler U.S. Census Bureau U.S. Census Bureau Kashka Kubzdela Diane K. Willimack National Center for Education Statistics U.S. Census Bureau Jaki McCarthy Stephanie Willson National Agricultural Statistics Service National Center for Health Statistics Kathy Ott National Agricultural Statistics Service , , Evaluating Survey Questions: An Inventory of Methods Contents Introduction ................................................................................................................................................................... 1 Question Development and Evaluation Methods (QDEM) ...................................................................................... 1 Question Development ............................................................................................................................................. 2 Question Development Methods........................................................................................................................... 4 Development using Respondents Actively ........................................................................................................... 4 Development using Respondents Passively .......................................................................................................... 6 Question Evaluation .................................................................................................................................................. 7 Data Collection Vehicles .......................................................................................................................................... 8 Combining Multiple Methods ................................................................................................................................. 10 Question Evaluation Methods ..................................................................................................................................... 12 Expert Reviews (Methodological) .......................................................................................................................... 12 Cognitive interviews ............................................................................................................................................... 14 Embedded Probing/Web Probing ........................................................................................................................... 16 Vignette Studies and Fictional Scenarios ............................................................................................................... 17 Usability Testing ..................................................................................................................................................... 19 Feedback from Survey Personnel ........................................................................................................................... 21 Respondent Debriefing ........................................................................................................................................... 22 Response Analysis Surveys .................................................................................................................................... 23 Randomized Experiments ....................................................................................................................................... 25 Validation Studies ................................................................................................................................................... 27 Re-interview / Content Evaluation ......................................................................................................................... 28 Analysis of Paradata ............................................................................................................................................... 30 Response Quality Indicators ................................................................................................................................... 33 Item Response Theory ............................................................................................................................................ 36 Latent Class Analysis ............................................................................................................................................. 37 Attachment A: Multi-Method Examples ................................................................................................................... 40 Multi-Method Questionnaire Evaluation and Testing for the Census of Agriculture ............................................. 40 Multi-Method Questionnaire Development and Evaluation for National Postsecondary Student Aid and Beginning Postsecondary Student Longitudinal Studies ........................................................................................ 42 Multi-Method Questionnaire Development and Evaluation of a Sexual Identity Question ................................... 44 Evaluating Survey Questions: An Inventory of Methods Introduction This document was developed by the Federal Committee on Statistical Methodology (FCSM) Subcommittee on Question Evaluation Methodology to provide an inventory of methods used by federal statistical agencies to evaluate survey questions. Since question development methods play a crucial role in assuring the quality of survey questions and the data they produce, this document also includes a section on developing survey questions which outlines important steps in creating the survey questions that will subsequently be tested and evaluated. The question development section is followed by a discussion of question evaluation, a description of data collection methods, and a list of the methods federal agencies use to evaluate survey questions, along with a definition, common uses, strengths, and limitations for each evaluation method. Question Development and Evaluation Methods (QDEM) The many techniques employed to assure that survey questions evoke the intended response and to evaluate question quality come under the broad umbrella label of “question development and evaluation methods” (QDEM). This includes assuring: (1) that the survey questions ask what the researcher intended, (2) that they are consistently understood as intended, (3) that respondents are willing and able to consistently answer the questions as intended, and (4) that providing the full response is not overly burdensome (Groves et al., 2009). This process involves both development and evaluation of the questions (Figure 1). While conceptually distinct, the development and evaluation processes become intertwined in practice. Question development involves identifying and defining the concepts to be measured, and drafting the questions to measure these concepts. After questions have been developed, they should be assessed using one or more of the evaluation methods defined in this inventory. The process should be iterative, where questions