The Future of Survey Research: Challenges and Opportunities A Report to the National Science Foundation Based on Two Conferences Held on October 3-4 and November 8-9, 2012 Report Presented By The National Science Foundation Advisory Committee for the Social, Behavioral and Economic Sciences Subcommittee on Advancing SBE Survey Research: Jon A. Krosnick – Stanford University (chair) Stanley Presser – University of Maryland Kaye Husbands Fealing – University of Minnesota Steven Ruggles – University of Minnesota Project Coordinator: David L. Vannette – Stanford University When the National Science Foundation Advisory Committee for the Social, Behavioral and Economic Sciences formed the subcommittee on Advancing SBE Survey Research, Janet Harkness was a member. We dedicate this report to her memory, to recognize her contribution to this subcommittee and the survey research community and the work of the National Science Foundation. Any opinions, findings, conclusions or recommendations presented in this material are only those of the authors; and do not necessarily reflect the views of the National Science Foundation. May, 2015 Table of Contents TABLE OF CONTENTS ....................................................................................................................................... 2 OVERVIEW ............................................................................................................................................................. 4 SECTION 1 – BEST PRACTICES FOR SURVEY RESEARCH .................................................................................................. 6 SECTION 2 – RECOMMENDATIONS ............................................................................................................... 12 CONCLUDING REMARKS – WAYS FORWARD FOR NSF ........................................................................................... 13 REFERENCES: .................................................................................................................................................................... 14 APPENDIX: ........................................................................................................................................................... 18 DOCUMENTATION OF CONFERENCES ................................................................................................. 18 BACKGROUND ................................................................................................................................................... 18 PRESENTERS ...................................................................................................................................................... 22 SUMMARIES OF PRESENTATIONS ........................................................................................................... 34 SECTION 1: CONVENTIONAL SURVEY RESEARCH .................................................................................................... 34 Reasons for Optimism about the Accuracy of Survey Research – Jon A. Krosnick ................................... 34 Probability vs. Non-probability Sampling Methods – Gary Langer ................................................................ 35 Sampling for Single and Multi-Mode Surveys using Address-based Sampling - Colm O’Muircheartaigh ................................................................................................................................................................. 39 The Impact of Survey Nonresponse on Survey Accuracy – Scott Keeter ........................................................ 42 Optimizing Response Rates – J. Michael Brick ........................................................................................................ 44 Modes of Data Collection – Roger Tourangeau ....................................................................................................... 47 The Use and Effects of Incentives in Surveys – Eleanor Singer ......................................................................... 53 Building Household Rosters Sensibly – Kathleen T. Ashenfelter ...................................................................... 59 Proxy Reporting – Curtiss Cobb ..................................................................................................................................... 64 Improving Question Design to Maximize Reliability and Validity – Jon A. Krosnick .............................. 67 Perception of Visual Displays and Survey Navigation – Stephen Kosslyn .................................................... 71 Cognitive Evaluation of Survey Instruments: State of the Science and Future Directions – Gordon Willis ........................................................................................................................................................................................... 74 Survey Interviewing: Deviations from the Script – Nora Cate Schaeffer ...................................................... 77 Challenges and Opportunities in Open-Ended Coding – Arthur Lupia ......................................................... 85 What Human Language Technology can do for you (and vice versa) – Mark Liberman ....................... 90 Confidentiality and Anonymity – Roger Tourangeau ............................................................................................. 93 Respondent Attrition vs. Data Attrition and Their Reduction - Randall J. Olsen ....................................... 97 Computation of Survey Weights – Matthew DeBell ................................................................................................ 99 SECTION 2: OPPORTUNITIES TO EXPAND DATA COLLECTION .......................................................................... 102 Paradata – Frauke Kreuter ............................................................................................................................................ 102 Interviewer Observations – Brady T. West .............................................................................................................. 106 Leave-behind Measurement Supplements – Michael W. Link ......................................................................... 109 Experience Sampling and Ecological Momentary Assessment – Arthur A. Stone .................................. 110 Biomarkers in Representative Population Surveys – David Weir .................................................................. 114 Specialized Tools for Measuring Past Events – Robert Belli ........................................................................... 117 SECTION 3: LINKING SURVEY DATA WITH EXTERNAL SOURCES .................................................................... 121 Linking Survey Data to Official Government Records – Joseph W. Sakshaug ......................................... 121 Linking Knowledge Networks Web Panel Data with External Data – Josh Pasek ................................. 123 2 Linking Survey Data with the Catalist Commercial Database – Bob Blaemire ...................................... 125 Election Administration Data – Michael P. McDonald ..................................................................................... 126 Challenges with Validating Survey Data – Matthew K. Berent ...................................................................... 128 Improving Government, Academic and Industry Data-Sharing Opportunities – Robert Groves ..... 130 SECTION 4: IMPROVING RESEARCH TRANSPARENCY AND DATA DISSEMINATION ................................... 132 Data Curation – Steven Ruggles .................................................................................................................................. 132 Evaluating the Usability of Survey Project Websites – David L. Vannette ................................................ 135 Research Transparency and the Credibility of Survey-Based Social Science - Arthur Lupia............ 139 CONFERENCE WEBSITE AND ORGANIZER CONTACT INFORMATION .............................. 142 LIST OF CONFERENCE PRESENTERS .................................................................................................. 143 LIST OF CONFERENCE DISCUSSANTS ................................................................................................. 144 CONFERENCE PROGRAMS ........................................................................................................................ 145 3 Overview For more than thirty years, the National Science Foundation (NSF) has supported data for research on a wide variety of topics by making awards to three major long-term survey efforts, the American National Elections Studies (ANES), the Panel Study of Income Dynamics (PSID), and the General Social Survey (GSS). In February 2012, the Advisory Committee for the Social, Behavioral, and Economic Sciences (SBE) was asked to provide advice about future investments in these surveys and others. The Advisory Committee then charged a subcommittee to provide that advice. The Subcommittee on Advancing SBE Survey Research is comprised of Jon Krosnick (Stanford University, chair), Janet Harkness (University of Nebraska, deceased), Kaye Husbands-Fealing (University of Minnesota), Stanley Presser (University of Maryland), and Steven Ruggles (University of Minnesota). The Subcommittee submits this report to the Assistant Director of the SBE Directorate, with the purpose of providing advice related to how the Foundation can best use its resources to support research through survey