Falsification in Surveys
Total Page:16
File Type:pdf, Size:1020Kb
FALSIFICATION IN SURVEYS Prepared for AAPOR Council and the Executive Committee of the American Statistical Association by the members of the Task Force on Data Falsification: Jill M. DeMatteis, Westat, Task Force Co-Chair Linda J. Young, National Agricultural Statistics Service, Task Force Co-Chair James Dahlhamer, National Center for Health Statistics Ronald E. Langley, University of Kentucky Joe Murphy, RTI International Kristen Olson, University of Nebraska-Lincoln Sharan Sharma, University of Maryland This report was commissioned by the AAPOR Executive Council as a service to the profession. The report was reviewed and accepted by AAPOR Executive Council. The opinions expressed in this report are those of the author(s) and do not necessarily reflect the views of the AAPOR Executive Council. The authors, who retain the copyright to this report, grant AAPOR a non-exclusive perpetual license to the version on the AAPOR website and the right to link to any published versions. The task force gratefully acknowledges the contributions of Mario Callegaro, Michael Larsen, and Yan Li, who contributed to early discussions and development of the framework for this report. We also thank Rosalynn Yang, who conducted a comprehensive literature review. We also acknowledge helpful comments from AAPOR Executive Council members, ASA Executive Committee members, and their designees. The task force also gratefully acknowledges the editorial support provided by Westat. September 2020 Table of Contents Chapter Page 1 Introduction ........................................................................................................ 1 2 Types of Fabrication and Falsification ............................................................ 4 2.1 Introduction ........................................................................................... 4 2.2 Types of Falsification Based on Steps of the Survey Process .................................................................................................... 5 2.2.1 Counting and Listing (Interviewer, Field Supervisor) ............................................................................. 6 2.2.2 Identifying and Contacting the Sample Unit (Interviewer) .......................................................................... 7 2.2.3 Household Rostering and Screening (Interviewer) .......................................................................... 8 2.2.4 Conducting the Interview (Interviewer, Field Supervisor) ............................................................................. 9 2.2.5 Finalizing the Case (Interviewer, Field Supervisor) ............................................................................. 14 2.2.6 Recontact/Reinterview (Field Supervisor, Other Field Staff) ............................................................................. 15 2.2.7 Data Processing/Aggregation (Data Entry Staff, Research Staff) ...................................................................... 15 3 Preventing Falsification ..................................................................................... 18 3.1 Introduction ........................................................................................... 18 3.2 Organizational Factors ......................................................................... 19 3.2.1 Organizational Culture ......................................................... 19 3.2.2 Research Ethics and Other Human Subjects Training .................................................................................. 21 3.3 Preventing Falsification: Study Design Factors ................................ 22 3.4 Preventing Falsification: Personnel Factors ...................................... 23 3.4.1 Hiring Practices ..................................................................... 23 3.4.2 Outside Stressors .................................................................. 24 3.4.3 Interviewer Pay and Pay Structure ..................................... 24 3.4.4 Length of Employment ....................................................... 25 3.5 Preventing Falsification: Methods Used Before Data Collection ............................................................................................... 25 Contents (continued) Chapter Page 3.6 Preventing Falsification: Methods Used During Data Collection That May Be a Deterrent .................................................. 27 3.6.1 Monitoring Interviews ......................................................... 27 3.6.2 CARI ....................................................................................... 27 3.6.3 Paradata Analyses ................................................................. 28 3.6.4 Verification Interviews/Reinterviews ................................ 29 3.6.5 GPS ......................................................................................... 29 3.7 Preventing Falsification: Methods Used as Consequences That May Be a Deterrent ..................................................................... 30 4 Detecting Falsification ....................................................................................... 31 4.1 The Need for and Challenges of Detection ...................................... 31 4.2 Methods/Technology Available for Detection ................................ 32 4.2.1 Methods That Involve Review of the Interview “Process” ............................................................................... 32 4.2.2 Statistical Detection Methods by Reviewing “Outputs” .............................................................................. 38 4.2.3 Using Methods/Technologies in Combination ............... 47 4.3 Fitting the Methods to the Survey at Hand....................................... 49 5 Impacts of Falsification on Study Results ....................................................... 50 5.1 Introduction ........................................................................................... 50 5.2 Approach ................................................................................................ 51 5.3 Impact of Falsification on Exploratory Data Analysis (EDA) ..................................................................................................... 52 5.4 Impact of Falsification on the Sample Mean for a Continuous Variable ............................................................................. 54 5.5 Impact of Falsification on the Precision of the Sample Mean for a Continuous Variable ......................................................... 57 5.6 Impact of Falsification on a Sample Proportion and Its Precision ................................................................................................. 61 5.7 Impact of Falsification on the Regression Slope and Its Precision for Simple Linear Regression ............................................. 62 5.8 Conclusion ............................................................................................. 66 Contents (continued) Chaper Page 6 Existing Organizational Guidelines and Policies ........................................... 68 6.1 Introduction ........................................................................................... 68 6.2 Review of Existing Professional Codes of Ethics and Practices .................................................................................................. 68 6.3 Description of Process Used to Obtain Information from Survey Research Organizations ........................................................... 70 6.4 Preventing Falsification ........................................................................ 72 Workplace Culture and Practices ........................................................ 73 Institutional-Level Training ................................................................. 74 6.5 Detecting Falsification .......................................................................... 74 During Data Collection ........................................................................ 75 Post-Data Collection ............................................................................ 76 6.6 Evidence of Success ............................................................................. 76 7. Summary and Recommendations ..................................................................... 77 References ............................................................................................................................... R-1 Appendixes A Letter Used for SRO Request for Guidelines ................................................ A-1 B Questionnaire Used for SRO Request for Guidelines .................................. B-1 C Survey Methods and Results ............................................................................. C-1 D Charge of the Task Force .................................................................................. D-1 Contents (continued) Tables Page 5-1 Simulation scenarios ........................................................................................... 59 6-1 Characteristics of SROs responding to task force request for information on policies, guidelines, and procedures ..................................... 72 Figures 5-1 Comparison of the distribution of true values for a variable (left panel) with distributions where 5 percent and 10 percent values have been falsified (center and right panels). Falsified values mask the bimodal nature of the true distribution. ................................................... 53 5-2 Scatterplots