THE EFFECT OF INTERVIEWER CHARACTERIStiCS ON DATA QUALitY IN DHS SURVEYS DHS METHODOLOGICAL REPORTS 24 SEPTEMBER 2018 This publication was produced for review by the United States Agency for International Development. It was prepared by Thomas W. Pullum, Christina Juan, Nizam Khan, and Sarah Staveteig. DHS Methodological Reports No. 24 The Effect of Interviewer Characteristics on Data Quality in DHS Surveys Thomas W. Pullum1,2 Christina Juan1,2 Nizam Khan1 Sarah Staveteig2,3 ICF Rockville, Maryland, USA September 2018 1 ICF 2 The DHS Program 3 Avenir Health Corresponding author: Thomas W. Pullum, International Health and Development, ICF, 530 Gaither Road, Suite 500, Rockville, MD 20850, USA; phone: +1 301-572-0950; fax: +1 301-572-0999; email: [email protected] Acknowledgments: The authors wish to thank Sunita Kishor and survey managers on the DHS staff for valuable insights into the interview process. We thank Yoonjoung Choi for very helpful comments on an earlier draft. Editor: Bryant Robey Document Production: Joan Wardell This study was carried out with support provided by the United States Agency for International Development (USAID) through The DHS Program (#AID-OAA-C-13-00095). The views expressed are those of the authors and do not necessarily reflect the views of USAID or the United States Government. The DHS Program assists countries worldwide in the collection and use of data to monitor and evaluate population, health, and nutrition programs. Additional information about The DHS Program can be obtained from ICF, 530 Gaither Road, Suite 500, Rockville, MD 20850 USA; telephone: +1 301-572-0200, fax: +1 301-572-0999, email: [email protected], internet: www.DHSprogram.com. Recommended citation: Pullum, Thomas W., Christina Juan, Nizam Khan, and Sarah Staveteig. 2018. The Effect of Interviewer Characteristics on Data Quality in DHS Surveys. DHS Methodological Reports No. 24. Rockville, Maryland, USA: ICF. CONTENTS TABLES AND LISTS .................................................................................................................................... v FIGURES .................................................................................................................................................... vii PREFACE .................................................................................................................................................... xi ABSTRACT................................................................................................................................................ xiii 1 BACKGROUND ............................................................................................................................... 1 1.1 Related Research on Interviewer Effects ........................................................................... 1 1.2 Previous DHS Research on Data Quality ........................................................................... 2 1.3 Objectives and Structure of This Report ............................................................................. 4 2 INTERVIEWER CHARACTERISTICS ............................................................................................. 5 2.1 Data ..................................................................................................................................... 5 2.2 Interviewer Characteristics and Survey Work Experience .................................................. 6 2.3 Interviewer Workloads in the Women’s Survey .................................................................. 9 2.4 Interviewer Workloads in the Household Survey .............................................................. 12 3 METHODS TO IDENTIFY INTERVIEWER EFFECTS .................................................................. 15 4 NONRESPONSE AND REFUSALS .............................................................................................. 21 4.1 Indicators ........................................................................................................................... 21 4.2 Identifying Problematic Surveys and Indicators ................................................................ 22 4.3 Concentration of Problematic Interviews .......................................................................... 27 4.4 Associations with Interviewer Characteristics ................................................................... 31 4.5 Sex Effects on Nonresponse for the Individual Interviews................................................ 34 5 UNDER-5 MORTALITY ................................................................................................................. 39 5.1 Indicators ........................................................................................................................... 39 5.2 Identifying Problematic Surveys and Indicators ................................................................ 40 5.3 Concentrations of Problematic Interviewers ..................................................................... 43 5.4 Associations with Interviewer Characteristics ................................................................... 46 6 REPORTING OF AGES AND DATES .......................................................................................... 51 6.1 Indicators ........................................................................................................................... 51 6.2 Identifying Problematic Surveys and Indicators ................................................................ 54 6.3 Measuring the Concentration of Problematic Interviewers ............................................... 60 6.4 Associations with Interviewer Characteristics ................................................................... 69 7 DISCUSSION AND CONCLUSIONS ............................................................................................ 83 REFERENCES ............................................................................................................................................ 89 APPENDIX 1 TABLES TO ACCOMPANY CHAPTER 2 .......................................................................... 93 APPENDIX 2 GLM MODELS AND RESIDUAL DEVIANCE .................................................................... 97 APPENDIX 3 TABLES FOR COVARIATE MODELS ............................................................................... 99 iii TABLES AND LISTS Table 2.1 Survey types and countries included in this study ................................................. 6 Table 2.2 Percentage distribution of interviewers by interviewer characteristics and country ................................................................................................................... 7 Table 2.3 Length of fieldwork and total number of interviewers and interviews by country, for the survey of women ......................................................................... 10 Table 2.4 Percentage distribution of interviewers by work load and country (Women’s Survey) ................................................................................................................. 11 Table 2.5 Length of fieldwork and total number of interviewers and interviews by country, for the household survey ........................................................................ 12 Table 2.6 Percentage distribution of interviewers by workload and country (Household Survey) ................................................................................................................. 13 List 4.1 Combinations of country and indicator for which variation across interviewers is significant at the .01 level and the prevalence is above 2%. “Chi2 ratio” is the ratio of the calculated chi-square to the critical value for a .01 test of significance. ......................................................................................................... 25 List 4.2 Interviewer-level distribution of women’s refusals for height, weight, and/or hemoglobin measurement in the Armenia survey, with deviance residuals. ....... 30 List 5.1 Combinations of country and indicator for which variation across interviewers is significant at the .01 level. “Chi2 ratio” is the ratio of the calculated chi-square to the critical value for a .01 test of significance. ............................... 41 List 5.2 Interviewer-level distribution of reported deaths in month 0 in the Timor-Leste survey, with deviance residuals. Interviewers with dr in the range -1.96 to +1.96 are omitted. ................................................................................................ 46 List 6.1 Interviewer-level distribution of incompleteness of women’s age in the Kenya survey, with deviance residuals. Interviewers with dr<=1.96 are omitted. ................................................................................................................ 62 List 6.3 Interviewer-level distribution of heaping of age at final digits 0 or 5 in the Nigeria household survey, with deviance residuals. Interviewers with dr<=1.96 are omitted. ........................................................................................... 65 List 6.4 Interviewer-level distribution of the indicator of downward displacement of women’s age in the Sierra Leone survey, with deviance residuals. Interviewers with dr>=-1.96 or dr<=1.96 are omitted. .......................................... 68 List 6.5 Interviewer-level distribution of the indicator of over-dispersion of children’s ages in the Timor-Leste