Niger IMAGINE Long-Term Evaluation
Total Page:16
File Type:pdf, Size:1020Kb
FINAL REPORT Niger IMAGINE Long-Term Evaluation June 6, 2016 Emilie Bagby Anca Dumitrescu Cara Orfield Matt Sloan Submitted to: Millennium Challenge Corporation 1099 14th Street NW Suite 700 Washington, DC 20005 (202) 521-3600 Project Officer: Carolyn Perrin Contract Number: MCC-10-0114-CON-20-TO08 Submitted by: Mathematica Policy Research 1100 1st Street, NE 12th Floor Washington, DC 20002-4221 Telephone: (202) 484-9220 Facsimile: (202) 863-1763 Project Director: Matt Sloan Reference Number: 40038.530 This page has been left blank for double-sided copying. ACKNOWLEDGEMENTS This report reflects the combined efforts of many people, including our current Millennium Challenge Corporation (MCC) project officer, Mike Cooper, our previous MCC project officers Sophia van der Bijl and Amanda Moderson-Cox, and Jennifer Gerst, Jennifer Sturdy and Malik Chaka, also at MCC, who together provided us guidance and support throughout the project. This study would not have been possible without the contributions of our Niger Education and Community Strengthening (NECS) and IMAGINE project partners. We would first like to acknowledge the wide range of Niger Threshold Program implementers and coordinators who generously shared their time and attention to help improve the quality, comprehensiveness, and depth of the study. We are grateful to Government of Niger staff at the Ministry of Education and the National Institute of Statistics for providing important feedback on the survey instrument and data collection plan, as well as providing feedback to the report content. We also received indispensable advice and support from several staff at USAID, especially Jennifer Swift-Morgan. This report depended on contributions from a wide range of data collection, supervisory, and support staff. We are grateful to the Centre International d’Etudes et de Recherches sur les Populations Africaines (CIERPA) for the successful implementation of the nationwide survey data collection effort. We would also like to thank the many people who responded to our surveys. At Mathematica Policy Research, Dan Levy and Evan Borkum provided technical input and useful comments on the analysis plan and draft report. Chris Rankin provided technical input on the analysis. Mark Beardsley delivered timely and detail-oriented programming assistance to help clean and process the survey data. We would also like to thank the editorial and administrative support staff at Mathematica, as well as our colleagues who assisted with translation to French. The opinions, conclusions, and any errors in this report are the sole responsibility of the authors and do not reflect the official views of MCC or Mathematica. iii This page has been left blank for double-sided copying. CONTENTS EXECUTIVE SUMMARY .............................................................................................................................. ix A. Overview of the evaluation ........................................................................................................ x 1. Evaluation design ................................................................................................................ x 2. Data collection .....................................................................................................................xi B. Sustainability of infrastructure ................................................................................................... xi C. Impacts ...................................................................................................................................... xi D. Conclusion ............................................................................................................................... xiii I. INTRODUCTION .............................................................................................................................. 1 A. Primary schooling context in Niger ............................................................................................ 1 B. Overview of the short-term impacts of IMAGINE ...................................................................... 4 C. Long-term impact evaluation of IMAGINE ................................................................................. 5 II. OVERVIEW OF IMAGINE................................................................................................................ 7 A. Project logic ............................................................................................................................... 8 B. Implementation summary .......................................................................................................... 9 III. EVIDENCE GAPS THAT THE LONG-TERM IMAGINE EVALUATION FILLS ............................. 11 IV. IMPACT EVALUATION DESIGN ................................................................................................... 13 A. Evaluation questions ............................................................................................................... 13 B. Methodology ............................................................................................................................ 15 1. Random assignment ......................................................................................................... 15 2. Estimation strategy ............................................................................................................ 17 3. Assessing the evaluation design ....................................................................................... 19 4. Assessing the generalizability of results ........................................................................... 23 5. Sampling strategy and power calculations ....................................................................... 24 C. Data collection strategy ........................................................................................................... 26 1. Survey instruments ........................................................................................................... 27 2. Description of the sample ................................................................................................. 29 V. IMPACT FINDINGS ....................................................................................................................... 31 A. Estimated impact on key outcomes and pathways through which effects (or lack thereof) are explained .............................................................................................................. 31 1. Impacts on school availability and functionality of school infrastructure ........................... 31 2. Impacts on school enrollment and absenteeism ............................................................... 33 v CONTENTS MATHEMATICA POLICY RESEARCH 3. Impacts on test scores ...................................................................................................... 35 4. Subgroup analysis by gender ........................................................................................... 36 5. Subgroup analysis by socioeconomic status .................................................................... 39 B. Other impact-related questions ............................................................................................... 41 1. Did impacts of the project vary by age? ............................................................................ 41 2. Did parental attitudes toward education change as a result of IMAGINE? ....................... 41 3. What are the main factors affecting a parent’s decision to send his or her child to school? .............................................................................................................................. 43 4. What are project impacts on other child outcomes related to education? ........................ 47 C. Robustness of results .............................................................................................................. 47 1. Sensitivity of results to different regression specifications................................................ 47 2. Sensitivity of results to weights ......................................................................................... 51 3. Sensitivity of results to the sample specification ............................................................... 53 4. Estimates of treatment effect on the treated villages ........................................................ 57 5. Estimates of treatment effect on in-school children .......................................................... 58 D. Alternative explanations .......................................................................................................... 60 1. Threats to the design ........................................................................................................ 60 2. Threats to the timing of measurement .............................................................................. 61 VI. CONCLUSIONS ............................................................................................................................. 65 A. Comparison to one-year follow-up IMAGINE evaluation ......................................................... 65 B. Policy implications ................................................................................................................... 67 VII. NEXT STEPS AND FUTURE ANALYSIS ...................................................................................... 69 A. Dissemination