Impact Evaluation of Burkina Faso's BRIGHT Program Final Report June 12, 2009 Dan Levy Matt Sloan Leigh Linden Harounan Kazianga Contract Number: Impact Evaluation of Burkina Faso's MCC-05-0192-CFO TO01 BRIGHT Program MPR Reference Number: 6275-003 Final Report Submitted to: Millennium Challenge Corporation June 12, 2009 875 Fifteenth Street, NW Washington, DC 20005-2221 Dan Levy Project Officer: Sophia van der Bijl Matt Sloan Submitted by: Leigh Linden Mathematica Policy Research, Inc. Harounan Kazianga 600 Maryland Avenue, S.W., Suite 550 Washington, DC 20024-2512 Telephone: (202) 484-9220 Facsimile: (202) 863-1763 Project Director: Dan Levy CONTENTS Chapter Page EVALUATION OF THE BRIGHT PROGRAM: EXECUTIVE SUMMARY .......................... IX A. Overview of the Evaluation........................................................................... x 1. Evaluation Design ................................................................................. x 2. Data Collection ................................................................................... xi B. Implementation .......................................................................................... xi C. Impacts ......................................................................................................xii D. Policy Implications ..................................................................................... xiii I INTRODUCTION ................................................................................................. 1 A. Context of Primary Schooling in Burkina Faso .............................................. 1 B. The BRIGHT Program ................................................................................... 4 C. Overview of Evaluation Design ..................................................................... 6 1. Method to Estimate Impacts ................................................................. 6 2. Data Collection Strategy ....................................................................... 7 3. Description of the Sample .................................................................... 7 II DATA COLLECTION .......................................................................................... 10 A. Sample Design .......................................................................................... 10 B. Questionnaire Design ................................................................................ 10 C. Data Collection .......................................................................................... 13 D. Data Editing .............................................................................................. 14 E. Response Rates ......................................................................................... 14 III IMPACT EVALUATION DESIGN ........................................................................... 15 A. Evaluation Questions ................................................................................. 15 B. Process Used to Select Beneficiary Villages ................................................. 15 C. Evaluation Design Selected ........................................................................ 16 D. Assessing the Evaluation Design ................................................................ 19 iv CONTENTS (continued) Chapter Page IV IMPACTS OF BRIGHT......................................................................................... 20 A. Presence of a School in the Village ............................................................. 20 B. Interpretation of Impact Estimates ............................................................. 21 C. Impacts on School Enrollment ..................................................................... 22 D. Impacts on Test Scores .............................................................................. 26 E. Impacts by Gender and Age ......................................................................... 29 F. Other Impact-related Questions ................................................................. 30 1. Which of the interventions’ components were most responsible for the observed impacts? ........................................................................ 30 2. Did parental attitudes toward education change as a result of BRIGHT? ............................................................................................. 31 3. Would the same impacts have been observed had the program built schools of lower quality infrastructure (like the ones typically available in Burkina Faso)? .................................................................. 32 V CONCLUSIONS ................................................................................................. 33 REFERENCES..................................................................................................... 34 APPENDICES 1 COPY OF APPLICATION FORM 2 RULES TO CALCULATE ELIGIBILITY SCORE 3 LIST OF COMMUNITIES WITH ELIGIBILITY SCORES 4 SCHOOL AND HOUSEHOLD QUESTIONNAIRES 5 REQUEST FOR PROPOSAL FOR DATA COLLECTION 6 TRAINING MANUAL USED FOR DATA COLLECTION 7 STATISTICAL ANALYSES TO VERIFY APPROPRIATENESS OF REGRESSION DISCONTINUITY DESIGN 8 STATISTICAL ANALYSES TO VERIFY ROBUSTNESS OF IMPACT ESTIMATES 9 ACCESS VS QUALITY v TABLES Table Page 1 BRIGHT Schools vs. Comparison Schools .....................................................xii 2 Impacts of BRIGHT......................................................................................xii 3 Parents’ Reasons for Enrolling Children in BRIGHT Schools—Participant Group ....................................................................................................... xiii I.1 Evolution of Primary Education Indicators: Burkina Faso, 1970-2005 ........... 2 I.2 Gross Enrollment Rates in Primary Education: West Africa, 2005 ................. 2 I.3 Summary of Village and Household Characteristics ...................................... 8 II.1 Household Questionnaire Respondent Native Language ............................. 12 IV.1 School Characteristics ............................................................................... 21 IV.2 Impacts of Bright on School Enrollment...................................................... 23 IV.3 Impacts of BRIGHT Schools on Self-Reported Enrollment............................. 24 IV.4 Impacts of BRIGHT Schools on Observed Enrollment .................................. 25 IV.5 Impacts of BRIGHT Schools on Test Scores ................................................. 28 IV.6 Effect of BRIGHT Schools on School Enrollment and Test Scores: By Gender ................................................................................................. 29 IV.7 Parents’ Reasons for Enrolling Children in Schools ..................................... 31 IV.8 Main Reason for Not Enrolling Children in School: Parental Reports ............ 31 IV.9 Parental Attitudes toward Schooling: How Many Years Should Child Attend School ............................................................................................ 32 vi FIGURES Figure Page I.1 Average Number of Schools: BRIGHT and non-BRIGHT Provinces ................... 3 I.2 Implementation of the BRIGHT Program ....................................................... 4 III.1 Hypothetical Illustration of Impact Estimation Using the Regression Discontinuity Design ................................................................................. 17 III.2 Presence of a Bright School as a Function of Relative Score ........................ 19 IV.1 Presence of Any School as a Function of Relative Score .............................. 20 IV.2 Impacts of BRIGHT on Self-reported Enrollment .......................................... 22 IV.3 Impacts of BRIGHT on Observed Enrollment ............................................... 23 IV.4 Impact of BRIGHT on Math Test Scores ...................................................... 26 IV.5 Impact of BRIGHT on French Test Scores .................................................... 27 IV.6 Treatment Effects by Age of Child .............................................................. 30 vii ACRONYMS AND DEFINITIONS Bisongo: Child Care Center BRIGHT: Burkinabe Response to Improve Girl’s Chances to Succeed CERFODES: Centre d’Etudes, de Recherches et de Formation pour le Développement Economique et Social CRS: Catholic Relief Services FAWE: Forum for Africa Women Educationalists MCC: Millennium Challenge Corporation MEBA: Ministère de l'Enseignement de Base et de l'Alphabétisation du Burkina Faso (Ministry of Basic Education) PDDEB: Plan Décennal de Développement de l’Education de Base RD: Regression Discontinuity TCP: Threshold Country Program UNICEF: The United Nations Children's Fund USAID: United States Agency for International Development ACKNOWLEDGMENTS This report would not have been possible without the cooperation of several officials. We are particularly grateful to Sophia van der Bijl at the Millennium Challenge Corporation (MCC) for her commitment and guidance throughout the evaluation. We also thank Malik Chaka and Sophia Sahaf at MCC for their help throughout the project, and Franck Wiebe for his insightful comments on the draft reports and presentations. We are also grateful to the many people in Burkina Faso who were instrumental in making this evaluation feasible. We are particularly grateful to Michel Kabore (USAID) for his invaluable help in organizing many meetings and field trips with key decisionmakers and for his strong commitment to