FINAL REPORT Evaluation Design Report for the Benin Power Compact's Electricity Generation Project and Electricity Distribution Project March 7, 2018 Christopher Ksoll Kristine Bos Arif Mamun Anthony Harris Sarah Hughes Submitted to: Millennium Challenge Corporation 1099 14th Street, NW Suite 700 Washington, DC 20005 Project Monitor: Hamissou Samari Contract Number: MCC-16-CON-0058 Submitted by: Mathematica Policy Research 1100 1st Street, NE, 12th Floor Washington, DC 20002-4221 Telephone: (202) 484-9220 Facsimile: (202) 863-1763 Project Director: Sarah Hughes Reference Number: 50339.01.200.031.000 This page has been left blank for double-sided copying. BENIN ENERGY II: EVALUATION DESIGN REPORT MATHEMATICA POLICY RESEARCH CONTENTS ACRONYMS ............................................................................................................................................... VII I. INTRODUCTION ........................................................................................................................................ 1 II. OVERVIEW OF THE COMPACT, GENERATION ACTIVITY, AND DISTRIBUTION ACTIVITY ......................................................................................................................................... 5 A. Overview of the Benin Power Compact ..................................................................................... 5 B. Overview of the theory of change .............................................................................................. 9 C. Economic rate of return and beneficiary analysis.................................................................... 10 III. LITERATURE REVIEW.................................................................................................................. 13 A. Grid-level outcomes ................................................................................................................. 15 1. Increased supply of electricity ........................................................................................... 15 2. Improved electricity quality and reliability.......................................................................... 15 B. Business outcomes ................................................................................................................. 16 1. Longer hours of operation for businesses ........................................................................ 16 2. Lower reliance on costlier energy sources........................................................................ 16 3. Reduced degradation of equipment due to unstable electricity ........................................ 17 4. Reduced losses of products and perishable goods .......................................................... 17 5. Increased productivity for businesses ............................................................................... 17 C. Household outcomes ............................................................................................................... 17 1. Lower reliance on costlier energy sources........................................................................ 17 2. Improved education outcomes and increased productivity for households ...................... 18 D. Gaps in the literature and contribution of the Benin II Energy evaluation ............................... 19 IV. EVALUATION DESIGN.................................................................................................................. 21 A. Evaluation questions ............................................................................................................... 21 B. Overview of the evaluation strategy ........................................................................................ 22 V. QUANTITATIVE ANALYSIS OF GRID-LEVEL OUTCOMES ........................................................ 27 A. Methods ................................................................................................................................... 27 1. Interrupted time-series (ITS) approach. ............................................................................ 27 2. Pre-post and descriptive analysis ..................................................................................... 29 B. Outcomes and their anticipated time frame for realization ...................................................... 30 1. Short-term outcomes......................................................................................................... 30 2. Medium- and longer-term outcomes ................................................................................. 32 C. Analysis plan and statistical power .......................................................................................... 32 FINAL iii BENIN ENERGY II: EVALUATION DESIGN REPORT MATHEMATICA POLICY RESEARCH 1. ITS approach ..................................................................................................................... 32 2. Pre-post analysis ............................................................................................................... 35 D. Data sources............................................................................................................................ 36 1. Placement and measurement on a simple network ............................................................ 36 2. Additional considerations for grid monitoring .................................................................... 38 3. Data collection ................................................................................................................... 40 E. Challenges ............................................................................................................................... 41 VI. QUANTITATIVE ANALYSIS OF END-USER OUTCOMES .......................................................... 43 A. Methods ................................................................................................................................... 43 1. ITS approach ..................................................................................................................... 43 2. Pre-post approach with longitudinal data .......................................................................... 43 B. Outcomes and their anticipated time frame for realization ...................................................... 44 1. Short-term outcomes......................................................................................................... 44 2. Longer-term outcomes ...................................................................................................... 45 C. Analysis plan and statistical power .......................................................................................... 46 1. ITS analysis ....................................................................................................................... 46 2. Pre-post analysis with longitudinal data ............................................................................ 47 D. Data sources............................................................................................................................ 48 1. Sampling ........................................................................................................................... 50 2. Data collection approach .................................................................................................. 51 E. Challenges ............................................................................................................................... 52 VII. QUALITATIVE ANALYSIS .................................................................................................................... 55 A. Methodology ............................................................................................................................ 55 1. Implementation analysis ................................................................................................... 55 2. Qualitative assessment ..................................................................................................... 56 B. Time frame of exposure ........................................................................................................... 56 C. Study sample ........................................................................................................................... 57 D. Approach to data collection ..................................................................................................... 59 E. Analysis plan............................................................................................................................ 62 F. Challenges ............................................................................................................................... 63 VIII. ADMINISTRATIVE ISSUES ........................................................................................................... 65 A. Summary of IRB requirements and clearances....................................................................... 65 B. Data access, privacy, and file preparation .............................................................................. 65 FINAL iv BENIN ENERGY II: EVALUATION DESIGN REPORT MATHEMATICA POLICY RESEARCH C. Dissemination plan .................................................................................................................. 66 D. Evaluation team: Roles and responsibilities ...........................................................................