EVALUATION DESIGN FOR EVALUATION SERVICES IN SUPPORT OF THE BENIN II OFF-GRID ENERGY ACCESS PROJECT Submitted: February 27, 2019 The views and opinions expressed herein are those of the author(s) and do not necessarily represent those of MCC or any other U.S. Government entity. This page left intentionally blank. EVALUATION DESIGN FOR EVALUATION SERVICES IN SUPPORT OF THE BENIN II OFF-GRID ENERGY ACCESS PROJECT Submitted: February 27, 2019 Submitted to: Millennium Challenge Corporation 875 Fifteenth Street, NW Washington, DC 20005-2221 Contracting Officer’s Representative: Shreena Patel Contract Number 95332418C0281 Submitted by: Social Impact, Inc. 2300 Clarendon Blvd., Suite 1000 Arlington, VA 22201 703.465.1884 www.socialimpact.com i This page left intentionally blank. ii TABLE OF CONTENTS Tables ............................................................................................................................. v Figures ........................................................................................................................... v Acronyms ...................................................................................................................... vi 1.0 Introduction & Background .................................................................................... 1 1.1 Country Context ........................................................................................................................................................... 1 1.2 Objectives of the Report ........................................................................................................................................... 5 2.0 Presentation of the Benin Compact and Interventions to be Evaluated ............ 6 2.1 Summary Overview of the Compact and the Interventions Evaluated ................................................. 6 2.1.1 Enabling Environment for Off-Grid Electricity ....................................................................................... 6 2.1.2 Off-Grid Clean Energy Facility ....................................................................................................................... 7 2.2 Overview of Compact Theory of Change ............................................................................................................ 9 2.3 Status of Implementation to Date ....................................................................................................................... 14 3.0 Literature review in relation to theory of change ............................................... 15 3.1 Existing Literature .................................................................................................................................................... 15 3.1.1. The uptake and effects of energy access interventions ................................................................... 15 3.1.2. Facility-based development interventions ........................................................................................... 21 3.2 Gaps in Literature ...................................................................................................................................................... 22 3.3 Policy Relevance of This Evaluation .................................................................................................................. 23 3.4 Discussion of Theory of Change .......................................................................................................................... 24 4.0 Evaluation Design ................................................................................................. 29 4.1 Evaluability and Brief Overview of Approach ............................................................................................... 29 4.1.1 Evaluability Assessment ................................................................................................................................ 29 4.1.2. Overview of Evaluation Design .................................................................................................................. 30 4.2. Evaluation of the Off-Grid Enabling Environment and Facility ............................................................. 30 4.2.1. Major Evaluation Questions ........................................................................................................................ 30 4.2.2. Process/Performance Evaluation Design .............................................................................................. 32 4.3. Impact Evaluation of the Off-Grid Investments ........................................................................................... 36 4.3.1. Major Evaluation Questions ........................................................................................................................ 36 4.3.2 Grant-Level Impact Evaluation ................................................................................................................... 37 4.4 Quantitative Data Collection and Analysis ...................................................................................................... 57 iii 4.4.1 Desk Review ........................................................................................................................................................ 57 4.4.2 Household Surveys ........................................................................................................................................... 58 4.4.3 Enterprise Surveys ........................................................................................................................................... 59 4.4.4 Community Surveys ......................................................................................................................................... 60 4.4.5 Applicant Survey ............................................................................................................................................... 60 4.4.6 Data Collected From the Routine Grantee Reporting ........................................................................ 61 4.4.7 Analysis Plan ....................................................................................................................................................... 62 4.5 Qualitative Data Collection and Analysis ......................................................................................................... 65 4.5.1 Key informant Interviews ............................................................................................................................. 65 4.5.2 Complementary Qualitative Data Collection for the IE ..................................................................... 67 4.5.3 Analysis Plan ....................................................................................................................................................... 67 4.6 Sampling Approach................................................................................................................................................... 67 4.7 Cost Benefit Analysis and Beneficiary Analysis ............................................................................................ 71 5.0 Administrative ....................................................................................................... 74 5.1 Summary of IRB Requirements and Clearances ........................................................................................... 74 5.2 Data Protection ........................................................................................................................................................... 74 5.3 Preparing Data Files for Access, Privacy, and Documentation ............................................................... 75 5.4 Dissemination Plan ................................................................................................................................................... 75 5.5 Evaluation Team Roles and Responsibilities ................................................................................................. 76 5.6 Evaluation Timeline and Reporting Schedule ............................................................................................... 77 6.0 References ............................................................................................................. 78 7.0 Annexes ................................................................................................................. 85 7.1 Project Evaluability Assessment ......................................................................................................................... 85 7.2 Stakeholder Comments and Evaluator Responses ...................................................................................... 96 7.3 Evaluation Budget ..................................................................................................................................................... 97 iv TABLES Table 1: Summary Of Grant Window Project Types and Status .................................................................... 8 Table 2: Typology of Benefits Associated With Off-Grid Energy Investments ...................................... 27 Table 3: Summary Of Evaluation Approach ........................................................................................................... 35 Table 4: Grant-Level Evaluation Questions ..........................................................................................................