*Updated Evaluation Design MCC Water and Sanitation Projects in El Salvador Miguel Almanzar†, Maureen Cropper‡ and Raymond Guiteras§ University of Maryland- College Park Consortium led by Social Impact Revised: April 2014 * We would like to acknowledge Kristin Komives, Christine Poulos and Subhrendu Pattanayak who wrote the initial impact evaluation design proposal † Miguel Almanzar is a Senior Research Analyst at the International Food Policy Research Institute (IFPRI) and a Research Associate at the University of Maryland. [email protected] ‡ Maureen Cropper is a Distinguished Professor of Economics at the University of Maryland and Senior Research Fellow at Resources for the Future. [email protected] § Raymond Guiteras is an Assistant Professor at the University of Maryland. [email protected] i. List of Acronyms ANDA : Administración Nacional de Acueductos y Alcantarillados National Water and Sewerage Administration DID : Difference in Difference DIGESTYC : Dirección General de Estadísticas y Censos General Directorate for Statistics and Census in El Salvador EHPM : Encuesta de Hogares de Propósitos Múltiples Multi-purpose household survey ERR : Economic Rate of Return FOMILENIO : Fondo del Milenio Millenium Fund FUSADES : Fundación Salvadoreña para el Desarrollo Económico y Social Salvadoran Foundation for Economic and Social Development GOES : Government of El Salvador IHSN : International Household Survey Network ITT : Intention to Treat MCC : Millennium Challenge Corporation MDE : Minimum Detectable Effect MSPAS : Ministry of Public Health M&E : Monitoring and Evaluation PSM : Propensity Score Matching WSS : Water and Sanitation Sub-Activity WHO : World Health Organization UNICEF : United Nations Children’s Fund ii ii. Table of Contents 1. Introduction................................................................................................................................1 2. Overview of the Compact and the intervention evaluated .............................................................2 2.1. Program Logic - Input, output, outcomes and ultimate impact ................................................2 2.1.1. Compact-level ..............................................................................................................2 2.1.2. Project-level .................................................................................................................3 2.2. Link to ERR and Beneficiary Analysis ......................................................................................6 3. Literature review of the evidence .................................................................................................6 3.1. Evidence gaps that current evaluation fills .............................................................................9 4. Evaluation Design ........................................................................................................................9 4.1. Evaluation type ....................................................................................................................9 4.2. Evaluation questions ............................................................................................................9 Household welfare ..................................................................................................................9 Coping costs and cash expenditure on water ........................................................................... 10 Health ................................................................................................................................... 11 Education .............................................................................................................................. 13 Service, use, and sustainability ............................................................................................... 13 Gender and social exclusion ................................................................................................... 14 4.2.1. Country-specific and international policy relevance of evaluation.................................. 14 4.2.2. Definition of key outcomes linked to program logic ...................................................... 15 4.3. Methodology ..................................................................................................................... 16 4.3.1. Treatment Assignment................................................................................................ 17 Projects................................................................................................................................. 18 Households ........................................................................................................................... 19 Communities ......................................................................................................................... 19 4.3.2. Identifying the Treatment Segments ............................................................................ 20 4.3.3. Selection of comparison segments............................................................................... 21 Assumptions of the PSM and Balancing................................................................................... 24 4.4. Study Population................................................................................................................ 26 4.5. Power calculations and sample size requirements................................................................ 27 4.5.1. Sample design and power for 2009 baseline................................................................. 27 iii 4.5.1. Selection of households .............................................................................................. 30 4.5.1. Updated Power Calculations ....................................................................................... 31 4.5.2. Implications for measuring other impacts..................................................................... 35 4.5.3. Implications for measuring health impacts ................................................................... 37 4.5.4. Geographical Distribution of Sample Census Segments ................................................. 39 4.6. Timeframe ......................................................................................................................... 40 4.6.1. Justification for proposed exposure period to treatment .............................................. 41 4.7. Challenges for evaluating the impact of the investments ...................................................... 42 5. Data Sources and Outcome Definitions....................................................................................... 44 5.1. Data Sources...................................................................................................................... 45 5.1.1. Quantitative – Household Survey................................................................................. 46 5.1.2. Quantitative and Qualitative - Community Survey ........................................................ 46 6. Analysis Plan ............................................................................................................................. 48 6.1. Statistical techniques ......................................................................................................... 48 6.1.1. Separate populations and outcomes ............................................................................ 50 6.1.2. Complementary Methodology ..................................................................................... 50 7. Monitoring Plan ........................................................................................................................ 51 7.1. Adherence to treatment and control areas .......................................................................... 51 8. Administrative .......................................................................................................................... 51 8.1. Summary of Institutional Review Board requirements and clearances ................................... 51 8.2. Data Access, Privacy and Documentation Plan ..................................................................... 52 8.3. Dissemination Plan............................................................................................................. 52 8.4. Evaluation Team roles and responsibilities .......................................................................... 52 8.5. Budget .............................................................................................................................. 53 9. References................................................................................................................................ 54 Annex 1: List of eligible municipalities ............................................................................................... 58 Annex 2: Project Selection Criteria .................................................................................................... 60 Annex 3: List and description of projects ............................................................................................ 62 Annex 4: Matching Results ................................................................................................................ 62 Annex 5: Water quality testing .......................................................................................................... 68 iv