IMPACT EVALUATION OF DEVELOPMENT INTERVENTIONS A Practical Guide Howard White David A. Raitzer ASIAN DEVELOPMENT BANK IMPACT EVALUATION OF DEVELOPMENT INTERVENTIONS A Practical Guide Howard White David A. Raitzer ASIAN DEVELOPMENT BANK Creative Commons Attribution 3.0 IGO license (CC BY 3.0 IGO) © 2017 Asian Development Bank 6 ADB Avenue, Mandaluyong City, 1550 Metro Manila, Philippines Tel +63 2 632 4444; Fax +63 2 636 2444 www.adb.org Some rights reserved. Published in 2017. Printed in the Philippines. ISBN 978-92-9261-058-6 (print), 978-92-9261-059-3 (electronic) Publication Stock No. TCS179188-2 DOI: http://dx.doi.org/10.22617/TCS179188-2 The views expressed in this publication are those of the authors and do not necessarily reflect the views and policies of the Asian Development Bank (ADB) or its Board of Governors or the governments they represent. ADB does not guarantee the accuracy of the data included in this publication and accepts no responsibility for any consequence of their use. The mention of specific companies or products of manufacturers does not imply that they are endorsed or recommended by ADB in preference to others of a similar nature that are not mentioned. By making any designation of or reference to a particular territory or geographic area, or by using the term “country” in this document, ADB does not intend to make any judgments as to the legal or other status of any territory or area. This work is available under the Creative Commons Attribution 3.0 IGO license (CC BY 3.0 IGO) https://creativecommons.org/licenses/by/3.0/igo/. By using the content of this publication, you agree to be bound by the terms of this license. For attribution, translations, adaptations, and permissions, please read the provisions and terms of use at https://www.adb.org/terms-use#openaccess. This CC license does not apply to non-ADB copyright materials in this publication. If the material is attributed to another source, please contact the copyright owner or publisher of that source for permission to reproduce it. ADB cannot be held liable for any claims that arise as a result of your use of the material. Please contact [email protected] if you have questions or comments with respect to content, or if you wish to obtain copyright permission for your intended use that does not fall within these terms, or for permission to use the ADB logo. Photos in this publication are the property of ADB. Notes: In this publication, “$” refers to US dollars. Corrigenda to ADB publications may be found at http://www.adb.org/publications/corrigenda. Contents Tables, Figures, and Boxes ...........................................................................................vi Foreword .......................................................................................................................viii Acknowledgments..........................................................................................................x Abbreviations ................................................................................................................xi 1. Introduction: Impact Evaluation for Evidence-Based Development ..............1 1.1 Why Does Impact Evaluation ................................................Matter? 1 1.2 The Purposes of Impactvaluation ....................................................E 3 1.3 What Questions Can Impact Evaluation ...........................Answer? 4 1.4 Impact valuationE in the Project ............................................ Cycle 11 1.5 Impact valuation,E Evaluation, and Economic ..............Analysis13 1.6 About This ....................................................................................Book 15 2. Using Theories of Change to Identify Impact Evaluation Questions ...........20 2.1 Introduction............................................................................................20 2.2 What s I a Theory of .............................................................Change? 21 2.3 Issues Raised by the Theory ............................................ of Change 23 2.4 Constructing the Theory of .................................................. Change 26 2.5 Applying the Theory of ......................................................... Change 28 3. The Core Concepts of Impact Evaluation ..........................................................31 3.1 What s I Impact Evaluation? ................................................................31 3.2 Identifying Control and Comparison .................................. Groups 33 3.3 Biases and Challenges for Causalence .................................... Infer 36 3.4 Time Dimension of ................................................................Impacts 40 3.5 Unit of Assignment, Treatment, and ................................. Analysis 42 3.6 Different Impact Measures – Impacts ........................for Whom?43 3.7 Internal andxternal E Validity ............................................................45 iv Contents 4. Randomized Controlled Trials ............................................................................. 47 4.1 Introduction............................................................................................47 4.2 Why Randomize? ...................................................................................48 4.3 Types of Randomized Controlled Trial ............................. Designs 49 4.4 Steps in Implementing a Randomized Controlled ...............54 Trial 4.5 Reporting Randomized Controlled ........................................Trials 59 4.6 Working with Randomized Controlled Trials ............in 59Practice 4.7 Summary..................................................................................................64 5. Nonexperimental Designs ....................................................................................66 5.1 Introduction............................................................................................66 5.2 Difference-in-Differences Estimates .................................................68 5.3 Synthetic Controls .................................................................................72 5.4 Propensity Score Matching .................................................................73 5.5 Propensity Score Weighting and Double Robust ...... 78Techniques 5.6 Regression Discontinuity Design and Interrupted ..79Time Series 5.7 Instrumental Variables ........................................................................ 84 5.8 Endogenous reatmentT and Control Function Approaches .........87 5.9 Endogenous witchingS Regressions ..................................................89 5.10 Summary..................................................................................................90 6. What and How to Measure: Data Collection for Impact Evaluation ............96 6.1 The Importance of QualityHigh- Data .............................................96 6.2 Determining Whatata D Are Needed ................................................97 6.3 Data Sources ...........................................................................................98 6.4 Designing aurvey S ............................................................................. 101 6.5 Sampling ............................................................................................... 108 6.6 Managing Data Collection ................................................................ 110 7. Sample Size Determination for Data Collection.............................................. 116 7.1 Power Calculations: An Introduction............................................. 116 7.2 What s I Power? ................................................................................... 117 7.3 Power Calculations for Simple Study ............................. Designs 121 7.4 Power Calculations for Cluster ........................................Designs 123 7.5 The angerD of Underpowered ..........................................Studies 126 7.6 Aids for Considering erformingand P Power Calculations ............................................................................. 127 Contents v 8. Managing the Impact Evaluation Process ......................................................... 131 8.1 Introduction to Managing Impact Evaluations ............................ 131 8.2 Planning Impactvaluations E .......................................................... 131 8.3 Selecting an Impactvaluation E Design ......................................... 134 8.4 Timing andudgeting B for Impact Evaluation ............................ 137 8.5 Where Help for Impact Evaluation Can ................ Be 143Obtained 8.6 Interpreting and Presenting Impact Evaluation .........Findings145 8.7 Conclusions: Toward Evidence-Based Directions for Development ............................................................. 147 Appendix 1. Application of Estimation Methods for Impact Evaluation ................153 Section 1: The Potential Outcomes Framework ...................................... 153 Section 2: Randomized Controlled Trials ................................................. 157 Section 3: Difference-in-Differences and Fixed Effects Models .......... 159 Section 4: Synthetic Controls ...................................................................... 163 Section 5: Propensity Score Based Approaches (Matching, Weighting, and Double Robust Estimators) ............ 165 Section 6: Instrumental Variables Based Approaches (Two-Stage Least Squares, Endogenous Treatment Regressions, and Endogenous