Public Disclosure Authorized Public Disclosure Authorized Public Disclosure Authorized Public Disclosure Authorized Impact Evaluation in Practice Second Edition Please visit the Impact Evaluation in Practice book website at http://www.worldbank .org/ieinpractice. The website contains accompanying materials, including solutions to the book’s HISP case study questions, as well as the corresponding data set and analysis code in the Stata software; a technical companion that provides a more formal treatment of data analysis; PowerPoint presentations related to the chapters; an online version of the book with hyperlinks to websites; and links to additional materials. This book has been made possible thanks to the generous support of the Strategic Impact Evaluation Fund (SIEF). Launched in 2012 with support from the United Kingdom’s Department for International Development, SIEF is a partnership program that promotes evidence-based policy making. The fund currently focuses on four areas critical to healthy human development: basic education, health systems and service delivery, early childhood development and nutrition, and water and sanitation. SIEF works around the world, primarily in low-income countries, bringing impact evaluation expertise and evidence to a range of programs and policy-making teams. Impact Evaluation in Practice Second Edition Paul J. Gertler, Sebastian Martinez, Patrick Premand, Laura B. Rawlings, and Christel M. J. Vermeersch © 2016 International Bank for Reconstruction and Development / The World Bank 1818 H Street NW, Washington, DC 20433 Telephone: 202-473-1000; Internet: www.worldbank.org Some rights reserved 1 2 3 4 19 18 17 16 The fi nding, interpretations, and conclusions expressed in this work do not necessarily refl ect the views of The World Bank, its Board of Executive Directors, the Inter- American Development Bank, its Board of Executive Directors, or the governments they represent. The World Bank and the Inter-American Development Bank do not guarantee the accuracy of the data included in this work. The boundaries, colors, denominations, and other information shown on any map in this work do not imply any judgement on the part of The World Bank or the Inter-American Development Bank concerning the legal status of any territory or the endorsement or acceptance of such boundaries. Nothing herein shall constitute or be considered to be a limitation upon or waiver of the privileges and immunities of The World Bank or IDB, which privileges and immunities are specifi cally reserved. Rights and Permissions This work is available under the Creative Commons Attribution 3.0 IGO license (CC BY 3.0 IGO) http://creativecommons .org/licenses/by/3.0/igo. Under the Creative Commons Attribution license, you are free to copy, distribute, transmit, and adapt this work, including for commercial purposes, under the following conditions: Attribution—Please cite the work as follows: Gertler, Paul J., Sebastian Martinez, Patrick Premand, Laura B. Rawlings, and Christel M. J. Vermeersch. 2016. Impact Evaluation in Practice, second edition. Washington, DC: Inter-American De velopment Bank and World Bank. doi:10.1596/978-1-4648-0779-4. License: Creative Commons Attribution CC BY 3.0 IGO Translations—If you create a translation of this work, please add the following disclaimer along with the attribution: This translation was not created by The World Bank and should not be considered an offi cial World Bank translation. The World Bank shall not be liable for any content or error in this translation. Adaptations—If you create an adaptation of this work, please add the following disclaimer along with the attribution: This is an adaptation of an original work by The World Bank. Views and opinions expressed in the adaptation are the sole responsibility of the author or authors of the adaptation and are not endorsed by The World Bank. Third-party content—The World Bank does not necessarily own each component of the content contained within the work. The World Bank therefore does not warrant that the use of any third-party-owned individual component or part contained in the work will not infringe on the rights of those third parties. The risk of claims resulting from such infringement rests solely with you. If you wish to re-use a component of the work, it is your responsibility to determine whether permission is needed for that re-use and to obtain permission from the copyright owner. Examples of components can include, but are not limited to, tables, fi gures, or images. All queries on rights and licenses should be addressed to the Publishing and Knowledge Division, The World Bank, 1818 H Street NW, Washington, DC 20433, USA; fax: 202-522-2625; e-mail: [email protected]. ISBN (paper): 978-1-4648-0779-4 ISBN (electronic): 978-1-4648-0780-0 DOI: 10.1596/978-1-4648-0779-4 Illustration: C. Andres Gomez-Pena and Michaela Wieser Cover Design: Critical Stages Library of Congress Cataloging-in-Publication Data Names: Gertler, Paul, 1955- author. | World Bank. Title: Impact evaluation in practice / Paul J. Gertler, Sebastian Martinez, Patrick Premand, Laura B. Rawlings, Christel M. J. Vermeersch. Description: Second Edition. | Washington, D.C.: World Bank, 2016. | Revised edition of Impact evaluation in practice, 2011. Identifi ers: LCCN 2016029061 (print) | LCCN 2016029464 (ebook) | ISBN 9781464807794 (pdf ) | ISBN 9781464807800 | ISBN 9781464807800 () Subjects: LCSH: Economic development projects—Evaluation. | Evaluation research (Social action programs) Classifi cation: LCC HD75.9.G478 2016 (print) | LCC HD75.9 (ebook) | DDC 338.91—dc23 LC record available at https://lccn.loc.gov/2016029061 CONTENTS Preface xv Acknowledgments xxi About the Authors xxiii Abbreviations xxvii PART ONE. INTRODUCTION TO IMPACT EVALUATION 1 Chapter 1. Why Evaluate? 3 Evidence-Based Policy Making 3 What Is Impact Evaluation? 7 Prospective versus Retrospective Impact Evaluation 9 Effi cacy Studies and Effectiveness Studies 11 Complementary Approaches 13 Ethical Considerations Regarding Impact Evaluation 20 Impact Evaluation for Policy Decisions 21 Deciding Whether to Carry Out an Impact Evaluation 26 Chapter 2. Preparing for an Evaluation 31 Initial Steps 31 Constructing a Theory of Change 32 Developing a Results Chain 34 Specifying Evaluation Questions 36 Selecting Outcome and Performance Indicators 41 Checklist: Getting Data for Your Indicators 42 PART TWO. HOW TO EVALUATE 45 Chapter 3. Causal Inference and Counterfactuals 47 Causal Inference 47 The Counterfactual 48 Two Counterfeit Estimates of the Counterfactual 54 v Chapter 4. Randomized Assignment 63 Evaluating Programs Based on the Rules of Assignment 63 Randomized Assignment of Treatment 64 Checklist: Randomized Assignment 81 Chapter 5. Instrumental Variables 89 Evaluating Programs When Not Everyone Complies with Their Assignment 89 Types of Impact Estimates 90 Imperfect Compliance 92 Randomized Promotion as an Instrumental Variable 101 Checklist: Randomized Promotion as an Instrumental Variable 110 Chapter 6. Regression Discontinuity Design 113 Evaluating Programs That Use an Eligibility Index 113 Fuzzy Regression Discontinuity Design 117 Checking the Validity of the Regression Discontinuity Design 119 Limitations and Interpretation of the Regression Discontinuity Design Method 124 Checklist: Regression Discontinuity Design 126 Chapter 7. Difference-in-Differences 129 Evaluating a Program When the Rule of Assignment Is Less Clear 129 The Difference-in-Differences Method 130 How Is the Difference-in-Differences Method Helpful? 134 The “Equal Trends” Assumption in Difference-in-Differences 135 Limitations of the Difference-in-Differences Method 141 Checklist: Difference-in-Differences 141 Chapter 8. Matching 143 Constructing an Artifi cial Comparison Group 143 Propensity Score Matching 144 Combining Matching with Other Methods 148 Limitations of the Matching Method 155 Checklist: Matching 156 Chapter 9. Addressing Methodological Challenges 159 Heterogeneous Treatment Effects 159 Unintended Behavioral Effects 160 Imperfect Compliance 161 Spillovers 163 Attrition 169 Timing and Persistence of Effects 171 vi Impact Evaluation in Practice Chapter 10. Evaluating Multifaceted Programs 175 Evaluating Programs That Combine Several Treatment Options 175 Evaluating Programs with Varying Treatment Levels 176 Evaluating Multiple Interventions 179 PART THREE. HOW TO IMPLEMENT AN IMPACT EVALUATION 185 Chapter 11. Choosing an Impact Evaluation Method 187 Determining Which Method to Use for a Given Program 187 How a Program’s Rules of Operation Can Help Choose an Impact Evaluation Method 188 A Comparison of Impact Evaluation Methods 193 Finding the Smallest Feasible Unit of Intervention 197 Chapter 12. Managing an Impact Evaluation 201 Managing an Evaluation’s Team, Time, and Budget 201 Roles and Responsibilities of the Research and Policy Teams 202 Establishing Collaboration 208 How to Time the Evaluation 213 How to Budget for an Evaluation 216 Chapter 13. The Ethics and Science of Impact Evaluation 231 Managing Ethical and Credible Evaluations 231 The Ethics of Running Impact Evaluations 232 Ensuring Reliable and Credible Evaluations through Open Science 237 Checklist: An Ethical and Credible Impact Evaluation 243 Chapter 14. Disseminating Results and Achieving Policy Impact 247 A Solid Evidence Base for Policy 247 Tailoring a Communication Strategy to Different Audiences 250 Disseminating Results 254 PART FOUR. HOW TO GET DATA FOR AN IMPACT EVALUATION 259 Chapter 15. Choosing a Sample 261 Sampling and Power