WORLD BANK OPERATIONS EVALUATION DEPARTMENT Social Funds Assessing Effectiveness 2002 THE WORLD BANK http://www.worldbank.org/oed Washington, D.C. OPERATIONS EVALUATION DEPARTMENT ENHANCING DEVELOPMENT EFFECTIVENESS THROUGH EXCELLENCE AND INDEPENDENCE IN EVALUATION The Operations Evaluation Department (OED) is an independent unit within the World Bank; it reports directly to the Bank’s Board of Executive Directors. OED assesses what works, and what does not; how a borrower plans to run and maintain a project; and the lasting contribution of the Bank to a country’s overall development. The goals of evaluation are to learn from experience, to provide an objective basis for assessing the results of the Bank’s work, and to provide accountability in the achievement of its objectives. It also improves Bank work by identifying and disseminating the lessons learned from experience and by framing recommendations drawn from evaluation findings. © 2002 The International Bank for Reconstruction and Development / The World Bank 1818 H Street, NW Washington, DC 20433 All rights reserved. Manufactured in the United States of America First Printing May 2002 1 2 3 4 03 02 1 The findings, interpretations, and conclusions expressed here are those of the author(s) and do not necessarily reflect the views of the Board of Executive Directors of the World Bank or the governments they represent. The World Bank cannot guarantee the accuracy of the data included in this work. The boundaries, colors, denomina- tions, and other information shown on any map in this work do not imply on the part of the World Bank any judgment of the legal status of any territory or the endorsement or acceptance of such boundaries. Rights and Permissions The material in this work is copyrighted. No part of this work may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopying, recording, or inclusion in any information storage and retrieval system, without the prior written permission of the World Bank. The World Bank encourages dissemination of its work and will normally grant permission promptly. For permission to photocopy or reprint, please send a request with complete information to the Copyright Clearance Center, Inc., 222 Rosewood Drive, Danvers, MA 01923, USA, telephone 978-750-8400, fax 978-750-4470, www.copyright.com. All other queries on rights and licenses, including subsidiary rights, should be addressed to the Office of the Publisher, World Bank, 1818 H Street NW, Washington, DC 20433, USA, fax 202-522-2422, e-mail [email protected]. Cover photo: Courtesy of Curt Carnemark. ISBN 0-8213-5141-9 Library of Congress Cataloging-in-Publication Data has been applied for. Printed on Recycled Paper Contents vii Acknowledgments xi Foreword, Prólogo, Avant-Propos xv Executive Summary, Resumen, Résumé Analytique xxiii Abbreviations and Acronyms xxv Overview 1 1. Introduction 1 Social Fund Projects Have Diverse Features . 2. And Their Objectives Have Evolved 2 Therefore, the Evaluative Framework Must Be Broad 52.Outcome 5 What Have Social Fund Projects Aimed to Achieve? 6To What Extent Have Social Fund Projects Been Relevant, Efficacious, and Efficient? 19 What is the Record of Bank Performance in Social Funds? 21 3. Subproject Sustainability 21 Subproject Sustainability Can Be Assessed by Direct Observation . 24 . Or by Examining Conditions for Sustainability 31 4. Institutional Development Impact 32 Most Social Fund Agencies Are Independent and Competent . 34 . And Their Wider Institutional Effects Have Been Limited 44 Social Funds’ Institutional Role Continues to Change 47 5. Future Bank Support to Social Funds: Issues to Consider 53 6. Recommendations iii Social Funds: Assessing Effectiveness Annexes 57 Annex A. Social Funds Portfolio 61 Annex B. Review Instruments 65 Annex C. Regional Distribution of Social Funds 67 Annex D. Social Fund Financing by Activity 69 Annex E. Social Funds 2000 Results 83 Annex F. Promotion, Outreach, and Targeting 91 Annex G. Social Fund Winners and Losers 93 Annex H. Cost Data 97 Annex I. Bank Performance 99 Annex J. Stakeholder Survey Results 101 Annex K. Highlights of Beneficiary Assessment Findings on Participation 103 Annex L. Community-Based Contracting 105 Annex M. OED Household Survey Results 133 Annex N. Best Practice Examples/Elements of Social Fund Projects 137 Annex O. Community Participation and Social Capital 149 Annex P. Statement of the External Advisory Panel 151 Annex Q. Management Response 161 Annex R. CODE Report 165 Endnotes 179 Bibliography Boxes 3 Box 1.1. How Social Fund Agencies Expect to Achieve Development Effectiveness 4 Box 1.2. Social Funds Face Tradeoffs To Be Effective 11 Box 2.1. Quantity, Access, and Utilization of Physical Infrastructure Has Improved 12 Box 2.2. Outcomes and Welfare Impacts Are Varied 16 Box 2.3. Lessons From Experience: Allocation Rules for Reaching the Poor 18 Box 2.4. Solving Transport Problems Is Not Just a Matter of Building Roads 23 Box 3.1. Zambia: Tackling Tradeoffs Between Community Self-Help and Construction Quality 27 Box 3.2. Learning Curve: Trends in Sustainability Mechanisms 29 Box 3.3. When Should Social Funds Support Communities “Acting Alone” to Provide Goods and Services, and When Should Government Be Involved? 34 Box 4.1. Institutional Development Effects Have Many Channels 42 Box 4.2. Social Capital Impacts in Four Surveyed Countries Have Been Mixed 50 Box 5.1. What Key Factors Influence the Suitability and Design of Social Funds? iv Contents Tables 6Table 2.1. Most Social Funds Have Many Objectives 23 Table 3.1. Household Perception of Operations and Maintenance Performance 32 Table 4.1. Institutional Development Objectives Have Become More Local 38 Table 4.2. Social Fund Budgetary Arrangements Figures 13 Figure 2.1. Social Funds Have Mildly Progressive Geographic Targeting 14 Figure 2.2. Social Funds Have Neutral or Very Mildly Progressive Household Targeting v Acknowledgments his Operations Evaluation Department based methodology; a social fund evaluation (OED) review responds to the interest toolkit (institutional analysis at the govern- T of the World Bank’s Board of Executive ment and social fund agency levels); household Directors in the development effectiveness of the surveys comprising quantitative and qualitative Bank’s rapidly expanding social funds port- questions and complemented by focus groups folio, reiterated most recently at the Board dis- and key informant interviews (community-level cussion of the Social Protection Sector Strategy analysis); and an Internet-based “Activity Room” Paper in September 2000. Work on this review that served as a discussion forum. The findings coincided with, and benefited from, a parallel and recommendations of this review are pre- self-evaluation of social fund projects in six sented in a spirit of stock-taking, learning, and countries managed by the Human Develop- knowledge-sharing related to the development ments Network’s Social Protection Unit and the effectiveness of social fund projects. Poverty Reduction and Economic Management The task manager of the review was Soniya Nerwork’s Poverty Analysis Unit, Social Funds Carvalho. This report was written by Soniya 2000 Impact Evaluation. In addition to pre- Carvalho, Gillian Perkins, and Howard White, senting an independent perspective on issues with contributions from Caroline Bahnson, covered in the self-evaluation, including the Anju Gupta Kapoor, and Svenja Weber- issues of targeting and subproject sustainability, Venghaus. It was an activity of the Urban this OED review provides fresh insights into Cluster, coordinated by Roy Gilbert and com- other areas, notably the participatory process in prising Anna Amato, Soniya Carvalho, and social fund projects and their institutional devel- Ronald Parker. William Hurlbut edited the opment impacts in the government, non- report. In addition to the immediate team, government, and community arenas. background analysis and/or data collection The social fund portfolio is relatively young, were undertaken by Charles Abelmann, Rodrigo with only 23 closed projects in the OED data- Cisneros, Geske Dijkstra, Jean Paul Fauget, base as of June 30, 2000. This review is based Jennifer Green, Ana Maria Ibanez, Thomas on a portfolio assessment of all 66 social fund Kuby, Sohail Malik, Julian May, Azedine projects approved by the Bank as of end-fiscal Ouerghi, Vijayendra Rao, Susan Razzaz, David 1999, a literature review, task manager inter- Sahn, Rodrigo Serrano, Jan Kees van Donge, views, stakeholder surveys, household surveys and Stephen Younger. Contributions from the and community-level key informant interviews Development Economics, EASHD, PRMGE, and and focus groups in 4 countries, and institu- the World Bank Institute are much appreciated. tional analysis based on field research in 7 Andres Jarrin and Stanley Khaila managed the countries. A number of tools were developed household survey work and qualitative research or applied in the context of this review: theory- on behalf of Urbana Consultores, Ecuador, and vii Social Funds: Assessing Effectiveness the Center for Social Reseach, Malawi. George Sherburne-Benz, David Steel, Susan Stout, and Campbell and Ruel Cooke, Environmental Solu- Julie Van Domelen, and review by Livia tions, Jamaica, conducted additional fieldwork. Benavides, Judy Baker, Anush Bezhanyan, Can Adamoglu, Alexander Arenas, Arif Syed Christopher Chamberlin, Samantha De Silva, Husain, Enver Kamal, Stig Kjeldsen, Kalpana Shantayanan Devarajan, William