Performance Indonesia: Decentralized Basic Evaluation Report Education Project Independent Evaluation Performance Evaluation Report November 2014 IndonesiaIndonesia:: Decentralized Basic Education Project This document is being disclosed to the public in accordance with ADB's Public Communications Policy 2011. Reference Number: PPE:INO 201 4-15 Loan and Grant Numbers: 1863-INO and 0047-INO Independent Evaluation: PE-774 NOTES (i) The fiscal year of the government ends on 31 December. (ii) In this report, “$” refers to US dollars. (iii) For an explanation of rating descriptions used in ADB evaluation reports, see Independent Evaluation Department. 2006. Guidelines for Preparing Performance Evaluation Reports for Public Sector Operations. Manila: ADB (as well as its amendment effective from March 2013). Director General V. Thomas, Independent Evaluation Department (IED) Director W. Kolkma, Independent Evaluation Division 1, IED Team leader H. Son, Principal Evaluation Specialist, IED Team member S. Labayen, Associate Evaluation Analyst, IED The guidelines formally adopted by the Independent Evaluation Department on avoiding conflict of interest in its independent evaluations were observed in the preparation of this report. To the knowledge of the management of the Independent Evaluation Department, there were no conflicts of interest of the persons preparing, reviewing, or approving this report. In preparing any evaluation report, or by making any designation of or reference to a particular territory or geographic area in this document, the Independent Evaluation Department does not intend to make any judgment as to the legal or other status of any territory or area. Abbreviations ADB – Asian Development Bank BOS – Bantuan Operational Sekolah (School Operational Assistance Program) DBEP – Decentralized Basic Education Project MONE – Ministry of National Education MORA – Ministry of Religious Affairs NTB – Nusa Tenggara Barat (West Nusa Tenggara) NTT – Nusa Tenggara Timur (East Nusa Tenggara) PCR – project completion report TA – technical assistance Currency Equivalents Currency Unit – rupiah (Rp) At Project At Appraisal Completion At Independent Evaluation (15 October 2001) (15 June 2012) (31 August 2014) Rp1.00 = $0.0001 $0.0002 $0.0001 $1.00 = Rp9,935 Rp9,420 Rp11,705 Contents Acknowledgments vii Basic Data ix Executive Summary xi Chapter 1: Introduction 1 A. Evaluation Purpose and Process 1 B. Expected Results 2 C. Evaluation Method 2 Chapter 2: Design and Implementation 4 A. Formulation 4 B. Rationale 5 C. Cost, Financing, and Executing Arrangements 6 D. Procurement, Construction, and Scheduling 8 E. Outputs 8 F. Consultants 9 G. Covenants 10 H. Policy Framework 10 Chapter 3: Performance Assessment 12 A. Overall Assessment 12 B. Relevance 12 C. Effectiveness 19 D. Efficiency 27 E. Sustainability 28 Chapter 4: Other Assessment 30 A. Impact 30 B. ADB and Executing Agency Performance 30 Chapter 5: Issues, Lessons, and Follow-up Actions 32 A. Issues 32 B. Lessons 33 C. Follow-up Actions 34 APPENDIXES 1 Summary Design and Monitoring Framework 36 2 List of Interviewees 41 3 Questionnaire for Schools 44 4 Number and Type of Project Schools 50 5 Rating Matrix for Core Evaluation Criteria 51 6 Education Statistics 53 7 Poverty Statistics 55 Acknowledgments A team of staff and consultants from the Independent Evaluation Department (IED) contributed to this study by conducting data gathering and analysis, desk reviews, interviews, research, and surveys. Hyun H. Son (team leader) prepared this report and Stella Labayen (associate evaluation analyst) formatted it. The report benefited from the overall guidance of Vinod Thomas and Walter Kolkma. The team acknowledges the valuable inputs of Rizza Leonzon and Ekki Syamsulhakim, the consultants who worked on this evaluation. The team also thanks Wolfgang Kubitzki, Ferny Suhandi, and Imelda Marquez for their assistance in organizing meetings with central and district government officials, development partners, and project schools. The team is grateful to Asian Development Bank (ADB) staff, school representatives, and the Government of Indonesia for their hospitality, as well as assistance and participation in the interviews. Their insights into the decentralized basic education sector were most helpful in writing the report. The report was peer reviewed by Tu Chi Nguyen of the World Bank and Farzana Ahmed of IED. Valuable comments were received on an earlier draft from ADB’s Southeast Asia Department and Indonesia Resident Mission. The report was also shared with the Government of Indonesia, which welcomed the findings and recommendations of the study. Basic Data Decentralized Basic Education Project (Loan 1863-INO; Grant 0047-INO) Key Project Data As per ADB Loan/Grant ($ million) Documents Actual Total Loan Cost 125.0 138.4 Foreign Exchange Cost 21.1 26.3 Local Currency Cost 103.9 112.1 ADB Financed 100.0 112.1 Borrower Financed 25.0 26.3 Total Grant Cost 36.3 29.5 Foreign Exchange Cost 36.3 29.5 Borrower Financed 8.3 1.5 Government of the Netherlands 28.0 28.0 Key Dates Expected Actual Appraisal 6–26 July 2001 Loan Negotiations 3–10 October 2001 Board Approval - Loan 29 November 2001 - Grant 14 March 2006 Loan Agreement 20 February 2002 Loan Effectiveness 20 May 2002 20 May 2002 Closing Date - Loan 31 December 2008 30 June 2009 - Grant 31 December 2008 30 September 2011 Closing Extensions 2 (Grant) Loan Effectiveness to Closing (months) 81 87 Project Completion June 2012 Borrower Government of Indonesia Executing Agency Ministry of National Education Mission Data Type of Mission No. of Missions No. of Person-Days Inception 1 138 Handover 1 14 Review 12 309 Special project administration 2 82 Midterm 1 75 Final 1 10 Project completion review 1 20 Independent evaluation 1 9 Executive Summary Decentralization is a policy option that has been adopted by many countries to improve access to basic services, including education. Since 2001, the Government of Indonesia has embarked on a program to decentralize delivery of basic education. By making the education system more attuned to local needs, decentralization—the process of delegating authority or functions from central government to local government units— will be able to play a crucial role in improving the access to and quality of basic education in Indonesia. To help Indonesia roll out its decentralized basic education program, the Asian Development Bank (ADB) supported a project to improve the participation in and completion of 9 years of basic education, particularly among the poor. The project included primary and junior secondary levels in the provinces of Bali, East Nusa Tenggara (NTT), and West Nusa Tenggara (NTB). It supported three outcomes: (i) improved participation, transition, completion, and performance in basic education among poor children in Bali, NTB, and NTT; (ii) implementation of school-based management in project schools; and (iii) effective district education management in the three provinces. To achieve these objectives, the project had three components: school development; district basic education development; and monitoring, evaluation, and reporting. The project was financed by a combination of loan and grant. Of the $138.4 million loan component, $112.1 million was financed by ADB and $26.3 million by the Government of Indonesia. The grant component totaled $29.5 million, of which $28.0 million was funded by the Government of the Netherlands and $1.5 million by the Government of Indonesia. This report presents performance evaluation findings of the Decentralized Basic Education Project based on four core evaluation criteria (relevance, effectiveness, efficiency, and sustainability) and two additional criteria (institutional development and impact). Overall, the project performance is rated successful. The project is rated highly relevant. The timing of its implementation was responsive to addressing insufficient capacity during Indonesia’s transition to a decentralized basic education system. The project is also aligned with the education policies and strategies of both the government and ADB. Changes in the project coverage involving withdrawal of five Jakarta districts and redirection of this funding to two districts in impoverished NTT improved its relevance. Modifications in project design during the early phase of implementation, as well as weak coordination between the Ministry of National Education (MONE) and the Ministry of Religious Affairs (MORA), the project’s executing and co-executing agencies, respectively, posed challenges to implementation but did not compromise relevance either in terms of targeting schools and districts or in achieving the objective of improving access of the poor to basic education. The project is effective. Analysis of primary time-series data obtained during the evaluation indicates that statistically significant improvements in education outcomes xii Indonesia: Decentralized Basic Education Project including enrollment, completion, transition, and performance were observed in selected project districts after the project was implemented. The project also helped to improve school capacity by promoting an evidence-based approach to formulating school development plans; encouraging community involvement in education matters; and improving district capacity for basic education, particularly in teacher development and the evaluation of school development plans. The project is efficient. Despite the limited size of grants, the project supported notable achievements in improving