Performance Indonesia: Decentralized Basic Evaluation Report Education Project

Independent Evaluation

Performance Evaluation Report November 2014

IndonesiaIndonesia:: Decentralized Basic Education Project

This document is being disclosed to the public in accordance with ADB's Public Communications Policy 2011.

Reference Number: PPE:INO 201 4-15 Loan and Grant Numbers: 1863-INO and 0047-INO Independent Evaluation: PE-774

NOTES

(i) The fiscal year of the government ends on 31 December. (ii) In this report, “$” refers to US dollars. (iii) For an explanation of rating descriptions used in ADB evaluation reports, see Independent Evaluation Department. 2006. Guidelines for Preparing Performance Evaluation Reports for Public Sector Operations. Manila: ADB (as well as its amendment effective from March 2013).

Director General V. Thomas, Independent Evaluation Department (IED) Director W. Kolkma, Independent Evaluation Division 1, IED

Team leader H. Son, Principal Evaluation Specialist, IED Team member S. Labayen, Associate Evaluation Analyst, IED

The guidelines formally adopted by the Independent Evaluation Department on avoiding conflict of interest in its independent evaluations were observed in the preparation of this report. To the knowledge of the management of the Independent Evaluation Department, there were no conflicts of interest of the persons preparing, reviewing, or approving this report.

In preparing any evaluation report, or by making any designation of or reference to a particular territory or geographic area in this document, the Independent Evaluation Department does not intend to make any judgment as to the legal or other status of any territory or area.

Abbreviations

ADB – Asian Development Bank BOS – Bantuan Operational Sekolah (School Operational Assistance Program) DBEP – Decentralized Basic Education Project MONE – Ministry of National Education MORA – Ministry of Religious Affairs NTB – Nusa Tenggara Barat (West Nusa Tenggara) NTT – Nusa Tenggara Timur (East Nusa Tenggara) PCR – project completion report TA – technical assistance

Currency Equivalents

Currency Unit – rupiah (Rp)

At Project At Appraisal Completion At Independent Evaluation (15 October 2001) (15 June 2012) (31 August 2014) Rp1.00 = $0.0001 $0.0002 $0.0001 $1.00 = Rp9,935 Rp9,420 Rp11,705

Contents

Acknowledgments vii Basic Data ix Executive Summary xi Chapter 1: Introduction 1 A. Evaluation Purpose and Process 1 B. Expected Results 2 C. Evaluation Method 2 Chapter 2: Design and Implementation 4 A. Formulation 4 B. Rationale 5 C. Cost, Financing, and Executing Arrangements 6 D. Procurement, Construction, and Scheduling 8 E. Outputs 8 F. Consultants 9 G. Covenants 10 H. Policy Framework 10 Chapter 3: Performance Assessment 12 A. Overall Assessment 12 B. Relevance 12 C. Effectiveness 19 D. Efficiency 27 E. Sustainability 28 Chapter 4: Other Assessment 30 A. Impact 30 B. ADB and Executing Agency Performance 30 Chapter 5: Issues, Lessons, and Follow-up Actions 32 A. Issues 32 B. Lessons 33 C. Follow-up Actions 34

APPENDIXES 1 Summary Design and Monitoring Framework 36

2 List of Interviewees 41 3 Questionnaire for Schools 44 4 Number and Type of Project Schools 50 5 Rating Matrix for Core Evaluation Criteria 51 6 Education Statistics 53 7 Poverty Statistics 55

Acknowledgments

A team of staff and consultants from the Independent Evaluation Department (IED) contributed to this study by conducting data gathering and analysis, desk reviews, interviews, research, and surveys. Hyun H. Son (team leader) prepared this report and Stella Labayen (associate evaluation analyst) formatted it. The report benefited from the overall guidance of Vinod Thomas and Walter Kolkma. The team acknowledges the valuable inputs of Rizza Leonzon and Ekki Syamsulhakim, the consultants who worked on this evaluation.

The team also thanks Wolfgang Kubitzki, Ferny Suhandi, and Imelda Marquez for their assistance in organizing meetings with central and district government officials, development partners, and project schools. The team is grateful to Asian Development Bank (ADB) staff, school representatives, and the Government of Indonesia for their hospitality, as well as assistance and participation in the interviews. Their insights into the decentralized basic education sector were most helpful in writing the report.

The report was peer reviewed by Tu Chi Nguyen of the World Bank and Farzana Ahmed of IED. Valuable comments were received on an earlier draft from ADB’s Southeast Asia Department and Indonesia Resident Mission. The report was also shared with the Government of Indonesia, which welcomed the findings and recommendations of the study.

Basic Data

Decentralized Basic Education Project (Loan 1863-INO; Grant 0047-INO)

Key Project Data As per ADB Loan/Grant ($ million) Documents Actual Total Loan Cost 125.0 138.4 Foreign Exchange Cost 21.1 26.3 Local Currency Cost 103.9 112.1

ADB Financed 100.0 112.1 Borrower Financed 25.0 26.3

Total Grant Cost 36.3 29.5 Foreign Exchange Cost 36.3 29.5

Borrower Financed 8.3 1.5 Government of the Netherlands 28.0 28.0

Key Dates Expected Actual Appraisal 6–26 July 2001 Loan Negotiations 3–10 October 2001 Board Approval - Loan 29 November 2001 - Grant 14 March 2006 Loan Agreement 20 February 2002 Loan Effectiveness 20 May 2002 20 May 2002 Closing Date - Loan 31 December 2008 30 June 2009 - Grant 31 December 2008 30 September 2011 Closing Extensions 2 (Grant) Loan Effectiveness to Closing (months) 81 87 Project Completion June 2012

Borrower Government of Indonesia Executing Agency Ministry of National Education

Mission Data Type of Mission No. of Missions No. of Person-Days Inception 1 138 Handover 1 14 Review 12 309 Special project administration 2 82 Midterm 1 75 Final 1 10 Project completion review 1 20 Independent evaluation 1 9

Executive Summary

Decentralization is a policy option that has been adopted by many countries to improve access to basic services, including education. Since 2001, the Government of Indonesia has embarked on a program to decentralize delivery of basic education. By making the education system more attuned to local needs, decentralization—the process of delegating authority or functions from central government to local government units— will be able to play a crucial role in improving the access to and quality of basic education in Indonesia.

To help Indonesia roll out its decentralized basic education program, the Asian Development Bank (ADB) supported a project to improve the participation in and completion of 9 years of basic education, particularly among the poor. The project included primary and junior secondary levels in the provinces of Bali, East Nusa Tenggara (NTT), and West Nusa Tenggara (NTB). It supported three outcomes: (i) improved participation, transition, completion, and performance in basic education among poor children in Bali, NTB, and NTT; (ii) implementation of school-based management in project schools; and (iii) effective district education management in the three provinces. To achieve these objectives, the project had three components: school development; district basic education development; and monitoring, evaluation, and reporting.

The project was financed by a combination of loan and grant. Of the $138.4 million loan component, $112.1 million was financed by ADB and $26.3 million by the Government of Indonesia. The grant component totaled $29.5 million, of which $28.0 million was funded by the Government of the Netherlands and $1.5 million by the Government of Indonesia.

This report presents performance evaluation findings of the Decentralized Basic Education Project based on four core evaluation criteria (relevance, effectiveness, efficiency, and sustainability) and two additional criteria (institutional development and impact). Overall, the project performance is rated successful.

The project is rated highly relevant. The timing of its implementation was responsive to addressing insufficient capacity during Indonesia’s transition to a decentralized basic education system. The project is also aligned with the education policies and strategies of both the government and ADB. Changes in the project coverage involving withdrawal of five Jakarta districts and redirection of this funding to two districts in impoverished NTT improved its relevance. Modifications in project design during the early phase of implementation, as well as weak coordination between the Ministry of National Education (MONE) and the Ministry of Religious Affairs (MORA), the project’s executing and co-executing agencies, respectively, posed challenges to implementation but did not compromise relevance either in terms of targeting schools and districts or in achieving the objective of improving access of the poor to basic education.

The project is effective. Analysis of primary time-series data obtained during the evaluation indicates that statistically significant improvements in education outcomes

xii Indonesia: Decentralized Basic Education Project

including enrollment, completion, transition, and performance were observed in selected project districts after the project was implemented. The project also helped to improve school capacity by promoting an evidence-based approach to formulating school development plans; encouraging community involvement in education matters; and improving district capacity for basic education, particularly in teacher development and the evaluation of school development plans.

The project is efficient. Despite the limited size of grants, the project supported notable achievements in improving school and district capacity for managing basic education and enhancing school infrastructure. The progress of its loan component was initially slow, but this picked up throughout project execution. The grant component was twice extended, as the implementation period of only 2.5 years proved too short for severely poor and often inaccessible districts.

The project is likely sustainable. The formulation of school development plans, which was pioneered by the project, has become standard practice among schools. School committees continue to participate in creating school strategies and budget plans. School and district officials who received training under the project continue to impart knowledge to their counterparts who were not part of the project. The central and district governments provide the primary source of funding to help project schools deliver continued support for poor students through scholarships, transport allowances, remedial and catch-up programs, and tutoring for students missing school.

The project is rated as having a significant impact on enhancing local capacity for basic education management, improving education outcomes, and shaping government policies related to decentralized basic education services. ADB’s performance is rated less than satisfactory, given that high turnover of its project officers constrained effective and continued supervision and management of the project. MONE’s performance is rated satisfactory, since it eventually gained sufficient capacity after initial difficulties.

The evaluation draws the following lessons:

i. Decentralization of basic education can be reinforced through strong collaboration between MONE and MORA. Structural misalignment hampers the coordination of these ministries in managing basic education because MORA remains centralized while MONE has been decentralized. Although both ministries may have misaligned structures and varying levels of resources and expertise, they are nevertheless jointly tasked with providing basic education. A more collaborative effort is needed for its successful delivery and management.

ii. Good technological infrastructure at the school and district levels is imperative for data management. Although there is an online database for school indicators, called Dapodik, schools have yet to fully maximize this medium because many lack internet access. Capacity building of school staff is also necessary for the online system to work effectively.

iii. High staff turnover impedes capacity building and disrupts efforts to promote decentralized basic education management. Frequent staff changes among ADB’s project officers and district governments hampered the transfer and sharing of knowledge, as it did communication and rapport-building among stakeholders.

Executive Summary xiii

Given these lessons, the project performance evaluation report makes these recommendations: i. Coordination between MORA and MONE, which is crucial for decentralized basic education, requires improvement. Best practices in education management need to be shared with officials of both ministries. Moreover, the role of provincial governments in decentralized basic education should be strengthened, particularly in monitoring and evaluation. This is because MORA’s provincial offices are primarily tasked with providing madrasah education, while responsibility for general education lies with district governments and district MONE offices. ii. Endeavors to enhance the delivery of basic education in Indonesia should be tied in with continued investments in enhancing transport and technological infrastructure and school facilities. Improving the access of the poor to transport is a priority everywhere, but particularly in remote areas. In these areas, improving access to junior secondary schools, which are normally located in subdistrict capital cities, is especially needed. Improvements in technological infrastructure in schools and districts are necessary to facilitate continuous data updates on the Dapodik system. Continued rehabilitation of classrooms in remote areas in Bali, NTB, and NTT is also necessary. iii. More than a decade since it was first implemented, decentralized basic education in Indonesia remains beset with challenges. These include its impact on teacher policy, given the considerable district authority in staffing (including over transfer of teachers) and the perceived politicization of education appointments made by district governments. Should ADB choose to continue providing support in this area, it should consider how such challenges could be best addressed.

CHAPTER 1 Introduction

A. Evaluation Purpose and Process

1. This project performance evaluation report assesses the Decentralized Basic Education Project, which was designed to support decentralized planning, management, and delivery of basic education services in Indonesia. The project aimed to enhance participation in and completion of 9 years of basic education, particularly among the poor in Bali, West Nusa Tenggara (NTB), and East Nusa Tenggara (NTT). The project was evaluated based on four core criteria (relevance, effectiveness, efficiency, and sustainability) and two additional criteria (institutional development and impact), as specified under Asian Development Bank (ADB) guidelines. 1 The evaluation also assesses the project’s impact on student performance in beneficiary districts, analyzes the roles of its stakeholders, and reviews other factors that may have affected project performance. This evaluation’s findings will feed into the forthcoming validation of Indonesia’s County Partnership Strategy 2012–2014 final review.

2. The project completion report (PCR) was prepared in June 2012. Overall, the project was rated successful based on the four core evaluation criteria. It was rated highly relevant to the poverty reduction priorities of ADB and the Government of Indonesia. Changes in the project’s scope—which included adding districts in impoverished NTT—further improved its relevance. The project was rated effective, based on (i) improvements in net enrollment, completion, and transition rates and examination scores (although no targets had been specified); (ii) implementation of school-based management in 3,000 schools and the establishment of effective community-school partnerships; (iii) increased district education funding over 2001 levels; (iv) development of data-based district education development plans; and (v) implementation of plans for quality and equity improvement. It was rated efficient because progress under the loan component picked up during implementation despite initial delays. It was rated partly sustainable. The project helped to mainstream school and district planning and school-based management through national policies. As noted in the PCR, however, some stakeholders took the view that activities financed under the project would not be sustainable without project support.

3. The project was assessed to have a likely significant impact on project districts and provinces given improved school management that involved both school officials and communities. The PCR noted, however, that it was difficult to assess the project’s impact on district capacity.

1 Independent Evaluation Department. 2006. Guidelines for Preparing Performance Evaluation Reports for Public Sector Operations. Manila: ADB (as well as its amendment effective from March 2013).

2 Indonesia: Decentralized Basic Education Project

B. Expected Results

4. The project sought to support the decentralized planning, management, and delivery of basic education services to improve participation in and completion of 9 years of basic education. It was implemented in 26 districts across 3 provinces (Bali, NTB, and NTT).2 The project targeted 5,238 schools in poor villages, of which 4,329 were primary and 909 were junior secondary schools. It supported three outcomes: (i) improved participation, transition, completion, and performance in basic education among poor children in the three provinces; (ii) implementation of school-based management in project schools; and (iii) effective district education management in the three provinces.

5. To achieve its objectives, the project had three components: school development; district basic education development; and monitoring, evaluation, and reporting. It supported the following major outputs: (i) implementation of school- based management, (ii) improved school quality with School Development Fund assistance, (iii) improved capacity of districts to plan and manage decentralized basic education as reflected in their 5-year district education development plans, (iv) improved district education with District Education Development Fund assistance, and (v) effective program and financial monitoring compliance. The project’s design and monitoring framework is presented in Appendix 1.

C. Evaluation Method

6. The evaluation used qualitative and quantitative techniques in examining the project’s performance. The qualitative approach was used for in-depth interviews with officers from the Ministry of National Education and district education offices, as well as school principals, teachers, and staff involved in the project. Appendix 2 provides a complete list of government and school officials interviewed during the evaluation mission. Appendix 3 presents the questionnaire for school officials.

7. The quantitative approach was particularly useful for examining education outcomes and poverty status. Various secondary data were used in analyzing education outcomes, including those for enrollment, transition, and completion rates; national exam scores; and poverty status in selected provinces and districts. The evaluation also used the findings of a survey conducted during the mission that provided primary time- series data on education indicators and other information related to the project. Survey questionnaires were distributed to selected project schools. The quantitative approach used in this evaluation study makes before-and-after comparisons. For some quantitative outcomes, such as enrollment, completion, and transition rates as well as exam scores, the analysis involved mean-difference testing to check robustness of the comparison analysis.

8. The survey’s sample of project schools was selected using a mixture of probabilistic and non-probabilistic approaches to ensure that each school in each province had an equal opportunity to be selected as part of the sample. The sample for this survey covered 56 schools, which are representative in terms of districts and types of schools.

2 Jakarta was initially included as one of the three target provinces. However, it was withdrawn from the project because its income was sufficient to achieve objectives similar to those of the project. The funds previously allocated to Jakarta were then redirected to NTT.

Introduction 3

9. Although not the case of Bali, schools in NTB and NTT are scattered on various islands within the two provinces. Hence, the study purposively selected the islands within these provinces. Lombok Island in NTB and Timor Island in NTT were chosen. The numbers of districts within the islands was then calculated proportionately, resulting in two districts each for Bali and NTT and three districts for NTB. A simple random sampling mechanism was used to select the districts on each of the islands. On that basis, the selected districts were Bangli and Buleleng in Bali; Lombok Barat, Lombok Timur, and Mataram in NTB; and Timor Tengah and Timor Tengah Utara in NTT. Taking into account differences in the local education policies, as well as socioeconomic characteristics in each of the provinces and districts, this study calculated proportionately the number of project schools to be surveyed for each district. This calculation was based on the number and types of schools covered by the project as presented in the PCR (Appendix 4). A stratified random sampling mechanism based on the level of school was then used to decide which project schools were selected as part of the sample.

CHAPTER 2 Design and Implementation

A. Formulation

10. The Government of Indonesia has implemented numerous measures to improve the access to and quality of basic education, including decentralizing the management and delivery of basic education following the passage of Law No. 22/1999 on Regional Governance and Law No. 25/1999 on Fiscal Decentralization. To this end, district governments have been tasked with providing financing for basic education.

11. In 1997, the government requested ADB support for improving the delivery of basic education in the provinces of Bali and West Nusa Tenggara (NTB) and the Special Capital Region Jakarta (footnote 2). The government selected these provinces because of their gaps in basic education and the regional demarcation between ADB and the World Bank.3 Jakarta withdrew from the project, however, and was replaced by East Nusa Tenggara (NTT).

12. ADB in 1998 approved small-scale project preparatory technical assistance (TA), but the Asian financial crisis caused project preparation to be suspended.4 This was resumed in 2000. The government requested that the project support the decentralization of basic education management to build on achievements of the ADB- funded Social Protection Sector Development Program. ADB subsequently approved small-scale TA to redesign the project.5 A loan appraisal mission was fielded during July 2001.

13. The project had no attached TA, but two related TA projects and a small project funded by the Japan Fund for Poverty Reduction complemented project implementation. Bridging TA providing support for decentralized education management commenced implementation in March 2002 and supported the initial steps in decentralizing the management of basic education to improve service delivery. This included capacity building for a special decentralization unit in the Ministry of National Education (MONE).6 Additional TA on integrating poverty considerations into decentralized education management began implementation in April 2003, shortly after project consultants were fielded. The TA supported poverty mapping for 16 districts covered by the project and 50 participatory poverty studies. 7 The project,

3 ADB. 2001. Report and Recommendation of the President to the Board of Directors on a Proposed Loan to the Republic of Indonesia for the Decentralized Basic Education Project. Manila. 4 ADB. 1998. Technical Assistance to the Republic of Indonesia for Basic Education in Bali and Nusa Tengara Barat Project. Manila (TA 3007-INO, $150,000, approved 16 April 1998). 5 ADB. 2000. Technical Assistance to the Republic of Indonesia for the Preparation of Decentralized Basic Education Project. Manila. (TA 3456-INO, $150,000, approved 14 June 2000). 6 ADB. 2006. Technical Assistance Completion Report: Support for Decentralized Education Management in Indonesia. Manila (TA 3701-INO). 7 ADB. 2007. Technical Assistance Completion Report: Integration of Poverty Considerations in Decentralized Education Management in Indonesia. Manila (TA 3957-INO).

Design and Implementation 5

backed by Japan Fund for Poverty Reduction, was approved in May 2002 and consultants were fielded 12 months later.8 It pilot tested school-based management and community-based education as a pro-poor service delivery scheme in particularly underserved schools.

14. The project was funded by a $100.0 million loan from ADB and grant cofinancing of $28.0 million from the Government of the Netherlands. The loan was approved on 29 November 2001 and closed on 31 December 2008. The grant was approved on 14 March 2006 and extended for 1 year, to 31 December 2009, to allow for project completion in NTT, and later to 31 December 2010 to cover emergency reconstruction work in Sumatra. Some $10.0 million was reallocated from the loan to rebuild schools after an earthquake and tsunami struck Aceh and North Sumatra in December 2004. Similarly, grant savings of $1.3 million were used in reconstructing 9 schools in West Sumatra after a 2009 earthquake.

B. Rationale

15. The project provided crucial support during Indonesia’s transition to a The project decentralized system of basic education. Its commencement coincided with the initial provided crucial implementation of Law No. 22/1999 on Regional Governance and Law No. 25/1999 on Fiscal Decentralization in 2001, during which a lack of technical guidance for provincial support during and district governments in executing the laws, especially for the education sector, Indonesia’s posed a challenge. The project therefore supported pioneering efforts in implementing transition to a 9 a decentralized education system. decentralized

16. Per the government’s request, the project supported the process of system of basic decentralizing basic education management, in accordance with the laws on regional education governance and fiscal decentralization (footnote 3). Prior to implementation of these laws, the education system was highly centralized. Basic education was managed centrally through complex and compartmentalized structures in three ministries: the Ministry of Home Affairs, MONE, and the Ministry of Religious Affairs (MORA). MONE was responsible for overall planning, coordination, and regulation for education at all levels and it directly administered and financed public junior secondary education. The Ministry of Home Affairs administered and delivered public primary education, overseeing teacher management, establishment and maintenance of primary schools, and provision of subsidies to private primary schools. MORA administered and financed public primary and junior secondary madrasah schooling and supervised and provided financial assistance to private madrasahs. This system had been replicated at provincial and district levels, where all three ministries were represented in parallel by local offices. Similar to this complex management structure, budgeting and financing for basic education involved five ministries: the Ministry of Finance, Ministry of Home Affairs, MONE, MORA, and the National Development Planning Agency, with a mix of financing arrangements and funds sources for each school type and level.

17. Under Indonesia’s decentralized basic education system, district governments are now responsible for planning, financing, managing, and delivering basic education. Their responsibilities also cover school staff appointments and transfers and school location planning. The central government sets policies, the national curriculum, and a national examination and assessment system. It also sets standards for (i) achievement at each age; (ii) learning materials; (iii) acceptance, transfer, and certification of

8 ADB. 2007. Implementation Completion Memorandum for INO: Supporting Community-Based Basic Education (JFPR 9016). Manila. 9 Study team’s interviews with stakeholders during evaluation mission.

6 Indonesia: Decentralized Basic Education Project

students; and (iv) learning times for basic education and fixing the educational calendar. The central government establishes procedures and standards for staff appointments and dismissals; determines pensions, salaries, benefits, and rights and obligations; and establishes the legal status of civil servants and regional civil servants, including teachers and principals.

18. Provincial governments (i) define policy on student selection and acceptance, including equity for minority students and students from poor families and remote areas; (ii) provide instructional materials for basic education; (iii) assist in higher education management, curriculum production, accreditation, staff appointments, and opening and closing of colleges; and (iv) manage special institutions, including training centers, teacher training institutions, and special education. Provincial governments may also decide on the transfer between districts of civil servants, including school principals and teachers, and their transfer between cities and provinces (footnote 3).

19. By supporting the decentralized system, the project also responded to challenges in improving the equity in and quality of delivering basic education services.10 The proportion of primary school-age children enrolled at this level—the net enrollment rate—was 93.0% at the start of project implementation in 2001, while the ratio of female-to-male primary enrollment was 98.0%. Nearly 660,000 primary school- aged children were out of school in that year, more than 60.0% of them girls.11

20. ADB’s development agenda for Indonesia also prioritizes basic education. Until 1990, ADB support for Indonesia’s education sector was channeled mainly to the vocational, technical, and higher education subsectors. Since then, however, ADB has increased support for basic education to improve poor children’s access to basic education and enhance learning outcomes at this level. Among ADB’s basic education investments in Indonesia is the Social Protection Sector Development Program, which supported a scholarship and school block grant program. The Social Protection Sector Development Program was introduced in the aftermath of the Asian financial crisis. It helped maintain stability in primary and junior secondary school enrollment and supported policy dialogue for improving social safety nets mechanisms.

C. Cost, Financing, and Executing Arrangements

21. At appraisal, the total cost of the loan component was $125.0 million. At loan closing, the actual project cost was $138.4 million, of which $112.1 million was financed by ADB and $26.3 million by the government. The appreciation of special drawing rights versus the dollar caused the 11.0% increase in the loan amount. There were no cost overruns or underruns since project expenditures were linked to school and district development plans.

22. In terms of cost categories, the PCR noted that expenditure on research, surveys, and studies (42.0% of the appraisal estimate) was low because the project had not carried out baseline and midterm evaluation surveys and other studies. 12 The evaluation team reviewed two impact evaluation documents produced under the project: (i) Decentralized Basic Education Project: Consulting Services for Baseline and Benefit Impact/Evaluation Studies, Initial Findings (2006); and (ii) Benefit, Monitoring,

10 Indonesia’s education system consists of preschool, primary education (grades 1 to 6), junior secondary education (grades 7–9), senior secondary education (grades 10–12), and tertiary education. Basic education covers 9 years of compulsory primary and junior secondary education. 11 ADB. 2013. Key Indicators for Asia and the Pacific. Manila. 12 ADB. 2012. Completion Report: Indonesia: Decentralized Basic Education Project. Manila (p. 7).

Design and Implementation 7

and Evaluation Studies of the Decentralized Basic Education Project (Baseline and Midterm Evaluation Studies): Final Report (2008).13 However, these impact evaluation studies did not clearly identify the type of impact measurement being calculated. Moreover, only baseline and midline evaluations were conducted. The baseline study did not incorporate the appropriate control group as the respondents at the school level only included principals, teachers, students, and school committees from 55 project schools. While the midterm report stated that it included non-project schools, it did not sufficiently discuss how the non-project schools were selected. Moreover, the baseline report used the district minimum service standard as the comparison benchmark, which may result in misleading findings since the minimum service standard may also include project schools. Given the different sampling methods and questionnaires used by the baseline and midterm evaluation studies, the study team assumed there to be no connection between the two. The study team also had no access to the data used in the studies. As the PCR had noted, due to the absence of data, information for measuring project benefits also relied on the memories of the stakeholders.14

23. The estimated cost of the grant component was $36.3 million, of which $28.0 million was to be funded by the Government of the Netherlands and $8.3 million by the Government of Indonesia. The actual cost was $29.5 million, of which the Netherlands funded $28.0 million and Indonesia $1.5 million.

24. At loan closing, the project had disbursed $112.1 million, about 112% of the original loan amount and 97.0% of the current loan value, which included an initial advance of $2.6 million (25 June 2002) to finance the cost of various small contracts. The project disbursed $28.0 million in grant funds, including the ADB administration cost of $0.5 million. The loan account was financially closed on 30 June 2009 and the grant on 30 September 2011.

25. MONE was the project’s executing agency. MORA was the co–executing agency and assigned staff to project management offices at the different levels of government. At national level, a project steering committee provided policy guidance. District coordination committees, education boards, and school committees assisted the steering committee. Meanwhile, the central technical committee assumed technical and programmatic functions and the central facilitation unit managed overall financial and administrative coordination. The facilitation unit was supported by provincial and district project management units. Although complex, this implementation structure was appropriate given the project’s large size and constant changes in administrative context (footnote 3).

13 The project’s baseline and midterm evaluations can be accessed in ADB’s Archives Section. See Insan Hitawasana Sejahtera. 2008. Decentralized Basic Education Project: Consulting Services for Baseline and Benefit Impact/Evaluation Studies, Initial Findings. Jakarta; and Indonesia Ministry of National Education and Indonesia University of Education. 2006. Benefit, Monitoring and Evaluation Studies of the Decentralized Basic Education Project (Baseline and Midterm Evaluation Studies) Final Report. Jakarta. 14 Proper impact evaluations should include appropriate design and identification of control and treatment groups. The design of an impact evaluation should be specified at the beginning of a program and include the type of measurement impact (e.g., average treatment effect, average treatment on the treated, spillover), sampling method of both treatment and control groups, and the evaluation method. The selection of treatment and control groups should also be adequately discussed. Impact evaluations should be done also at the baseline and endline of a project.

8 Indonesia: Decentralized Basic Education Project

D. Procurement, Construction, and Scheduling

26. The loan component was approved in 2001, took effect in 2002, and closed on schedule in 2008. Although there were initial delays, the pace of implementation picked up. Some variations in project implementation procedures were adopted. Phased implementation was originally identified at the planning stage, whereby the batches were based on schools within districts rather than on the districts themselves. Phased implementation was abandoned, however, because districts wanted to commence project activities immediately. Given this change, all original districts commenced project activities in mid-2003 and all districts continued to receive support over the next 5 years. Within these districts, three different batches of schools also received support for 3 years. Challenges were also encountered in use of the School Development Fund, which was released a year before consultants were mobilized and before schools had formulated their school development plans.

27. The grant was approved in 2006 and extended twice before it was closed in 2010 due to (i) a 1-year extension to make sure that the programs benefited NTT, and (ii) a second 1-year extension to allow grant savings to be used for reconstructing schools in Padang Pariaman.

28. Civil works involved rehabilitation of existing facilities as well as construction of earthquake-resistant schools on the remote island of Simeulue and rehabilitation of earthquake-ravaged schools in Padang Pariaman.

29. Schools and districts carried out most of the procurement for items eligible under the District Education Development Fund and the School Development Fund. District project management units procured eligible items under the district education fund through direct purchases using limited local tenders. On the other hand, schools procured eligible items under the School Development Fund through direct purchase and school committees managed procurements for the rehabilitation of schools. Cash expenditures used direct purchase and were supplemented by such in-kind contributions from the community as materials and labor. Training and workshops were provided by facilitators and trainers according to ADB and government guidelines.

30. Several procurement problems were uncovered by the external auditor, most of which were related to overpayment for workshops and training. As noted in the PCR, the findings of the external audit indicate that community monitoring of school activities and finances is a viable method of ensuring compliance.

E. Outputs

31. The project supported three major components: (i) school development; (ii) district basic education development; and (iii) monitoring, evaluation, and reporting. These components were supported by the following outputs.

32. Implementation of school-based management. The project improved the schools’ capacity in developing and implementing school development plans. Project schools were required to develop these plans to access project funds for scholarships for the poor, rehabilitation of classrooms and other facilities, and other activities identified in their plans. About 97% of the 4,244 project schools drew up these plans,

Design and Implementation 9

according to the baseline study.15 However, project schools were not required to report their achievements against their plans. The project also helped to strengthen community involvement in schools through the formation of school committees.

33. Improved school quality with School Development Fund support. This was the project’s funding mechanism to support activities identified in school development plans. To ensure transparency and accountability of the School Development Fund, its accounts were audited annually by external auditors. Public accountability systems were also established to monitor the project’s operation. As noted in the PCR, 95% of schools reported on the School Development Fund, program and budget plans, and program and budget implementation. Moreover, 85% of schools were reported to have established complaint mechanisms, though only 10% of school principals were seen as willing to receive complaints.

34. Improved capacity of districts as reflected in a 5-year district education development plan. The project established district education boards in compliance with Ministerial Regulation No. 044/2002. These were tasked with monitoring district education programs, as well as school development plans. The boards were provided training on education management, including budget analysis. The legal framework for the boards was revised in 2010 by a new regulation on the management of education (PP 17/2010). As a result, funding of the boards by districts became discretionary and the number of board members was reduced. Although board members work in a voluntary capacity, they nevertheless require funding for their operations. The project also required all districts to publish their education statistics annually and to complete their district education development plans. During the project, all districts published statistics annually and completed their district education development plans. Some districts continue to update their plans and publish their statistics, but weaknesses in information systems make it difficult to keep the data current and for them to be able to publish the statistics on time.

35. Improved district education with District Education Development Fund support. To improve district education, the project supported the preparation, implementation, and monitoring of strategic 5-year district education development plans and 5-year annual action plans by all districts. Under the plans, districts developed systems for planning, managing, financing, staffing, and monitoring teacher development; school supervision; recruitment, training, and performance review of school principals; facilities expansion; and district financing of education. The fund supported the implementation of priority district education development programs and capacity building activities.

36. Effective program and financial monitoring compliance. The project helped promote transparency and accountability as part of school-based management. Schools inform communities of their school development plans, programs, and budget plans, as well as their School Development Fund and District Education Development Fund accounts.

F. Consultants

37. Under the loan component, three packages of consulting services were envisaged during appraisal: (i) school-based management; (ii) district capacity building

15 Insan Hitawasana Sejahtera. 2008. Decentralized Basic Education Project: Consulting Services for Baseline and Benefit Impact/Evaluation Studies, Initial Findings. Jakarta.

10 Indonesia: Decentralized Basic Education Project

and project implementation support; and (iii) independent monitoring, evaluation, and reporting. Some 82 person-months of international consultant inputs (compared with 78 person-months at appraisal) and 8,403 person-months of national consulting inputs (compared with 7,782 person-months at appraisal) were used. The consultants for the loan component were mobilized a year after project activities commenced.

38. Under the grant component, two additional packages were planned: (i) school and district building and project implementation support, and (ii) independent monitoring and evaluation. The grant packages mobilized only national consultants, using 1,288 person-months of national consulting inputs (against the planned 960 person-months). All consultants were recruited in accordance with ADB’s Guidelines on the Use of Consultants. Several changes requested by the consulting firm led to considerably greater person-months and cost. A university research institute was contracted in 2006 to conduct the baseline and midterm evaluation study, but it performed poorly. The institute was replaced by a social survey firm in 2007, which conducted the required baseline and impact studies.

39. District project and school personnel considered package (i) and (ii) consultants under the loan component helpful or very helpful, but they found package (iii) consultants too focused on monitoring schools’ utilization of the School Development Fund. The consultants for package (iii) only monitored financial compliance in the schools and districts. They did not conduct program monitoring and evaluation after this activity was removed from their contract. The monitoring and evaluation activities for the loan project did not refer to the design and monitoring framework indicators. Meanwhile, as noted by the PCR, the consultants under the grant component performed well and achieved good results.

G. Covenants

40. Twenty-five of 27 covenants were complied with. The loan covenants on the role of the provincial project monitoring units and gender were only partially complied with. These units were envisaged to monitor or evaluate project activities and impact but failed to do so. As a result, only progress and performance were monitored, not the achievement of indicators and targets. The provinces were originally assigned to conduct performance monitoring and evaluation, but this was not carried out. An independent contractor was hired to conduct an impact study on beneficiary schools under the loan project and baseline and impact studies on beneficiary schools in NTT under the grant. Meanwhile, the covenant on gender required two women members per school committee. In NTT, almost 70% reached this target, but results in Bali and NTB were poor.

H. Policy Framework

41. The project influenced national government policy and procedures, and most notably the School Operational Assistance Program (BOS), which was launched in 2005. BOS partly adopted the project’s block grant system, and it provides block grants to all schools to enhance the implementation of school-based management and lower the cost of providing basic education. While compulsory fees from parents may not be retained, the central government permits local governments and parents to financially

Design and Implementation 11

support schools if BOS funds are insufficient.16 The disbursement of BOS funding is decided by school committees in a similar procedure established by the project.

42. The project also influenced legally binding national education standards introduced from 2006 to 2008 and minimum service standards for districts introduced in 2010. Moreover, school-based management is now official policy and implemented in all schools nationwide. However, the school-managed model for rehabilitation under the project has been discontinued under new legislation requiring a return to district contracting. Moreover, many districts now restrict parental and community financial support for schools, which was one of the project’s achievements.

16 J. Tobias et al. 2014. Towards Better Education Quality: Indonesia’s Promising Path. London: Overseas Development Institute.

CHAPTER 3 Performance Assessment

A. Overall Assessment

43. Overall, the project is rated successful. The overall assessment is based on equally weighted individual assessment criteria: relevance, effectiveness, efficiency, and sustainability (Table 1), which is in accordance with the Guidelines for Preparing Performance Evaluation Reports for Public Sector Operations. This rating is generally consistent with that of the PCR, but the rating for sustainability differs somewhat (Appendix 5). Other criteria, such as impact and ADB and borrower performance, are discussed, but not formally assessed because of difficulties in precise attribution and quantification.

Table 1: Assessment of Project Performance Weight Rating Weighted Criterion (%) Assessment Value Rating Relevance 25 Highly relevant 3 0.75 Effectiveness 25 Effective 2 0.50 Efficiency 25 Efficient 2 0.50 Sustainability 25 Likely sustainable 2 0.50 Impact Significant Institutional Development Satisfactory ADB Performance Partly satisfactory Borrower Performance Satisfactory Overall Rating 100 Successful 2.25 ADB = Asian Development Bank. Note: Highly Successful (>2.7), Successful (2.7 ≥ S > 1.6), Less than Successful (1.6 ≥ LS > 0.8), Unsuccessful (<0.8). Source: Study team.

B. Relevance

44. The project is rated highly relevant. As stated in the rationale section, the project and the timing of its implementation responded to challenges Indonesia faced during its transition to a decentralized basic education system, including weak institutional capacity; limited human and financial resources; and unclear delineation of roles, functions, and authority of central and local government agencies. The project’s components are therefore deemed appropriate to help address these gaps by empowering schools and district governments in identifying and addressing education needs in their communities.

45. The project also remained relevant despite legislative changes. Fiscal and administrative decentralization took effect on 1 January 2001. The government passed the National Education System Law in 2003 and the National Development Planning

Performance Assessment 13

System Law in 2004. Between 2002 and 2004, it passed other laws relevant to decentralization, such as those on regional government and central–regional financial balance, the national planning and financial system (budgeting), and the national treasury (payments mechanism).

46. The project’s objective of improving basic education outcomes, including participation, transition, completion, and performance, was also relevant to Indonesia’s education challenges at the time of project implementation. Although Indonesia has nearly achieved universal access to primary education, participation in secondary education remains a concern. The average net enrollment rate of children aged 7–15 years was 94% in primary school and 59% at the secondary level at around the time of project implementation in 1998–2002. 17 Figures 1 and 2 indicate that average net enrollment in Nusa Tenggara Timur (NTT) was generally lower than the national average for both primary and secondary levels during 2003–2013, while net enrollment in Bali and Nusa Tenggara Barat (NTB) was slightly higher for much of the period. Low average enrollment in NTT may be attributed to the poor quality of basic infrastructure and school facilities, which restrict children’s access to education services. The slump in Indonesia’s net enrollment rate from 2010 to 2011 coincides with a decrease in public spending on education. Public spending on education reached 3.5% of GDP in 2009 but decreased to 3.0% and 2.8% of GDP in 2010 and 2011, respectively.18 As discussed, the government selected the provinces based on education gaps and the regional demarcation between the Asian Development Bank (ADB) and the World Bank.19

Figure 1: Primary Net Enrollment in Selected Provinces, 2003–2013 100

96.6 95.5 95 94.3 93.6

Percent 90

B a l i Nusa Tenggara Barat

Nusa Tenggara Timur Indonesia 85 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013

Note: Since 2007, the net enrollment rate includes nonformal education (Package A, which is primary school, and Package B, which is junior secondary). Source: Study team calculations based on Indonesian Statistics Office data.

17 National Development Planning Agency. 2002. National Program for Indonesian Children: Education. Jakarta. 18 Figures are World Bank estimates accessed at data.worldbank.org on 6 October 2014. 19 The World Bank at the time supported basic education projects in the provinces of Central Java, East Java, Kalimantan, Nusa Tenggara Timur, Sulawesi, Sumatra, and West Java, and was preparing a new basic education project covering East Java and South and West Sumatra. See ADB. 2001. Report and Recommendation of the President to the Board of Directors on a Proposed Loan to the Republic of Indonesia for the Decentralized Basic Education Project. Manila.

14 Indonesia: Decentralized Basic Education Project

47. Improving enrollment at junior secondary level is also an area of concern in Indonesia (Figure 2). Several factors influence low enrollment at the junior secondary level, including household welfare, gender, religious background, child’s performance, supply-side readiness, and employment opportunity. Children from poor households are less likely to enroll in junior secondary school, as are girls and children from Muslim backgrounds. Children with higher national exam scores in primary school are more likely to enroll in junior secondary school. Enrollment at junior secondary level is also found to be positively affected by the number of schools built in those areas where the children live. Finally, children tend to continue their school through junior secondary level if there are few employment opportunities in the community.20

Figure 2: Junior Secondary Net Enrollment in Selected Provinces, 2003–2013

90 80.2 80.7 73.7

60 59.2

Percent 30

B a l i Nusa Tenggara Barat

Nusa Tenggara Timur Indonesia

0 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013

Note: Since 2007, the net enrollment rate includes nonformal education (Package A, which is primary school, and Package B, which is junior secondary). Source: Study team calculations based on Indonesian Statistics Office data.

48. Children dropping out of school also undermined efforts to improve Indonesia’s basic education system. About 600,000 Indonesian children dropped out of primary school in 2001, at the start of project implementation, putting the dropout rate at 2.7%.21 Among junior secondary school students in 2001, the dropout rate was 3.5%, meaning that about 300,000 students left junior secondary school in that year. NTB and NTT had higher dropout rates in 2001 than the national average, at 3.2% and 5.4%, respectively, while Bali had a lower rate of 2.1% (Table 2). In Indonesia, poor children tend to give up schooling for working. Some evidence nevertheless suggests that banning children from working could worsen their school attendance since they will not have sufficient financial resources to continue studying.22

20 Suryadarma et al. 2006. Causes of Low Secondary School Enrollment in Indonesia. SMERU. 21 National Office for Research and Development. 2002. Education Statistics. Ministry of National Education. Jakarta. 22 A. Suryahadi, A Priyambada, and S. Sumarto. 2005. Poverty, School, and Work: Children during the Economic Crisis in Indonesia. Development and Change. 36 (2). pp. 351–373.

Performance Assessment 15

Table 2: Dropout rate of Primary and Junior Secondary Students in Selected Provinces, 2001/2002 (%) Province Primary Junior Secondary Bali 2.1 1.9 Nusa Tenggara Barat 3.2 2.9 Nusa Tenggara Timur 5.4 3.4 Indonesia 2.7 3.5 Source: National Office for Research and Development. 2002.

49. The project was particularly relevant to improving the transition rate of junior secondary school-aged children in NTT. Some of the poorest villages are located near the border of Timor-Leste. According to interviews with stakeholders, the nearest junior secondary school was about 15 kilometers from these villages. In addition, the roads leading to the subdistrict capital, where junior secondary schools are normally located, were in very poor condition, making parents reluctant to send their children to junior secondary schools. Through the project, these districts received assistance to build junior secondary schools, such as those in Biboki Utara and Bikome Utara, allowing children to continue their studies at the junior secondary level.

50. The project was also relevant to the need at the time of implementation for rehabilitating school infrastructure. During 2001/2002, 24% of primary school buildings in Indonesia, including classrooms, showed signs of severe damage and 33% of slight damage (Appendix 6, Table A6.3). About 4% of junior secondary buildings were badly damaged and 10% slightly damaged (footnote 17). In NTT, Bali, and NTB, respectively, 66%, 55%, and 50% of primary classrooms were damaged. It was difficult to ascertain the project’s contribution to improving school facilities at the time of the evaluation mission because other sources of funding were made available for school building rehabilitation after the project’s life. Interviews with Ministry of National Education (MONE), school, and district officials revealed that the government has been rehabilitating schools since 2012, allocating as much as Rp17.5 trillion (equivalent to $1.5 billion) for this endeavor.

51. The project was also relevant to government and ADB education policies and strategies. As discussed earlier, it was highly aligned with the country’s decentralization strategy, which serves as a means for improving the quality of and access to basic education by building a more integrated education system at local level that can effectively respond to local needs and conditions. ADB’s Country Assistance Plan, 2001– 2003 for Indonesia acknowledged decentralization of basic education as a major area of ADB support, particularly in addressing the need to develop local management capacity and to ensure better provision of basic education for the poor. The ADB strategy also recognized that although decentralization could improve transparency, accountability, and community participation, there were risks, particularly during the transition period. These included inadequate technical capacity and human and financial resource constraints that could hamper the efficient delivery of basic services, including education. The strategy stated that ADB will help redefine the role and support the reorganization of MONE in line with the decentralization law that was being newly implemented at that time.23

52. Decentralization in Indonesia has attracted support from many donors. These include, among others, ADB, United Nations agencies, and the World Bank. Alongside the government, these three launched a partnership in April 2000 to support governance reforms, including decentralization. ADB assumed a major role in

23 ADB. 2000. Country Assistance Plan: Indonesia, 2001–2003. Manila.

16 Indonesia: Decentralized Basic Education Project

decentralization and various anticorruption initiatives. The project has been the model after which other donor-funded projects on decentralized basic education in Indonesia were patterned. The United States Agency for International Development implemented a similar project which was patterned after the project. ADB’s Country Assistance Plan, 2001–2003 for Indonesia also stated that ADB, UNICEF, and the World Bank shared sector support in education on a geographic basis (footnote 3). For instance, the The project’s project excluded Flores Island in NTT because the Australian government was already scope was implementing a primary education project in that area at the time. relevant to the 53. The project’s scope was relevant to the objective of improving the quality of objective of and access to basic education, particularly among the poor. The changes in scope, improving the which saw five districts in Jakarta withdraw and the corresponding funding reallocated quality of and to two districts in NTT, made the project even more relevant. NTT and NTB are among the poorest provinces in Indonesia (Table 3). access to basic education, Table 3: Poverty Incidence and Annual Changes in Poverty Rates in Indonesia (%) particularly Poverty Incidence Average Annual Change (2006– among the poor 1996 2006 2011 2011) Province (%) Rank (%) Rank (%) Rank (%) Rank Jakarta 2.4 1 4.6 1 3.8 1 3.6 32 Bali 7.8 2 7.1 2 4.2 2 8.1 3 South Kalimantan 8.5 3 8.3 3 5.3 3 7.3 8 Bangka Belitung 10.9 5 5.8 4 9.5 1 Banten 9.8 4 6.3 5 7.1 10 Central Kalimantan 13.5 4 11.0 6 6.6 6 8.1 5 East Kalimantan 9.7 6 11.4 8 6.8 7 8.1 4 Riau Islands 12.2 11 7.4 8 7.8 6 Riau 12.6 8 11.9 10 8.5 9 5.7 19 North Sulawesi 17.9 7 11.5 9 8.5 10 5.3 25 West Kalimantan 24.2 13 15.2 17 8.6 11 8.7 2 Jambi 14.6 5 11.4 7 8.7 12 4.8 28 West Sumatra 9.8 9 12.5 12 9.0 13 5.5 21 North Maluku 12.7 13 9.2 14 5.6 20 South Sulawesi 16.7 11 14.6 15 10.3 15 5.9 17 West Java 11.1 10 14.5 14 10.7 16 5.3 24 North Sumatra 13.2 12 15.0 16 11.3 17 4.9 27 Indonesia 17.6 17.8 12.5 5.9 West Sulawesi 20.7 19 13.9 18 6.6 11 East Java 22.1 16 21.1 21 14.2 19 6.5 13 South Sumatra 15.9 15 21.0 20 14.2 20 6.4 14 Southeast Sulawesi 29.2 20 23.4 25 14.6 21 7.5 7 Central Java 21.6 17 22.2 22 15.8 22 5.8 18 Central Sulawesi 22.3 21 23.6 26 15.8 23 6.6 12 DI Yogyakarta 18.4 14 19.2 18 16.1 24 3.2 33 Lampung 25.6 18 22.8 23 16.9 25 5.1 26 Bengkulu 16.7 19 23.0 24 17.5 26 4.8 29 Gorontalo 29.1 29 18.8 27 7.1 9 Aceh 12.7 23 28.3 28 19.6 28 6.2 15 Nusa Tenggara Barat 32.0 22 27.2 27 19.7 29 5.5 23 Nusa Tenggara Timur 38.9 24 29.3 30 21.2 30 5.5 22 Maluku 44.6 25 33.0 31 23.0 31 6.1 16 West Papua 41.3 32 31.9 32 4.6 31 Papua 42.3 26 41.5 33 32.0 33 4.6 30 Sources: United States Agency for International Development and DAI/Nathan Group. 2013. Provincial Poverty Rates in Indonesia, 2006–2011. Jakarta.

Performance Assessment 17

54. Although Bali may have lower poverty compared to other Indonesian provinces, some of its districts and villages continue to face rampant poverty.24 In the district of Buleleng, for example, some 37,644 people remained poor in 2010. Similarly, the percentage of poor in the district of Jembrana was 8.1% in 2010, compared to Bali’s 4.9% average in the same year. Meanwhile, 21.6% of NTB and 23.5% of NTT were deemed poor in 2010 (Figures 3, 4 and 5).

Figure 3: Percentage of Poor in Bali’s Districts, 2010 9 8.1 7.9 7.6 7.4 8 6.9 6.7 7 6.4 6 4.9 5 4 3.2 2.2

% of poor % of 3 2 1 0

District Source: Secretariat of the National Team for the Acceleration of Poverty Reduction, Office of the Vice President of the Republic of Indonesia. 2011.

Figure 4: Percentage of Poor in Nusa Tenggara Barat Districts, 2010 50 43.1 45 40 35 30 23.8 21.8 21.7 21.6 21.6 25 19.9 19.8 19.4 20

% of poor % of 14.4 15 12.8 10 5 0

District

Note: North Lombok is a new district, gaining its autonomy in 2008. It was previously part of the district of West Lombok. Source: Secretariat of the National Team for the Acceleration of Poverty Reduction, Office of the Vice President of the Republic of Indonesia. 2011.

24 Secretariat of the Acceleration of Poverty Reduction. 2011. Regional Welfare Indicator: Province of Bali. Jakarta (Office of the Vice President of the Republic of Indonesia).

18 Indonesia: Decentralized Basic Education Project

Figure 5: Percentage of Poor in Nusa Tenggara Timur Districts, 2010 45.0 41.1 40.0 34.0 32.8 35.0 32.4 31.7 29.9 30.0 28.7 26.7 25.9 22.9 22.7 23.5 25.0 21.6 21.2 20.8 20.4 20.0 15.5 13.4 15.0 12.7 12.1 10.6 9.6 10.0

5.0

0.0

Source: Secretariat of the National Team for the Acceleration of Poverty Reduction, Office of the Vice President of the Republic of Indonesia. 2011.

55. Although the project’s design was modified in the early stages of implementation, it remained relevant to the project’s objective of improving poor people’s access to basic education. The four-tier criteria for school selection (for primary and junior secondary schools and madrasah) were simplified to three tiers, in which subdistricts, villages, and schools were ranked. The targeting criteria were eventually simplified further for both the loan and grant to a one-tier procedure based on the poverty ranking of schools within each district. Similarly, demand-led financing was modified, and all schools were given standard amounts of block grant for rehabilitation and quality improvement.

56. These changes in design did not appear to compromise project relevance, particularly in targeting. MONE first selected the poorest districts and then the poorest subdistricts within these impoverished districts. In selecting schools, district education officials conducted site visits to check the actual conditions on the ground and the level of community involvement. Interviews with MONE, school, and district officials indicated that schools in areas where community involvement needs improvement were often considered candidates as project schools. MONE pointed out that the Ministry of Religious Affair’s (MORA’s) weak regulation of madrasah posed challenges during project implementation, noting MORA’s lack of clear guidelines for the establishment of new madrasah. 25 According to interviews with stakeholders, since the project provided grants that a school could use depending on its need, some new schools were

25 During the study team’s interviews, MONE said it follows certain criteria in establishing public schools, examining first the number of prospective students and the number of students already absorbed by existing schools. For example, there should be at least three primary schools in an area to allow for the construction of a junior secondary school. Moreover, the distance between the new and existing junior secondary school should be 5 kilometers. Unless existing schools do not meet minimum service standards or cannot accommodate the number of students, the request for a new school is rejected. In contrast, according to interviews with MONE officials, Islamic schools were built even if they were just 100 meters away from an existing one.

Performance Assessment 19

built in order to access the block grant. In Lombok, anecdotal evidence reveals that one primary Islamic school split into two. To address this problem, MONE issued a policy that only schools already existing for 5 years can access the block grant.

57. In selecting poor students, schools usually obtained student socioeconomic data during enrollment, including household income and parents’ employment status. Students who are orphans are often accorded priority, according to interviews with stakeholders. School officials also conducted house visits to verify students’ living conditions. Since the grants provided under the project are relatively small, it was imperative to be highly selective of the project districts and schools, as the interviews reveal.

58. The project’s design was also able to accommodate emergency needs for school infrastructure rehabilitation in the aftermath of the earthquake and tsunami that hit Aceh and North Sumatra in December 2004. As discussed in Chapter 2, loan and grant savings were used to rebuild damaged schools. About 42 schools were reconstructed in the affected areas and a model for earthquake-resistant school buildings was developed.

C. Effectiveness

59. The project is assessed as effective. This evaluation found evidence that education outcomes, including enrollment, completion, transition, and performance in selected beneficiary districts, improved after project implementation. Interviewed stakeholders deemed enhanced school capacity to formulate school development plans and greater community involvement in education concerns to be the project’s most important contributions.

60. Improved education outcomes. Access to basic education improved in the beneficiary provinces (Table 4). Between 2001 and 2011, net enrollment rates at primary level increased in NTB and NTT but slightly decreased in Bali. Primary net enrollment in NTT and Bali remained slightly below the national average in 2011. Net enrollment at the secondary level increased in all three provinces during 2001–2011. NTT’s secondary net enrollment increased from 38.0% in 2001 to 60.9% in 2011 but remained below the national average over the last decade.

61. Completion rates have also improved. NTT saw a notable increase in the secondary completion rate from 92.8% in 2001 to 97.7% in 2011. However, both primary and secondary completion rates in NTT and NTB remained below the national average in 2011, while Bali’s were higher than the national average in that same year (Table 5).

20 Indonesia: Decentralized Basic Education Project

2011 2011 91.07 73.27 94.15 75.29 91.58 60.89 92.00 69.83 99.49 98.90 98.89 96.81 98.72 97.73 99.05 96.96

2010 2010 90.91 77.10 95.65 79.49 91.34 52.15 91.55 69.67 99.26 99.27 99.20 97.79 98.46 98.00 98.84 97.50

2009 2009 92.18 75.54 94.71 77.84 89.61 51.20 91.76 67.97 99.01 99.07 99.91 96.94 98.00 95.98 98.97 97.01

2008 2008 94.78 67.63 94.50 71.66 92.01 49.71 94.15 67.26 99.28 93.12 99.16 95.60 98.88 95.37 99.18 95.66

2007 2007 94.02 73.97 95.47 67.00 91.57 38.68 93.66 66.13 99.12 99.14 98.76 97.29 97.82 97.88 99.04 97.08

2006 2006 93.33 70.15 94.50 69.62 91.58 47.23 93.54 66.52 98.83 96.90 98.57 97.22 98.38 97.17 98.70 97.11 (%) (%) (%) (%)

2005 2005 93.36 69.75 93.32 69.38 92.16 43.30 93.25 65.37 98.94 97.32 98.77 97.01 98.62 97.31 98.92 96.92 .

2004 2004 93.67 70.75 91.58 61.08 90.68 44.22 93.04 65.24 98.94 97.77 97.90 95.48 98.19 97.30 98.68 97.50

Table 5: Completion Rates in Selected Provinces, 2001–2011 Provinces, 2001–2011 in Selected Completion Rates Table 5: Table 4: Net Enrollment Rates in Selected Provinces, 2001–2011 2001–2011 Provinces, in Selected Enrollment Rates Net Table 4: 2003 2003 91.73 69.12 92.20 61.21 88.88 41.16 92.55 63.49 98.77 98.16 97.59 94.89 97.74 96.40 98.54 97.18

2002 2002 92.68 67.97 92.65 60.28 87.44 40.89 92.70 61.69 98.56 97.57 98.26 94.79 97.52 97.11 98.27 97.09

2001 2001 93.02 69.19 91.74 56.95 88.52 38.00 92.88 60.47 98.61 97.83 98.35 92.62 97.70 92.77 98.54 96.16 Item

Item Bali Primary Secondary Nusa Tenggara Barat Primary Secondary Nusa Tenggara Timur Primary Secondary Indonesia Primary Secondary Source: Study team’s calculations based on national socioeconomic survey data Bali Primary Secondary Nusa Tenggara Barat Primary Secondary Nusa Tenggara Timur Primary Secondary Indonesia Primary Secondary Source: Study team’s calculations based on national socioeconomic survey data. Performance Assessment 21

62. National exam scores also improved, indicating better student performance.26 Although average scores in Bali and NTB were lower than the national average in 2001, they were higher than that average in 2012. NTT’s average exam scores were lower than the national average in 2012 (Table 6).

Table 6: National Exam Scores in Selected Provinces Average Average Percentage Province (2001) (2012) Increase Bali 6.48 8.06 24.38 Nusa Tenggara Barat 6.32 7.87 24.52 Nusa Tenggara Timur … 6.83 … Indonesia 6.54 7.48 14.37 Sources: Study team’s calculations based on data from Indonesian Family and Life Survey and Ministry of National Education.

63. Improvements in enrollment, primary completion, junior secondary transition, and exam scores were observed in selected project districts after the project was implemented. Tables 7 and 8 present the education outcomes in 56 surveyed project schools located in six districts before and after project implementation. These improvements were observed both for poor students as a group and for all students. According to stakeholder interviews, the project addressed constraints on access to basic education, such as the poor condition of basic infrastructure and school facilities that largely prevented children from going to school or held down their national exam scores.

26 For primary school, the national examination consists of 3 subjects: mathematics, Indonesian language, and science. Each has a maximum score of 10, hence a student’s maximum score on the national examination will be 30. For junior secondary school, there are 4 subjects, namely mathematics, Indonesian language, science, and English (maximum of 40). Prior to 2003, the subjects examined at national level were 5 for primary school (moral education, Indonesian language, mathematics, social science, and science), and 6 for secondary school (moral education, Indonesian language, mathematics, social science, science, and English).

22 Indonesia: Decentralized Basic Education Project

6.78 7.14 7.28 7.23 6.54 7.32 6.70 6.84 6.82 7.07 6.70 7.23 After DBEP After DBEP

6.88 5.95 5.93 6.28 6.63 6.98 6.85 6.46 6.54 6.84 6.7 6.4 Average: children/year) Average: children/year) National Exam Score (District Before DBEP National Exam Score (District Before DBEP estimates are for estimates the primary level, are for estimates the primary level,

24 21 21 21 25 28 29 38 36 39 47 33 After DBEP After DBEP cts. cts. riods. Completion riods. Completion

overed under the examination was used, with 10 being overed under the examination was used, with 10 being

9 23 19 16 17 25 Transition (District 26 34 34 38 45 11 Transition (District Average: children/year) Before DBEP Average: children/year) Before DBEP rict in the given rict given pe in the rict given pe in the

25 30 31 26 25 30 43 52 51 56 55 55 After DBEP ect Implementation for All Students in Selected Districts for All ect Implementation After DBEP

ect Implementation for Poor Students in Selected Districts Districts Selected for Poor Students in Implementation ect

y school in the dist y school in the dist selected project schools in the six distri selected project schools in the six distri

30 25 23 23 25 17

Completion (District (District Completion 38 42 43 50 63 26 Average: children/year) Completion (District (District Completion Before DBEP Average: children/year) Before DBEP

26 26 28 28 36 29

32 46 45 45 49 44 e national exam score, the average score all of the subjects c subjects of the all score average the score, exam e national c subjects of the all score average the score, exam e national After DBEP After DBEP

e number of children enrolled a primar in enrolled of children e number a primar in enrolled of children e number 26 23 24 23 35 26 survey done by the team for 56 randomly survey done by the team for 56 randomly children/year/school) 30 39 50 45 49 43 children/year/school) Enrollment (District Average: Before DBEP Before DBEP Enrollment (District Average:

k cEducation Project. cEducation Project. Selected Districts Bangli Buleleng West Lombok East Lombo Mataram Timor Tengah Utara Selected Districts Bangli Buleleng West Lombok East Lombok Mataram Timor Tengah Utara Table 7: Changes in Education Outcomes before and after Proj before and Outcomes in Education Changes Table 7: Table 8: Changes in Education Outcomes before and after Proj before and Outcomes in Education Changes Table 8:

the maximum score. See footnote 26 for the description of Indonesia’s national exam. Source: Study team’s calculations based on a the maximum score. See footnote 26 for the description of Indonesia’s national exam. Source: Study team’s calculations based on a while transition estimates are for junior secondary level. For th while transition estimates are for junior secondary level. For th DBEP = Decentralized Basi EnrollmentNote: refer estimates to the averag Province Bali Bali Nusa Tenggara Barat Nusa Tenggara Barat Nusa Tenggara Barat Nusa Tenggara Timur DBEP = Decentralized Basi EnrollmentNote: refer estimates to the averag Province Province Bali Bali Nusa Tenggara Barat Nusa Tenggara Barat Nusa Tenggara Barat Nusa Tenggara Timur Performance Assessment 23

64. A paired t-test analysis was made to determine whether the changes in the education outcomes of the surveyed districts were statistically significant. The statistical analysis was used to determine whether the mean education outcomes improved after the project was implemented. It was carried out using data from 56 project schools selected using both probabilistic and non-probabilistic approaches to ensure that each school in each province had an equal opportunity to be selected as part of the sample. The impact of the project on education outcomes was examined in the selected schools, doing so for all students and for poor students as a group.27 Specifically, there is sufficient 65. The findings suggest that the project had an impact on education outcomes for evidence that all students in those districts examined (Table 9). Specifically, there is sufficient evidence that average education outcomes in the selected districts were significantly average higher after the project was implemented than before it was carried out.28 Increased education enrollment indicates that students had greater access to basic education, while outcomes in the increased transition and completion indicate greater opportunities for these students to finish basic education. Improved performance can also be observed in a rise in national selected districts exam scores. Among school participation outcomes, the highest average difference was were significantly observed on completion, followed by transition. higher after the project was Table 9: Paired t-test for All Students at District Level Average Average implemented Before After Standard t- Probability than before it Variable DBEP DBEP Difference Error statistic (T

Notes: H0: outcomes before DBEP = 0; Ha: outcomes after DBEP <0; degrees of freedom = 55. a Estimates are statistically highly significant at 1% significance level. Source: Study team’s calculations based on a survey done by the team for 56 randomly selected project schools in six districts: Bangli, Buleleng, East Lombok, Mataram, Timor Tengah Utara, and West Lombok.

66. The findings of the paired t-test analysis also show that the project helped increase access of the poor to basic education in the selected districts (Table 10). The increase in transition indicates greater opportunities for these poor students to complete basic education. The project also helped improve the performance of poor students in the national exam. Improvements in transition and completion were most notable among the enhanced education outcomes for poor students.

Table 10: Paired t-test for Poor Students at District Level Average Average Before After Standard t- Probability Variable DBEP DBEP Difference Error statistic (T

Note: H0: outcomes before DBEP = 0; Ha: outcomes after DBEP< 0; degrees of freedom = 55. a Estimates are highly statistically significant at 1% significance level. Source: Study team’s calculations based on a survey done by the team for 56 randomly selected project schools in six districts: Bangli, Buleleng, East Lombok, Mataram, Timor Tengah Utara, and West Lombok.

27 During the mission, the study team gathered data on education outcomes in the 56 selected schools for 2000–2013. 28 The statistical analysis was conducted using STATA version 13.1.

24 Indonesia: Decentralized Basic Education Project

67. School development. The project developed the schools’ capacity in determining priority education challenges and identifying corresponding strategies to address them. It provided school officials with training on how to formulate school development plans, involve communities in preparation of the plans, and monitor progress of the schools’ development. The interviewed principals noted that some ADB consultants were easily accessible for consultation and queries even beyond working hours. Given these contributions, the project effectively paved the way for implementation of school-based management in Bali, NTB, and NTT. Just 2 years into project implementation, such management was made official national policy in 2003.

68. The project has also effectively changed the mind-set of both school- and district-level officials in implementing school-based management. The project’s condition requiring schools to develop school development plans before they could access block grants bolstered the capacity and decision-making skills at school level. School development plans have become the most important consideration when selecting priority school programs, according to the interviews. School development plans examined by the evaluation team were deemed comprehensive. Their content covered the identified education problems, corresponding strategies to address these problems, work schedule, budget, and monitoring schedule.

69. The flexibility in the use of the block grants helped enhance the capacity of schools in basic education management. Such flexibility allowed project schools to determine priority needs and design innovative activities to address them, as interviews with stakeholders revealed. In contrast, such other transfers from the central government to schools as School Operational Assistance Program (BOS) grants employ more rigid regulations regarding those activities they can be used to fund.

While the 70. While the project’s block grants were mostly used for rehabilitation of school project’s block facilities, some schools adopted innovative approaches for school improvement. Anecdotal evidence gathered by the evaluation team reveals that a school in Anjani, grants were East Lombok used the block grant to address malnutrition among its students, 30.0% mostly used for of whom were found to be malnourished following medical checkups. The school rehabilitation of therefore used the block grant in conducting a lunch-feeding program to address this problem. Some schools in Bangli, Bali used their block grants to send teachers to school facilities, remote communities. Since these communities had no schools and children had to walk some schools several miles to reach the nearest school, these schools used the grants to dispatch adopted teachers who went to and taught children in these remote areas. According to the innovative World Bank’s school-based management national survey in Indonesia from 2010, budget allocation is one of those areas where school principals perceived they have approaches for high autonomy in decision making. On average, 90.0% of principals said they had school decision-making authority in allocating the school budget, recruiting teachers, setting improvement the school vision and curriculum, and selecting textbooks and teaching materials.29

71. The project was also effective in inducing communities to participate actively in school development through the formation of school committees. School committees usually are comprised of community leaders, local entrepreneurs, teachers, parents, and heads of villages or districts. According to the World Bank survey, parents account for roughly 75.0% of school committee members, followed by council representatives at 20.0% and teachers at 4.0%. The representation of the stakeholders is even more

29 The World Bank school-based management national survey used data from a nationally representative survey of about 400 public primary schools in Indonesia. The World Bank commissioned the RAND Corporation to conduct the survey. See Karam, K., Marshall, J., and Vernez, G. 2012. Implementation of School-based Management in Indonesia. RAND Corporation.

Performance Assessment 25

varied, as parents also serve in such other functions as religious leaders or representatives of nongovernment organizations (footnote 29).

72. School committees were actively engaged in the formulation and subsequent revisions, if necessary, of school development plans. Although schools were not required to report on achievements against their school development plans, according to interviews with stakeholders, school committees, along with principals, local government officials, and ADB staff and consultants, contributed to the formulation and subsequent revisions, if necessary, of the plans.

73. Local ownership of the project as fostered by community participation in school development also increased the communities’ commitment to contribute to the project voluntarily. Through the school committees, community members contributed labor and other in-kind assistance to support school projects. Stakeholders also deemed community involvement as having a significant effect on education quality. With the introduction of BOS, however, schools have been expected to eliminate tuition fees or other compulsory payments from parents.30 To some extent, this might have inhibited some parents who would like to assist the school financially. Nevertheless, the central government still permits schools to receive voluntary contributions from community members for school development.

74. District basic education development. Interviewed stakeholders reported that district education offices’ primary role under the project dealt with the provision of training, particularly in teacher development and evaluation of school development plans. The district education offices were much involved in evaluating school development plans and accordingly provided feedback to schools. Monitoring duties by supervisors and district staff also included providing feedback on principal and teacher performance, checking on the conditions of school facilities, monitoring or observing classrooms and instruction, assessing teacher training needs, and reviewing and approving the school budget (footnote 29).

75. The District Education Development Fund had been used to finance activities related to the district education officers’ capacity building such as trainings or seminars. The fund had also been used to assist teachers and school principals in improving the quality of education. Its support for training teachers, school principals, and supervisors included contextual teaching and learning, in-service training, competency-based training, mastery of teaching and learning materials, evaluation, and classroom management, among others. The training was provided to both project and non-project schools in a district, expanding capacity in both. Interviews with stakeholders revealed that district education boards are still in place as they are mandated under Ministerial Regulation No. 044/2002. As noted in the PCR, however, a new regulation on the management of education (PP 17/2010) made funding of the district education boards by districts discretionary and reduced the number of board members.

76. The project’s impact on capacitating district education offices to produce 5- year district education development plans and 5-year annual action plans was less notable than that of other outputs. As noted in the PCR, the district education development plans were supposed to be revised annually, but this was not done in all districts. Instead, districts base their annual implementation on the government’s annual work plan, in line with government procedures. Moreover, only a few districts

30 World Bank. 2010. Supporting the BOS Program. Education Update. Issue No. 01. Jakarta

26 Indonesia: Decentralized Basic Education Project

integrate school development plans into their annual work plans. In 2012, just 12.0% of districts consolidated school development plans for use in their district education planning process.31 Despite this, those district officials interviewed said the training they received in formulating development plans proved useful, as district education offices are required by law since 2004 to create development plans.

77. Challenges at the district level on decentralizing basic education have been observed. One of these—high staff turnover at the district level—has hampered the project’s capacity building efforts. Local district leaders, including mayors or heads of district education offices, have full authority to move officers in various units or departments, including principals and district education staff. In some districts, there was insufficient knowledge transfer. However, in some districts in which the project was supported by a team of district education staff, the remaining staff members were able to keep up with progress.

78. Second, some districts have too many teachers and others too few since teacher deployment and mobilization have also come under the authority of every district government. Moreover, education appointments are perceived to be politicized in that they are made by district heads. According to interviewees, some staff appointments are based on political ties.

79. Third, district governments have yet to maximize their education budgets and a huge portion is spent on personnel. District governments allocate some 30%–40% of their budgets to education. Because around 85% of their education budgets is spent on personnel costs, limited funds are left for education development.32

80. Fourth, there are flaws in information systems that project districts use to track and publish statistics. Data-keeping has been a problem, as documents are discarded or deleted after 5 years due to storage issues. Stakeholders reported that no soft copies of data are available.

31 The figure is based on a World Bank survey of 50 Indonesian districts on the quality of local education governance. See World Bank. 2013. Local Governance and Education Performance: A Survey of the Quality of Local Education Governance in 50 Indonesian Districts. Jakarta. 32 Ministry of Education and Culture. 2013. Overview of the Education Sector in Indonesia 2012: Achievements and Challenges. Jakarta.

Performance Assessment 27

D. Efficiency

81. The project is rated efficient. Despite the limited amount of grants, the project Despite the achieved notable results in enhancing local capacity for education governance and limited amount improving school infrastructure. Especially considering that the grants provided under the project were spread over a 3-year period, some stakeholders noted these to be of grants, the quite small. Interviews with Ministry of National Education, school, and district officials project achieved revealed that some Rp20 million to Rp25 million is usually required to rehabilitate one notable results in classroom in an elementary school (measuring 8 x 7 meters) and Rp30 million to Rp35 million for one classroom in a junior secondary school (measuring 9 x 7 meters). Since a enhancing local block grant amounted to around Rp90 million rupiah, three to four classrooms could capacity for be rehabilitated. The PCR nevertheless pointed out that junior secondary school education facilities improved, in particular within NTT, with 80% of junior secondary schools having administrative offices and 60% having libraries and science laboratories. governance and Moreover, community support helped lower the costs of rehabilitating facilities, as improving school some parents or school committee members offered in-kind assistance such as free infrastructure labor. In addition to contributing to school infrastructure, the project made huge contributions in improving schools’ and districts’ capacity for local education delivery.

82. Indeed, it is impossible for the project to address all the education-related infrastructure requirements in Bali, NTB, and NTT. Continued rehabilitation of school facilities is imperative, and particularly for those in impoverished areas such as a school in the city of Kefamenanu, district of Timor Tenga Utara, in NTT, which the evaluation team visited and found to have some classrooms that were badly damaged (Figure 6). As stated, the government has recently scaled up its efforts to rehabilitate schools in the country.

Figure 6: Condition of Classrooms in Timor Tengah Utara, Nusa Tenggara Timur, 2014

Source: Study team, taken during the evaluation mission.

28 Indonesia: Decentralized Basic Education Project

83. The PCR pointed out that progress under the loan was initially slow, but the pace did pick up throughout execution. On the other hand, the grant project required two extensions to achieve project objectives. The grant’s shorter implementation period of only 2.5 years proved inadequate for severely poor and often inaccessible districts. As noted in the PCR, funds were generally disbursed on time and the full budget allocation, as well as most of the additional funds made available by the rising exchange rate, was used each year. There had been an 11.0% increase from the appraisal amount of the project.33 The increase was actually 29.6% of the base cost of the project, the largest share of which was attributed to the payment for international consultants that was 127.0% of the appraisal amount. The person-months of the consultants, on the other hand, increased by only 12.8% (from 78 person-months at appraisal to an actual 88 person-months).

84. The external auditor found a few problems with procurement. Most of those problems identified related to overpayment for workshops and training. Under the loan component, 57 audit cases—relating mostly to fund misuse due to overpayment, unclaimed penalties, and undeposited bank interest—were identified. Under the grant component, 18 cases were identified. All 75 identified cases (involving less than 1.5% of schools) were resolved.

E. Sustainability

85. The project is rated likely sustainable. Its contributions to improving capacity for local education governance are reinforced by continued transfer of knowledge between school and district officials who received training and their counterparts who were not beneficiaries of the project. According to interviews, teachers from project schools often serve as trainers in a forum called Teacher Networks or MGMP The project’s (Musyawarah Guru Mata Pelajaran). MGMP is a non-structural organization established in accordance with 1991 guidelines issued by the Directorate General of Primary and spillover effects Secondary Education, and it seeks to promote teacher professionalism. are particularly notable in school 86. Moreover, the project’s spillover effects are particularly notable in school development plan development plan formulation, which has been assimilated even by non-project schools and has become standard practice among schools in the country. Continued formulation, community involvement in school development through the school committees also which has been constitutes an important aspect of the project outputs’ sustainability. School assimilated even committees remain part of the formulation process of school activity and budget plans, which are similar to school development plans. School committees also take part in by non-project decision making on the use of BOS grants, which, according to interviews with schools and has stakeholders, must be based on agreement between the BOS management team in the become standard school, teacher’s council, and school committee.

practice among 87. In contrast to the PCR’s finding that some stakeholders had deemed the project schools in the activities unsustainable without project support, certain activities funded by the block country grants are still being implemented. In particular, support for poor students through scholarships, transport allowances, remedial programs, catch-up programs, and tutoring for students missing school are still being provided by some schools.34 The central and district governments provide the bulk of funding to support continued

33 In the PCR (Appendix 4), the subtotal of the actual foreign base cost was reported to be $23.399 million. When the evaluation team recalculated the reported table, however, the actual foreign base cost was supposed to have been $23.401 million. This $2,000 difference was equal to Rp 18,840,000. 34 The study team surveyed 56 selected schools. Some 64% indicated that activities for poor students are still being implemented even after project completion.

Performance Assessment 29

implementation of activities targeting poor students (Figure 7). District governments, donor agencies, or local communities also provide funding to such initiatives.

Figure 7: Project Schools’ Sources of Funding for Continued Assistance to Poor Students (%) 4% Ministry of Education 13% 35%

District government

17% Donor agencies

Local communities and parents

Other sources 31%

Source: Study team’s calculations based on a survey of 56 schools.

88. The maintenance and rehabilitation of school facilities does not appear to be a priority of schools. Primary schools use about 8.0% of BOS grants for the maintenance of school facilities compared to 19.0% for examination activities and 13.0% for teacher honoraria. BOS grants can be utilized only for facility maintenance and small repairs such as painting or repairing a leaky roof.35 As already mentioned, however, the central government recently scaled up its efforts to rehabilitate school infrastructure.

35 World Bank. 2011. Independent Monitoring Report: BOS Program Implementation 2008–2009. Jakarta (World Bank Office in Indonesia).

CHAPTER 4 Other Assessment

A. Impact

89. The project’s impact is rated significant. As discussed in Chapter 3, improved education outcomes have been observed in the provinces of Bali, NTB, and NTT. Improvements in enrollment, transition, completion, and exam scores indicate students’ enhanced access to and greater opportunities to complete basic education, as well as their better learning performance.

90. Moreover, the project made pioneering contributions in improving local education governance capacity. At the start of project implementation in 2001, very few school officials were knowledgeable about decentralization of basic education, according to interviews with stakeholders. At the time, the project was the only available platform where schools were invited to discuss and suggest their needs to the government. Moreover, the project’s contributions to enhancing local capacity for decentralized basic education have been extended to non-project schools and districts.

91. Most importantly, the project has influenced government policies related to decentralizing basic education. The project piloted the concept of school-based management in Bali, NTB, and NTT by improving schools’ capacity to formulate school development plans and encouraging community involvement in school development prior to enactment of the National Education System Law in 2003. That law stipulates the promotion of school-based management. Similarly, BOS partly adopted the project’s block grant system. Also, Regulation Number 101 for 2013 of the Ministry of Education and Culture (Chapter 3, section E, Number 3) states schools must provide school activity and budget plans, which are essentially similar to school development plans, for liquidation of the BOS. While those plans’ formats may different from that of school development plans, the project has helped develop school officials’ capacity in preparing such plans.

B. ADB and Executing Agency Performance

92. The Asian Development Bank’s (ADB) performance is rated less than satisfactory. High turnover of ADB’s project officers—seven in total—hampered effective and continued project supervision and management. Such frequent changes in ADB project officers also caused difficulties in communication between ADB and MONE. As the project was managed directly from ADB headquarters in Manila, the Indonesia Resident Mission’s role largely was to facilitate communication between MONE and ADB staff at headquarters. The resident mission’s responsibilities and contributions could have been maximized by tapping resident mission staff for project management and monitoring tasks.

Other Assessment 31

93. The performance of MONE, the executing agency, is rated satisfactory. MONE Since MORA is initially encountered difficulties in managing consultants and gathering data, but one of five state eventually it gained sufficient capacity as project implementation progressed. MONE reported facing challenges, particularly in coordinating with MORA, the project’s co- ministries not executing agency. Since MORA is one of five state ministries not decentralized under decentralized the law, structural misalignment limits the capacity of MONE and MORA to coordinate under the law, their functions relating to decentralized basic education. MORA’s provincial offices are structural largely responsible for madrasah, while MONE’s district-level officials are responsible to manage schools under their jurisdictions. MORA’s district offices can play only an misalignment operational role in implementing education policies developed by its central and limits the provincial offices. capacity of

94. Coordination between MONE and MORA in involving provincial MORA officials MONE and in local educational development planning exercises is contingent on the attitude and MORA to relationships of individuals in the respective offices.36 Contact between madrasah and coordinate their MONE’s district education offices is also generally limited. Moreover, while MONE is focused solely on the delivery of education services, MORA performs multiple religion- functions relating related functions and only oversees Islamic education through a single directorate to decentralized general. As such, MORA has relatively less resources, expertise, and institutional basic education structure to ensure the same level of government service provision for Islamic schools that general schools receive from MONE (footnote 36).

36 United States Agency for International Development. 2007. Analysis of the Current Situation of Islamic Formal Junior Secondary Education in Indonesia. Washington, DC.

CHAPTER 5 Issues, Lessons, and Follow-up Actions

A. Issues

95. The project sought to promote decentralization of basic education in Indonesia primarily by improving local capacity at the school and district levels for basic education management. This evaluation found that the implementation of decentralized basic education in Indonesia continues to face challenges.

96. First, coordinating the functions and authority over basic education services between local and central governments is an area of concern. District governments now wield the greatest authority in basic education, as interviews with stakeholders revealed. How to coordinate central and district governments’ authority and responsibilities in managing basic education services is a concern. This problem is clearly highlighted in decentralization’s impact on teacher policy and student mobility. Although decentralization has made it easier for teachers to be redeployed within a district, transferring teachers between districts has reportedly become more difficult after decentralization. 37 Even if one district has many teachers and a neighboring district lacks teachers, teachers cannot be moved because they are under district authority. Similarly, student mobility is hampered. Students cannot move from one district to another, because each district usually has a quota for students coming from other districts.

97. Second, the election of district officials in Indonesia has somewhat affected the decentralization of basic education services. Although electing district officials supports the participation of local communities in democratic processes and ensures they have a voice in decision making, interviews with stakeholders highlighted that some district officials try to drum up support at election time through promises of free education. Given the notion of free education, some community members are reluctant to contribute toward school development. Furthermore, interviews with stakeholders revealed that some district officials appoint staff based on political ties, including of those involved in delivering basic education.

98. Third, interviews with stakeholders revealed that schools appear to be more compliant with the district governments than the central government. Although district authorities are now responsible for managing basic education services, the central government maintains certain responsibilities in decentralized basic education management, particularly in monitoring the delivery and quality of education services.

37 J. Tobias et al. 2014. Toward Improving Education Quality: Indonesia’s Promising Path. Working Papers in Economics and Development Studies No. 201412. Bandung: Department of Economics, Padjadjaran University.

Issues, Lessons, and Follow-up Actions 33

Prior to decentralization, the central government sent local governments questionnaires, which the latter would send to schools. Today, it is often much more difficult for the central government to request data from schools, such as on school profile. The central government through MONE has established the online school database Dapodik (Data Pokok Pendidikan), but it is still not easy to obtain data from schools.

B. Lessons

99. The project provides some important lessons for promoting decentralized basic education management in Indonesia. The PCR identified key lessons, including the importance of (i) flexible management by ADB and the government, coupled with good communication from the project team that enabled the project to maintain its rationale and objectives despite changing legal and administrative environments; (ii) systematic record keeping, given the poor documentation and recording of data on project activities, outputs, and outcomes at the central and district levels; (iii) improving data quality and the use of data in planning and management in districts; (iv) clearly explaining project objectives to stakeholders, as the project was primarily regarded as a funding mechanism and the goal of district capacity development was deemphasized; and (v) integrating project funds and activities into government processes, which could have further improved district capacity building.

100. In addition to these, this evaluation has identified the following lessons. One, decentralization of basic education can be bolstered through strong collaboration between MORA and MONE. To address the structural misalignment between MONE and MORA, the role of provincial governments in decentralized basic education should be strengthened, particularly in monitoring and evaluation. The Indonesian government appears to be moving toward intensifying the delegation of authority to provincial governments in basic education. From 2014, provincial governments have full authority in preparing the materials for the primary school final examination and oversee the district government’s implementation of this examination, as stipulated in Ministerial Regulation No. 102/2013 and another regulation of the head of the MONE Research and Development Center.

101. Two, good technological infrastructure at the school and district levels can help address concerns about data management. MONE has implemented the Dapodik online school database system. Schools are required regularly to submit data to this system. However, many schools do not have access to the internet to accomplish this task. School staff must also be capacitated to utilize this online database effectively.

102. Three, effective capacity building efforts are contingent on consistent supervision and support. High staff turnover among ADB’s project officers and district governments tended to hamper effective communication and rapport-building among stakeholders. Consequently, this disrupted efforts to promote decentralized basic education management. Means of effective transfer and sharing of knowledge through, for instance, continued training are imperative. For ADB’s part, increasing the role of the Indonesia Resident Mission to include project management and monitoring responsibilities could have helped address project implementation issues. As discussed earlier, the project was managed directly from ADB’s headquarters in Manila. Moreover, ADB could have exerted greater efforts in disseminating knowledge products to further promote the sharing of lessons learned and best practices in decentralized basic management to other Indonesian provinces or countries similarly endeavoring to decentralize basic education delivery and management.

34 Indonesia: Decentralized Basic Education Project

C. Follow-up Actions

103. The evaluation proposes the three follow-up actions for ADB to pursue given its continuing support for education-oriented projects in Indonesia. Effective decentralization of basic education largely hinges on collaboration between MONE and MORA, which needs to be improved. MORA remains centralized, while MONE has already been decentralized. ADB should support measures to improve the coordination between the two ministries. When implementing education interventions, ADB should help strengthen the role of provincial governments in decentralized basic education, particularly in monitoring and evaluation. Sharing more information regarding the best practices of national education management between the two ministries can help improve their coordination.

104. ADB’s efforts to further improve access to basic education in Indonesia should be complemented with continued investments in enhancing transport and technological infrastructure and school facilities. Improving transport infrastructure such as roads, and particularly in remote areas, would allow more students to continue their studies to junior secondary schools, which are normally located in the subdistrict capital city. The technological infrastructure of schools should also be enhanced to improve their access to and utilization of the online Dapodik system so that school data are regularly updated. Moreover, support is necessary for continued school rehabilitation work, including in such remote areas as those in Bali, NTB, and NTT. ADB should align its education interventions with programs by the government or other development partners in these areas.

105. If ADB’s Human and Social Development Division of the Southeast Asia Department will examine whether future ADB support is still required toward decentralization of basic education in Indonesia, it should consider the challenges which the decentralized system continues to face. These include the impact of decentralization on teacher policy and perceived politicization of appointments made by district governments in the education area.

Appendixes

APPENDIX 1: SUMMARY DESIGN AND MONITORING FRAMEWORK

Design Summary Performance Indicators and Targets PCR/PPER Update Sustainability 1. Impact Improved indicators for gross Gross Enrollment Rate, by Province and School Type While targets for gross To support the enrollment rates of 120.7% for Province 2010 2011 2012 2013 enrollment rate may government’s goal of primary education and 78.9% for Primary not have been achieving universal basic junior secondary education Bali 95.53 90.39 91.06 94.28 achieved, the impact education that serves the Nusa Tenggara Barat 95.16 92.69 94.56 96.63 is sustainable, needs of poor and Nusa Tenggara Timur 93.03 92.13 92.28 93.60 especially considering marginalized people and Indonesia 94.76 91.03 92.49 95.53 the long-term effects is responsive to local Junior Secondary of these needs Bali 67.83 69.16 75.07 80.69 improvements on Nusa Tenggara Barat 71.73 76.70 77.81 80.18 education outcomes Nusa Tenggara Timur 51.03 56.74 55.89 59.24 in terms of, for Indonesia 67.73 68.12 70.84 73.72 instance, Source: Indonesian Statistics Office. employability.

Improved human development Human Development Indicators, by Province indicators in targeted provinces and Province 2010 2011 2012 districts Bali 72.28 72.84 73.49 Nusa Tenggara Barat 65.20 66.23 66.89 Nusa Tenggara Timur 67.26 67.75 68.28 Indonesia 72.27 72.77 73.29 Source: Indonesian Statistics Office. 2. Outcome In each project district: Net Enrollment Rate, by Province and School Type Improvements in Poor children’s increased net enrollment rates; Province 2010 2011 2012 2013 education outcomes participation, transition, improved primary completion rates Primary among the poor are completion, and and junior secondary graduation Bali 95.53 90.39 91.06 94.28 expected to have performance in basic rates; Nusa Tenggara Barat 95.16 92.69 93.56 96.63 long-term impacts, for education in Bali, Nusa improved primary to junior Nusa Tenggara Timur 93.03 92.13 92.28 93.60 instance on Tengarra Barat (NTB), and secondary transition rates; and Indonesia 94.76 91.03 92.49 95.53 employability and Nusa Tengarra Timor improved 6th- and 9th-grade inequality. (NTT) improved national examination scores Junior Secondary Bali 67.83 69.16 75.07 80.69 Nusa Tenggara Barat 71.73 76.70 77.81 80.18 Nusa Tenggara Timur 51.03 56.74 55.89 59.24 Indonesia 67.73 68.12 70.84 73.72 Source: Indonesian Statistics Office.

Completion Rates, by Province and School Type Province 2008 2009 2010 2011 Primary Bali 98.28 99.01 99.26 99.49

Design Summary Performance Indicators and Targets PCR/PPER Update Sustainability Nusa Tenggara Barat 99.16 99.91 99.20 98.89 Nusa Tenggara Timur 98.88 98.00 98.46 98.72 Indonesia 99.18 98.97 98.84 99.05 Junior Secondary Bali 93.12 99.07 99.27 98.90 Nusa Tenggara Barat 95.60 96.94 97.79 96.81 Nusa Tenggara Timur 95.37 95.98 98.00 97.73 Indonesia 95.66 97.01 97.50 96.96 Source: Study team’s calculations based on national socioeconomic survey data.

National Exam Scores, by Province Average Average Province (2000) (2012) Bali 6.48 8.06 Nusa Tenggara Barat 6.32 7.87 Nusa Tenggara Timur … 6.83 Indonesia 6.54 7.48 Source: Study team’s calculations based on national socioeconomic survey data. Among children from the poorest Net Enrollment Rate for Poorest 40% of Students, by Province and 40% of families: School Type increased net enrollment rates; Province 2000 2003 2006

improved primary completion rates Primary Framework Monitoring and Design Summary and junior secondary graduation Bali 92 92 93 rates; Nusa Tenggara Barat 89 92 94 improved primary to junior Nusa Tenggara Timur 89 88 92 secondary transition rates; and Indonesia 92 93 94 improved 6th- and 9th-grade Junior Secondary national examination scores Bali 58 54 51 Nusa Tenggara Barat 53 51 66 Nusa Tenggara Timur 27 34 41 Indonesia 51 55 59 Source: ADB. 2012. Completion Report: Indonesia: Decentralized Basic Education Project. Manila.

Examination Scores and Transition Rates for Poor Students in NTT, by School Year and School type 05/06 06/07 07/08 08/09 Examination Score

Primary 6.0 6.0 6.0 5.9

Junior Secondary 5.7 5.7 5.0 5.7 37

38 Appendix 1

a national

Sustainability SBM is now a national policy supported by the National Education Law (20/2003). Districts are mainly focused on monitoring and evaluating schools’ improvement plans. There is weak capacity at the district level for gathering and updating data. SBM is now policy supported by the National Education Law (20/2003). The formation of school also committees is stipulated under this law. Moreover, the practice of SDP been formulation has assimilated even by non-project schools.

97 94

96 87

by only 35%–40% of the school

96 91 funding for education since 2001. funding for education since provement in districts were provement in districts were ty partnerships. PCR/PPER Update PCR/PPER 94 88

Transition Rate Basic Decentralized Indonesia: Report: Completion

Manila.

Primary Junior Secondary

Source: ADB. 2012. 2012. ADB. Source: Education Project. SBM was implemented in 5,240 project schools. plans (SDPs) were developed by all project School development schools. distributed to 3,522 schools grants were (67%) Small rehabilitation grants to all schools. improvement quality and to schools, leading in There was high community involvement stronger school–communi National Education Law (2003) requires national and local 20% of their allocate at least budgets, therebygovernments to considerably increasing district All districts developed district education development plans under the project, although use of data was limited. Some districts now produce annual plans in line with district budget processes rather than district development plans. education Plans for equity and quality im implemented. Participatory baseline studies and needs assessments were carried out before SDPs were developed. held regular school committees and established All schools meetings. 95% of stakeholders interviewed had attendedIn all provinces, meetings. form of labor, while mainly in the were Community contributions monetary contributions were made 25% in NTT. and Bali and NTB in community

in project

madrasah Performance Indicators and Targets SBM implemented in about 3,000 SBM implemented primary and junior secondary schools and districts meet minimum Project schools conditions for learning Effective community–school partnerships Increased funding for basic education over 2001 levels Data-based and goal-oriented 5-year development plans for basic education developed Plans for equity and quality improvement in project districts implemented Districts meet goals for achieving equitable access to basic education analysis baseline contain All SDPs and needs assessment, realistic targets and strategies for school improvement. in schools SBM implemented through: active school functioning and committees; integrated school budgets; school increased decision making at level; staff performance annual school Design Summary School-based management (SBM) implemented in project schools in Bali, NTB, and Districts NTT manage decentralized basic education effectively 3. Outputs 3.1. School-based management implemented in schools analysis School baseline and needs assessment carried out and school development plans (SDPs) developed SBM implemented involvement Community

Design Summary Performance Indicators and Targets PCR/PPER Update Sustainability in schools strengthened review; and school-initiated innovations for improving effectiveness 3.2 School quality Project schools meet SDP targets for See education outcomes above. Given knowledge improved with School (i) education outcomes including sharing and transfer Development Fund enrollment, completion, transition, The attendance of teachers and students increased steadily every among school assistance exam scores, and teacher discipline; year in the NTT districts, from an average of 83%–88% to 90%– officials, training and and (ii) classroom improvement, 95% (not measured in loan districts). capacity building Improved school including classroom number, efforts are expected to effectiveness teaching aids, and other facilities For NTB and Bali: 90% of stakeholders reported that the conditions be more sustainable. and materials of the schools, classrooms, furniture, and toilets improved. There Improvements in Increased number of poor was less improvement in libraries and laboratories (only a school facilities such children completing basic requirement for junior secondary schools). At the end of the as classrooms are less education project, 71% of classrooms were in good condition and furniture evident and was available in 60%–70% of schools. Availability of teaching and sustainable, however,

More schools meet learning materials improved moderately, but the needs remain inasmuch as the block minimum conditions of large. grants provided may learning be too small to For NTT: only 60% of primary and 47% of junior secondary schools accommodate classrooms were in good condition at the end of the project, but rehabilitation and almost all schools had adequate furniture. The proportion of maintenance of these primary schools with libraries increased from 23% to 40%, and facilities.

junior secondary schools with laboratories from 50% to 60%. Only Framework Monitoring and Design Summary 39% of toilets were in good condition by the end of the project. Schools received teaching and learning materials (generally textbooks) and science kits, but the need for other kinds of materials remains high. 3.3. District capacity to Education boards established with Education boards in all districts received training during the High turnover of staff plan and manage basic community representation, project, including training in budget analysis. The boards work in at the district level education increased according to decree partnership with the district education offices to improve district poses challenges on education while reporting directly to the head of the district proper transfer of District education Project districts revise 5-year plans government. The major work of the boards revolves around knowledge and leadership strengthened regularly on the basis of up-to-date monitoring and evaluation of education programs and education continued capacity information financing (School Operational Assistance Program and national building in basic 5-year district basic special allocation fund); capacity building and supervision of school education education development Timely and accurate district annual committees; and mediation among parents, schools, and district management. plan completed statistics produced education offices.

Information systems in

place All districts prepared education development plans during the 39

40 Appendix 1

Sustainability High staff turnover at the district level poses proper challenges for transfer of knowledge and continued in building capacity basic education management. Proper budgetingand monitoring is part of training provided under SDP and the project’s formulation. Manila. Manila.

ed in districts, albeit with a ed in districts, albeit successful. Only a small number successful. Only a small d that project-trained are staff ht after to fill positions. ht after to fill positions. PCR/PPER Update PCR/PPER

Completion Report: Indonesia: Decentralized Basic Education Project. was made was depending on data availability. project, but not all districts have continued the practice after project, but not all districts have project. produc Annual education statistics are to years). The failure of schools of several time lag (sometimes in this is lacking capacity addition, In is blamed. information submit offices. area in many district education in all systematic approaches developed generally have All districts these areas, with the weakest being teacher development. of the Decentralized Basic Education Project’s The effectiveness staff was of district the capacity in building programs acknowledged. All districts reporte highly regarded and keenly soug were systems compliance Financial of cases funds misuse were identified and all resolved.

and

nesian Statistics Office data; ADB. 2012. cited reportin the project completion Performance Indicators and Targets In district education offices, systems In district education offices, systems in place for planning, managing, and monitoring financing, staffing, of: teacher development; school supervision; principal recruitment, training, and performance review; facilities expansion and improvement; and district financing of education Fund The School Development District Education Development Fund are managed in compliance and Asian with government Development Bank requirements.

Design Summary 3.4. District education improved with District Education Development Fund assistance Management of basic education provision strengthened 3.5. Effective program financial compliance and monitoring Independent and internal monitoring of financial management and program procedures Note: Presentation of updates to indicators Sources: Study team’s calculations using Indo

APPENDIX 2: LIST OF INTERVIEWEES

A.1 Ministry of Education

1. Mr. Hamid Muhammad, Director General, Directorate General Office for Basic Education (formerly Manager of the Decentralized Basic Education Project [DBEP]) 2. Mr. Didik Suhardi, Director, Junior Secondary Education (formerly a member of the DBEP Central Technical Committee) 3. Mr. Dedi Karyana, Head, Section for Education Infrastructure (formerly a member of the DBEP Central Technical Committee) 4. Mr. Budi Susetyo, Asian Development Bank consultant

A.2 District Education Offices

Bali Province

Buleleng District

1. Mr. Ketut Arjana, Head, Basic Education Affairs, District of Buleleng (formerly DBEP District Manager) 2. Ms. Ni Ketut Santrini, Section Head, Basic Education Affairs District of Buleleng (formerly DBEP Treasurer, District of Buleleng)

Bangli District

1. Mr. I Nyoman Sumantra, Head of the District Education Office, District of Bangli 2. Mr. Wayan Sudiarsa, Secretary of the District Education Office, District of Bangli (formerly DBEP Manager, Bangli) 3. Mr. I Made Artana Yasa, Head of Finance, District Education Office, District of Bangli (formerly DBEP Manager, Bangli) 4. Mr. Ketut Punia, Staff, District Education Office, District of Bangli (formerly DBEP officer)

West Nusa Tenggara Province

Lombok Barat (West Lombok) District

1. Mr. Iswarta, Section Head for Teacher and Administration Staff, District of West Lombok (formerly DBEP District Manager) 2. Ms. Dini, District Education Officer, West Lombok

Lombok Timur (East Lombok) District

1. Mr. Supriyadi, Head, Basic Education Affairs, District of East Lombok (formerly DBEP District Manager) 2. Mr. Abdul Muin, Officer, District of East Lombok (formerly DBEP District officer)

Mataram City

1. Mr. Zaenal Arifin, Head of the Basic Education Affairs, District Education Office, Mataram City

42 Appendix 2

East Nusa Tenggara Province

Timor Tengah Utara District

1. Mr. Emanuel Anunu, District Crops Plantation Officer, former DBEP project manager 2. Mr. Thimateus Timo, District Education Officer, former DBEP officer

A.3 Project Schools1

Bali Province

Buleleng District

1. Mr. Pande Ketut Remawa (Principal, Junior Secondary School #3 KBT) 2. Mr. Made Imawan (Principal, Junior Secondary School #1 Tejakula) 3. Mr. Gede Suastika (Principal, Junior Secondary School #45 Sukasada) 4. Mr. Gede Sumatra Jaya (Principal, Junior Secondary School #4 Sukasada) 5. Mr. Nyoman Purnayasa (Principal, Junior Secondary School #2 Singaraja) 6. Mr. I Made Sutarjana (Principal, Junior Secondary School #1 Seririt) 7. Mr. Nyoman Suyasa (Principal, Junior Secondary School #2 Seririt) 8. Mr. Ketut Suardika (Principal, Junior Secondary School #1 Sawan) 9. Mr. I Wayan Kantun Toni (Principal, Primary School #1 Astina) 10. Ms. Gusti N. Reniasih (Principal, Primary School #1 Br Tegal) 11. Ms. Ketut Sudarmini (Principal, Primary School #5 Br Jawa) 12. Ms. Ni Wayan Resmi (Teacher, Primary School #1 Br Bali) 13. Mr. Made Ketu Raja (Principal, Primary School #2 Br Tegal) 14. Mr. Nyoman Sudiarsa (Principal, Junior Secondary School #5 Singaraja)

Bangli District

1. Mr. I Putu Jati (Principal, Primary School #3 Peninjoan) 2. Mr. I Nengah Sudana (Principal, Primary School #2 Abuan) 3. Mr. I Nyoman Wardana (Principal, Primary School #6 Tiga) 4. Mr. I Nengah Suara (Teacher, Primary School #3 Tiga) 5. Mr. I Ketut Disna (Teacher, Junior Secondary School #2 Bangli) 6. Mr. I Nengah Sulatra (Principal, Junior Secondary School #2 Kintamani) 7. Mr. Wayan Bukahan (Principal, Primary School #2 Selat) 8. Ms. Dayu KM Suastini (Teacher, Primary School #2 Selat) 9. Mr. I Nengah Genah (Principal, Primary School #4 Taman Bali)

West Nusa Tenggara Province

Lombok Barat (West Lombok) District

1. Mr. Syahrudin Saat (Principal, Primary School #7 Lambuak) 2. Ms. Raodah (Principal, Junior Secondary School #4 Gerung) 3. Mr. Abdul Haris (Principal, Junior Secondary School #1 Lempau) 4. Mr. Akhmad S. (Principal, Primary School #1 Kediri)

1 The study team collected information from representatives of 56 project schools. Some principals and teachers were not able personally to participate in the interviews held at the district education offices but were given questionnaires, which they completed and returned to the study team.

List of Interviewees 43

5. Mr. Abdul Gani (Principal, Junior Secondary School #1 Gunung Sari) 6. Mr. Apudin (Principal, Primary School #4 Gerung Utara) 7. Mr. Saringin (Principal, Junior Secondary School #3 Lingsar) 8. Mr. M Rusman (Principal, Primary School #1 Gelogor) 9. Mr. Tajudin (Principal, Junior Secondary School #5 Lingsar) 10. Ms. Sabariah (Principal, Primary School #1 Labuapi)

Lombok Timur (East Lombok) District

1. Mr. Agus Gunarto (Vice Principal, Junior Secondary School #1 Suralaga) 2. Mr. Abdul Mantif (Principal, Primary School #1 Anjani) 3. Ms. Hj. Seripa Fatimah (Principal, Primary School #2 Sandubaya) 4. Mr. M. Tohran (Principal, Primary School #1 Dusun Lekong) 5. Mr. Bambang Subagyo (Principal, Junior Secondary School #1 Masbagik) 6. Mr. Nursalim, Section Head for Curriculum Affairs, District of East Lombok (formerly DBEP school principal, Primary School #3 Sehket)

East Nusa Tenggara Province

Timor Tengah Utara District

1. Ms. Matilda K Bano (Principal, Primary School Baki) 2. Mr. Petrus Obe (Principal, Junior Secondary School Napan) 3. Mr. Fransiskus Meko (Teacher, Primary School Nunpene) 4. Mr. Marselinus Kolise (Principal, Junior Secondary School Biboki Utara) 5. Mr. Benekdiktus Falo (Principal, Primary School Koko) 6. Mr. Dominikus Tasoeb (Principal, Christian Secondary School St. Ga) 7. Mr. Benekdiktus Seo (Principal, Christian Secondary School St. Paulus) 8. Mr. Alexander Pilis (Principal, Secondary School Oeluele) 9. Mr. Yoseph Sila (Principal, Secondary School Teflasi) 10. Mr. Raymundus Fallo (Principal, Junior Secondary School #2 Kefa) 11. Mr. Nikolas Neonnub (Principal, Junior Secondary School #2 Muotire) 12. Mr. Gustaf Maf (Vice Principal, Christian Secondary School Kefa)

APPENDIX 3: QUESTIONNAIRE FOR SCHOOLS

KUESIONER SEKOLAH

Informasi Responden Nama Responden: Nama Sekolah: Posisi di Sekolah: Alamat Sekolah: Nomor Telepon:

Sekolah anda terpilih sebagai salah satu penerima bantuan dari proyek ADB, Desentralisasi Pendidikan Dasar (DBEP) Pada kuesioner ini, kami ingin menanyakan hal-hal yang terkait dengan pelaksanaan proyek tersebut Pada khususnya, kami ingin mengetahui bagaimana sekolah ini mengambil manfaat dari proyek tersebut Bagaimana proyek ini membantu sekolah dalam mengimplementasikan rencana pengembangan sekolah Bagaimana proyek ini membantu aktifitas sekolah dan bagaimana dana dari proyek ini dialokasikan ke berbagai aktifitas tersebut

PERTANYAAN 1: Berapa total dana ADB yang diterima oleh sekolah antara tahun 2001-2010, apabila ada? 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010

Kami akan bertanya mengenai berbagai kegiatan di sekolah Anda yang dibayai oleh dana ADB antara periode 2001 dan 2010.

PERTANYAAN 2: Berapa besar bantuan dari ADB yang digunakan untuk berbagai aktivitas berikut? 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010

i. Bantuan sekolah kepada siswa miskin ii. Program alternatif untuk siswa yang tidak dapat mengikuti kelas reguler iii. Segala jenis perbaikan fasilitas sekolah iv. Pembangunan ruang kelas v. Pembangunan fasilitas air bersih dan sanitasi vi. Pembelian buku ajar dan buku lainnya untuk perpustakaan vii. Penyediaan alat bantu ajar viii. Peningkatan kapasitas guru ix. Aktifitas lainnya, sebutkan…..

Kami akan bertanya mengenai dukungan sekolah untuk siswa miskin Tujuan kami menanyakan hal ini adalah untuk mengetahui proporsi dari siswa yang mendapatkan manfaat dari dana ADB dan besaran bantuan yang mereka terima.

Pertanyaan 3: Seberapa besar dana ADB digunakan untuk berbagai kegiatan berikut 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 diantara tahun 2001 dan 2010?

i. Beasiswa ii. Bantuan uang transport iii. Program remedial iv. Kegiatan kelas akselerasi v. Kelas tambahan bagi siswa yang absen di kelas vi. Bentuk bantuan lainnya, sebutkan…….

PERTANYAAN 4: Berapa banyak siswa di sekolah Anda yang 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 menerima bantuan sesuai dengan kegiatan yang tercantum di pertanyaan 3

PERTANYAAN 5: Berapa banyak siswa miskin di sekolah Anda 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 yang menerima bantuan sesuai dengan kegiatan yang tercantum di pertanyaan 3?

Kami akan bertanya mengenai proses penentuan berbagai aktivitas di sekolah Anda.

PERTANYAAN 6 : Dari berbagai aktivitas yang disebutkan diatas, faktor apakah yang menjadi pertimbangan untuk menentukan suatu kegiatan? Pilih faktor berikut Questionnaires for School

i. Rencana pembangunan sekolah ii. Permintaan orang tua siswa iii. Permintaan dari dinas pendidikan setempat iv. Permintaan dari guru v. Permintaan dari kepala sekolah vi. Permintaan dari siswa

vii. Permintaan dari lembaga donor 45 viii. Lainnya, sebutkan….

46 Appendix 3

Kepala Sekolah / Wakil Kepala Sekolah Komite Sekolah lokal Pemerintah Donor (ADB) Lainnya, sebutkan… Kepala Sekolah / Wakil Kepala Sekolah Komite Sekolah lokal Pemerintah Donor (ADB) Lainnya, sebutkan… Ya (ke pertanyaan 10) Tidak Ya Tidak i. i. i. i.

v. v. ii. ii. ii. ii. iv. iv. iii. iii. berikut jawaban Pilih sekolah? perencanan dokumen pembuatan menginiasi yang Siapa 7: PERTANYAAN PERTANYAAN 8: Siapa yang terlibat dalam pembuatan dokumen perencanaan sekolah? Pilih jawaban berikut PERTANYAANApakah 9: dokumenperencanaan sekolahpernah ini dievaluasi oleh Dinas Pendidikan PERTANYAAN 10: Apakah rekomendasi yang diberikan oleh Dinas Pendidikan bermanfaat?

Kami ingin bertanya apakah kegiatan yang pernah dibiayai oleh proyek ini masih berlangsung (sustainabel)

PERTANYAAN 11: Apakah aktifitas pada pertanyaan 3 di atas masih dilakukan setelah proyek selesai ?

i. Ya (Ke pertanyaan 12) ii. Tidak

PERTANYAAN 12: Kementerian (Dinas) dan/atau lembaga swasta mana yang membantu membiayai aktifitas tersebut?

Pilih jawaban berikut

i. Kementerian Pendidikan dan Kebudayaan ii. Dinas Pendidikan iii. Lembaga Donor, yaitu …. iv. Komunitas lokal atau orang tua siswa

v. Lainnya , sebutkan …

Sekarang kami akan bertanya mengenai bagaimana siswa mengambil manfaat dari proyek ini dalam hal tingkat pendidikannya

PERTANYAAN 13: Apakah proyek ini meningkatkan pendaftar ke sekolah ini? Mohon berikan jumlah pendaftar tahun 2000 - 2013 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013

Questionnaires for School

PERTANYAAN 14: Apakah proyek ini meningkatkan jumlah siswa yang diterima di jenjang yang lebih tinggi (SD ke SMP, atau SMP ke SMA)? Mohon berikan data tersebut dari 2000 - 2013. 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013

47

48 Appendix 3

lai UN siswa miskin

2013 2013 2013 2013 2013 2013

2012 2012 2012 2012 2012 2012

ftarkan diri tahun 2000 –2013 –2013 2000 tahun diri ftarkan 2011 2011 2011 2011 2011 2011

ng berhasil lulustahun dari 2000–2013. MP ke SMA)? Mohon berikan data tersebut MP ke SMA)? Mohon berikan ri tahun 2000–2013. ikanrata-rata data tersebut nilaiUN dari 2000 rikan data rata-rata ni

2010 2010 2010 2010 2010 2010

2009 2009 2009 2009 2009 2009

2008 2008 2008 2008 2008 2008

2007

2007 2007 2007 2007 2007

2006

s 9) di sekolah ini? Mohon ber 2006 2006 2006 2006 2006

2005

2005 2005 2005 2005 2005

h ini? Mohon berikan data siswa miskin ya anda? Berikan data siswa miskin yang menda

2004

2004 2004 2004 2004 2004

2003

2003 2003 2003 2003 2003

yang diterima di jenjang yang lebih tinggi (SD ke SMP, atau S 2002

2002 2002 2002 2002 2002 (untuk siswa kelas 6 atau kela 6 atau kelas siswa (untuk siswa miskin (kelas 6 atau kelas 9) di sekolah ini? Mohon be

2001

2001 2001 2001 2001 2001

2000 2000

2000 2000 2000 2000

an tingkat kelulusan di sekolah ini? Mohon berikan data siswa yang berhasil lulus da an pendaftar dari siswa miskin di sekolah an tingkat kelulusan siswa miskin di sekola

PERTANYAAN 15: Apakah proyek ini meningkatk PERTANYAAN 16: Apakah proyek ini meningkatkan nilai Ujian Nasional –2013 PERTANYAAN 17: Apakah proyek ini meningkatk PERTANYAAN 18: Apakah proyek ini meningkatkan jumlah siswa miskin dari 2000–2013. PERTANYAAN 19: Apakah proyek ini meningkatk PERTANYAAN 20: Apakah proyek ini meningkatkan nilai Ujian Nasional 2000–2013 dari tersebut

Kami ingin meminta pendapat mengenai kesuksesan proyek ini

PERTANYAAN 21: Bagaimana proyek ini memberi manfaat untuk sekolah ini? Apakah proyek ini meningkatkan kualitas pendidikan di sekolah ini?

PERTANYAAN 22: Apakah anda menginginkan sekolah anda untuk terus menerima bantuan ADB di masa yang akan datang?

Note: The English version of this questionnaire will be available upon request.

Questionnaires for School

49

APPENDIX 4: NUMBER AND TYPE OF PROJECT SCHOOLS

Table A4: Number and Types of Schools Covered by the Decentralized Basic Education Project

Primary Primary Junior Secondary Junior Secondary Number of School Madrasah School Madrasah Item Districts Public Private Public Private Public Private Public Private Total Loan Bali 9 715 0 15 16 114 58 5 3 926 Nusa Tenggara Barat 9 1,946 1 55 319 303 35 8 167 2,834 Nusa Tenggara Timur 2 308 94 0 7 57 16 0 0 482 Subtotal 20 2,969 95 70 342 474 109 13 170 4,242 Grant Nusa Tenggara Timur 6 468 363 3 19 96 44 1 2 996 Total 26 3,437 458 73 361 570 153 14 172 5,238 Source: ADB. 2012. Completion Report: Indonesia: Decentralized Basic Education Project. Manila.

APPENDIX 5: RATING MATRIX FOR CORE EVALUATION CRITERIA

Table A5.1: Rating Matrix for Core Evaluation Criteria Ratings PCR PPER Reasons for Disagreements/Comments Relevance Highly Highly  The Decentralized Basic Education Project (DBEP) was implemented Relevant Relevant during the start of the decentralization era in Indonesia and helped to fill gaps in technical capacity.  The project aimed to improve access to and quality of basic education, especially for the poor.  The project is aligned with the priorities of ADB and the Indonesian government. Effectiveness Effective Effective  Based on questionnaires sent to schools and mean-difference testing, beneficiaries on average had seen statistically significant improvements in terms of students’ enrollment, transition, completion, and exam scores. a This is also true for poor students.  The project made a significant contribution to lowering dropout rates of poor children in such poor districts as Bangli, East Lombok, and Timor Tengah Utara, either by sending teachers to remote areas or providing facilities for poor children.  High turnover of trained officers at the district level may have hindered the project’s effectiveness. For some districts in which DBEP personnel were organized in teams, however, the remaining staff were able to keep up with the turnover. Efficiency Efficient Efficient  Despite the limited size of grants, the project achieved notable results in improving local capacity for basic education delivery and rehabilitating some school infrastructure. Sustainability Partly Likely  Some of the project’s outputs are still implemented under current government programs, such as creating the proposal for a program or project similar to school development plans even though the format of such reports may not be exactly the same.  School committees, which were pioneered by the project, are still functioning.  It was difficult, however, to determine the project’s impact on the rehabilitation of school facilities since other sources of funding for school repairs and rehabilitation have been made available after the project’s life. Impact Significant Significant  Though difficult to measure because of the differences in method and sampling frame used between baseline survey (2006, 2008) and the study team’s analysis, DBEP was found to have a significant impact on school-based management and community involvement.  The project had a positive spillover impact or peer effect on nonbeneficiary schools. As some training funded by the District Education Development Fund was attended by all teachers and principals of both beneficiary and nonbeneficiary schools, DBEP had an effective impact on changing the behavior of teachers and principals toward school-based management and on improving education quality in their schools. Institutional Not Rated Significant  The project had made a significant contribution toward institutional Development development, in particular through the establishment of school committees and education boards at district level. Community involvement had also improved. a With an exception of the District of West Lombok, where the average student enrollment for the selected schools decreased after DBEP was implemented. Source: Study team’s assessment.

52 Appendix 5

Table A5.2: Core Evaluation Criteria Rating Values

Rating Value Relevance Effectiveness Efficiency Sustainability 3 Highly relevant Highly effective Highly efficient Most likely 2 Relevant Effective Efficient Likely 1 Less than relevant Less than Less than Less likely effective efficient 0 Irrelevant Ineffective Inefficient Unlikely Source: Independent Evaluation Department. 2006. Guidelines for Preparing Performance Evaluation Reports for Public Sector Operations. Manila: ADB (as well as its amendment effective from March 2013).

APPENDIX 6: EDUCATION STATISTICS

Table A6.1: Net Enrollment in Primary School by Beneficiary Provinces, 2003–2013 (%) Province 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 Bali 91.58 93.48 93.26 93.33 94.49 94.93 94.99 95.53 90.39 91.06 94.28 Nusa Tenggara Barat 92.48 92.42 92.99 94.50 94.20 94.20 94.75 95.16 92.69 93.56 96.63 Nusa Tenggara Timur 88.27 90.79 92.00 91.58 91.61 91.72 92.46 93.03 92.13 92.28 93.60 Indonesia 92.55 93.04 93.25 93.54 93.78 93.99 94.37 94.76 91.03 92.49 95.53 Note: Since 2007, the net enrollment rate includes nonformal education (“Package A,” which is primary school; “Package B,” which is junior secondary). Source: Indonesian Statistics Office.

Table A6.2: Net Enrollment in Junior Secondary School by Beneficiary Provinces, 2003–2013 (%) Province 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 Bali 68.63 69.37 70.03 70.15 66.69 67.34 67.38 67.83 69.16 75.07 80.69 Nusa Tenggara Barat 57.19 61.70 67.53 69.62 70.79 71.44 71.32 71.73 76.70 77.81 80.18 Nusa Tenggara Timur 39.10 43.26 43.00 47.23 49.75 49.87 50.21 51.03 56.74 55.89 59.24 Indonesia 63.49 65.24 65.37 66.52 66.90 67.39 67.43 67.73 68.12 70.84 73.72 Note: Since 2007 the net enrollment rate includes nonformal education (“Package A,” which is primary school; “Package B,” which is junior secondary). Source: Indonesian Statistics Office.

Table A6.3: Classroom Conditions by Province, 2002 (%) Primary School Junior Secondary School Heavily Lightly Heavily Lightly Province Damaged Damaged Good Damaged Damaged Good DKI Jakarta 4.48 14.64 80.88 2.26 9.76 87.98 Jawa Barat 32.40 36.45 31.15 5.20 10.92 83.88 Banten 25.74 28.25 46.00 4.93 10.59 84.48 Jawa Tengah 18.94 38.37 42.69 1.49 7.76 90.74 DI Yogyakarta 14.13 40.70 45.17 2.38 8.40 89.21 Jawa Timur 17.55 36.87 45.58 1.96 6.37 91.67 N Aceh Darussalam 23.25 34.91 41.84 4.21 10.93 84.86 Sumatera Utara 20.22 38.64 41.14 3.74 9.65 86.61 Sumatera Barat 18.24 38.88 42.89 4.47 10.99 84.54 Riau 30.54 28.62 40.84 0.92 4.20 94.88 Jambi 22.26 25.61 52.12 3.48 9.14 87.38 Sumatera Selatan 21.74 33.17 45.09 1.90 6.50 91.60 Bangka Belitung 13.26 29.93 56.81 1.95 6.56 91.50 Bengkulu 31.56 35.66 32.78 6.61 14.08 79.32 Lampung 28.70 45.60 25.70 2.86 8.83 88.31 Kalimantan Barat 28.39 33.32 38.29 2.87 9.67 87.46 Kalimantan Tengah 30.92 31.47 37.61 1.83 3.48 94.69 Kalimantan Selatan 27.52 33.18 39.30 3.40 11.01 85.58 Kalimantan Timur 22.31 36.00 41.70 3.42 9.29 87.29

54 Appendix 6

Primary School Junior Secondary School Heavily Lightly Heavily Lightly Province Damaged Damaged Good Damaged Damaged Good Sulawesi Utara 17.16 33.04 49.79 7.23 17.40 75.37 Gorontalo 21.86 32.95 45.19 7.54 12.87 79.59 Sulawesi Tengah 34.03 33.00 32.97 3.53 6.85 89.62 Sulawesi Selatan 19.69 31.52 48.78 4.08 11.08 84.83 Sulawesi Tenggara 31.99 33.12 34.89 3.74 9.01 87.25 Maluku 36.24 30.98 32.78 9.57 13.74 76.69 Maluku Utara 38.07 29.50 32.43 9.46 18.72 71.82 Bali 20.08 35.33 44.59 4.79 10.33 84.88 Nusa Tenggara Barat 16.64 32.97 50.39 1.68 9.63 88.69 Nusa Tenggara Timur 36.68 29.37 33.96 7.40 11.94 80.66 Papua 23.42 25.49 51.09 9.47 8.38 82.15 National Average 24.27 32.92 42.82 4.28 9.94 85.78 Source: National Development Planning Agency. 2002. National Program for Indonesian Children: Education. Jakarta.

APPENDIX 7: POVERTY STATISTICS

Table A7: Percentage of Poor People by Province, 2007–2012

Province 2007 2008 2009 2010 2011 2012 Papua 37.08 37.53 36.80 31.98 31.11 30.66 Papua Barat 35.12 35.71 34.88 31.92 28.20 27.04 Maluku 29.66 28.23 27.74 23.00 21.78 20.76 Nusa Tenggara Timur 25.65 23.31 23.03 21.23 20.88 20.41 Gorontalo 24.88 25.01 23.19 18.75 17.33 17.22 Nusa Tenggara Barat 23.81 22.78 21.55 19.73 18.63 18.02 Aceh 23.53 21.80 20.98 19.57 19.46 18.58 Lampung 20.98 20.22 18.94 16.93 16.18 15.65 Sulawesi Tengah 20.75 18.98 18.07 15.83 15.40 14.94 Bengkulu 20.64 18.59 18.30 17.50 17.70 17.51 Sulawesi Tenggara 19.53 18.93 17.05 14.56 13.71 13.06 Jawa Tengah 19.23 17.72 16.56 15.76 15.34 14.98 Jawa Timur 18.51 16.68 15.26 14.23 13.40 13.08 DI Yogyakarta 18.32 17.23 16.83 16.08 16.05 15.88 Sumatera Selatan 17.73 16.28 15.47 14.24 13.78 13.48 Sulawesi Barat 16.73 15.29 13.58 13.89 13.24 13.01 Sulawesi Selatan 13.34 12.31 11.60 10.29 10.11 9.82 Jawa Barat 13.01 11.96 11.27 10.65 10.09 9.89 Sumatera Utara 12.55 11.51 11.31 11.33 10.67 10.41 Maluku Utara 11.28 10.36 9.42 9.18 8.47 8.06 Kalimantan Barat 11.07 9.30 9.02 8.60 8.17 7.96 Sumatera Barat 10.67 9.54 9.50 9.04 8.19 8.00 Riau 10.63 9.48 8.65 8.47 8.22 8.05 Sulawesi Utara 10.10 9.79 9.10 8.51 8.18 7.64 Kalimantan Timur 9.51 7.73 7.66 6.77 6.68 6.38 Jambi 9.32 8.77 8.34 8.65 8.42 8.28 Kepulauan Riau 9.18 8.27 8.05 7.40 7.11 6.83 Kalimantan Tengah 8.71 7.02 6.77 6.56 6.51 6.19 Kepulauan Bangka Belitung 8.58 7.46 6.51 5.75 5.53 5.37 Banten 8.15 7.64 7.16 6.32 5.85 5.71 Kalimantan selatan 6.48 5.12 5.21 5.29 5.06 5.01 Bali 6.17 5.13 4.88 4.20 4.18 3.95 DKI Jakarta 4.29 3.62 3.48 3.75 3.69 3.70 Indonesia 15.42 14.15 13.33 12.49 11.96 11.66 Source: Indonesian Statistics Office. 2014.