Document of The World Bank

Public Disclosure Authorized Report No: ICR0000792

IMPLEMENTATION COMPLETION AND RESULTS REPORT (IBRD-46690)

ON A

Public Disclosure Authorized LEARNING AND INNOVATION LOAN (LIL)

IN THE AMOUNT OF US$ 2.66 MILLION

TO THE

REPUBLIC OF INDONESIA

FOR A

GLOBAL DEVELOPMENT LEARNING NETWORK PROJECT Public Disclosure Authorized

June 26, 2008

Human Development Sector Unit East Asia and Pacific Region

Public Disclosure Authorized

CURRENCY EQUIVALENTS

Exchange Rate Effective February 2008

Currency Unit = Rupiah (IDR) IDR 1.00 = US$ 0.0001102 US$ 1.00 = IDR 9,074.41

FISCAL YEAR January 1 – December 31 (at appraisal)

ABBREVIATIONS AND ACRONYMS BPKP Financial and Development Supervisory Board CAS Country Assistance Strategy CLR Center for Learning Resources CPCU Central Project Coordination Unit of DUE and QUE CPIU Central Project Implementation Unit DGHE Directorate General of Higher Education DLC(s) Distance Learning Center(s) Development of Undergraduate Education/Quality of DUE/QUE Undergraduate Education EAP East Asia Pacific EAPA East Asia Pacific Association FMR Financial Management Report GDLN Global Development Learning Network GoI Government of Indonesia IBRD International Bank for Reconstruction and Development IT Information Technology ISG Information Solutions Group LA Loan Agreement LIL Learning and Innovation Loan KPKN Ministry of Finance Treasury Office MoNE Ministry of National Education MOF Ministry of Finance NGO(s) Non-Governmental Organization(s) NOC Network Operations Center PAR Procurement Assessment Review PMM Project Management Manual UI University of Indonesia UNHAS Hasanuddin University UNRI University of Riau UNUD Udayana University WBI World Bank Institute

Vice President: James W. Adams Country Director: Joachim von Amsberg Sector Manager: James Stevens (Acting) Project Team Leader: Mae Chu Chang/Ratna Kesuma ICR Team Leader: Robert L. McGough

INDONESIA GLOBAL DEVEVELOPMENT LEARNING PROJECT

CONTENTS

Data Sheet A. Basic Information B. Key Dates C. Ratings Summary D. Sector and Theme Codes E. Bank Staff F. Results Framework Analysis G. Ratings of Project Performance in ISRs H. Restructuring I. Disbursement Graph

1. Project Context, Development Objectives and Design...... 1 2. Key Factors Affecting Implementation and Outcomes ...... 5 3. Assessment of Outcomes...... 16 4. Assessment of Risk to Development Outcome...... 24 5. Assessment of Bank and Borrower Performance ...... 24 6. Lessons Learned ...... 28 7. Comments on Issues Raised by Borrower/Implementing Agencies/Partners ...... 29 Annex 1. Project Costs and Financing...... 30 Annex 2. Outputs by Component ...... 31 Annex 3. Economic and Financial Analysis...... 32 Annex 4. Bank Lending and Implementation Support/Supervision Processes ...... 33 Annex 5. Beneficiary Survey Results...... 35 Annex 6. Stakeholder Workshop Report and Results...... 36 Annex 7. Summary of Borrower's ICR and/or Comments on Draft ICR...... 37 Annex 8. Comments of Cofinanciers and Other Partners/Stakeholders...... 38 Annex 9. List of Supporting Documents ...... 39 Map IBRD 36232...... 40

A. Basic Information Global Development Country: Indonesia Project Name: Learning Network Project (LIL) Project ID: P073970 L/C/TF Number(s): IBRD-46690 ICR Date: 06/26/2008 ICR Type: Core ICR REPUBLIC OF Lending Instrument: LIL Borrower: INDONESIA Original Total USD 2.7M Disbursed Amount: USD 1.5M Commitment: Environmental Category: C Implementing Agencies: Ministry of National Education Cofinanciers and Other External Partners:

B. Key Dates Revised / Actual Process Date Process Original Date Date(s) Concept Review: 04/03/2001 Effectiveness: 03/04/2003 03/04/2003 Appraisal: 04/10/2002 Restructuring(s): Approval: 06/28/2002 Mid-term Review: 10/11/2004 Closing: 09/30/2006 12/31/2007

C. Ratings Summary C.1 Performance Rating by ICR Outcomes: Unsatisfactory Risk to Development Outcome: Substantial Bank Performance: Unsatisfactory Borrower Performance: Moderately Satisfactory

C.2 Detailed Ratings of Bank and Borrower Performance (by ICR) Bank Ratings Borrower Ratings Quality at Entry: Unsatisfactory Government: Moderately Satisfactory Implementing Quality of Supervision: Moderately Satisfactory Moderately Satisfactory Agency/Agencies: Overall Bank Overall Borrower Unsatisfactory Moderately Satisfactory Performance: Performance:

i C.3 Quality at Entry and Implementation Performance Indicators Implementation QAG Assessments Indicators Rating Performance (if any) Potential Problem Project Quality at Entry Yes None at any time (Yes/No): (QEA): Problem Project at any Quality of Yes Moderately Satisfactory time (Yes/No): Supervision (QSA): DO rating before Satisfactory Closing/Inactive status:

D. Sector and Theme Codes Original Actual Sector Code (as % of total Bank financing) Telecommunications 55 45 Tertiary education 45 55

Theme Code (Primary/Secondary) Education for the knowledge economy Primary Primary Technology diffusion Primary Primary

E. Bank Staff Positions At ICR At Approval Vice President: James W. Adams Jemal-ud-din Kassum Country Director: Joachim von Amsberg Mark Baird Sector Manager: James A. Stevens Christopher J. Thomas Project Team Leader: Mae Chu Chang Jerry G. Strudwick ICR Team Leader: Robert L. McGough ICR Primary Author: Robert L. McGough

F. Results Framework Analysis

Project Development Objectives (from Project Appraisal Document) To (i) strengthen the capacity of policy makers in Indonesia by providing access to new information, seminars, and linkages with counterparts in the region and around the world. tand (ii) to provide the latest communication technology for quality distance learning and information exchange between institutions in Indonesia and regional and global network distance learning networks.

Revised Project Development Objectives (as approved by original approving authority)

ii

(a) PDO Indicator(s)

Original Target Formally Actual Value Values (from Revised Achieved at Indicator Baseline Value approval Target Completion or documents) Values Target Years A trend that reflects an increased number of development focused GDLN Indicator 1 : learning products subscribed to by public and private sector professionals 1000 person days in year 1 of DLC less than 1000 GDLN activity; person days over Value 2500 in year 2 of the life of the quantitative or 0 DLC GDLN project estimate Qualitative) activity; 3225 in only about 15% of year 3 of DLC target achieved. GDLN activity. Date achieved 09/06/2002 12/31/2007 12/31/2007 Comments (incl. % achievement) A trend that shows an increasing number and variety of organizations using Indicator 2 : GDLN DLCs.

From back of zero A trend that shows there was a greater an increasing number and variety Value number and of organizations quantitative or 0 variety of using DLCs by EoP Qualitative) organizations but data is using GDLN imprecise and trend DLCs. is difficult to demonstrate Date achieved 09/06/2002 12/31/2007 12/31/2007 Comments (incl. % achievement) Indicator 3 : DLC facility utilization rate of 55% by EOP less than 20% by Value EoP regional quantitative or 0 55% by EOP experience suggests Qualitative) 55% was a few ambitious target Date achieved 09/06/2002 12/31/2007 12/31/2007 Comments (incl. % achievement)

iii (b) Intermediate Outcome Indicator(s)

Original Target Actual Value Formally Values (from Achieved at Indicator Baseline Value Revised approval Completion or Target Values documents) Target Years Indicator 1 : DLCs certified by ISG All 4 DLCs certified by Value 12/31/2003. (quantitative 0 January 2007 Completed with or Qualitative) substantial delay (01/09/2007) Date achieved 09/06/2002 01/09/2007 01/31/2007 Comments (incl. % achievement) DLC Business Plans reviewed twice a year and modified to reflect progress and Indicator 2 : newly identified opportunities Value BPS to be Business Plans (quantitative 0 reviewed twice a were not routinely or Qualitative) year developed. Date achieved 09/06/2002 12/31/2007 12/31/2007 Comments (incl. % achievement) GDLN M&E system producing quality input to BPs and DLC management Indicator 3 : reviews Value Effective systems Unavailability of (quantitative 0 by MTR baseline data or Qualitative) Date achieved 09/06/2002 10/11/2004 12/31/2007 Comments (incl. % achievement) 45% of content is provided by national and international partners by end of year Indicator 4 : 2 of DLC GDLN activity 60% of content is No significant provided by content prepared in Value national and fist three year and (quantitative 0 international failure to reach or Qualitative) partners by end of even targets for year 3 of DLC 2004 by EoP GDLN activity Date achieved 09/06/2002 06/30/2009 12/31/2007 Comments (incl. % achievement)

iv

G. Ratings of Project Performance in ISRs

Actual Date ISR No. DO IP Disbursements Archived (USD millions) 1 12/29/2002 Satisfactory Satisfactory 0.00 2 06/27/2003 Satisfactory Satisfactory 0.03 3 12/22/2003 Unsatisfactory Unsatisfactory 0.53 4 06/22/2004 Unsatisfactory Unsatisfactory 0.91 5 12/21/2004 Unsatisfactory Unsatisfactory 0.91 6 05/10/2005 Unsatisfactory Unsatisfactory 0.91 7 12/19/2005 Satisfactory Satisfactory 0.98 8 07/14/2006 Satisfactory Satisfactory 1.36 9 06/22/2007 Satisfactory Satisfactory 1.36 10 06/12/2008 Unsatisfactory Unsatisfactory 1.46

H. Restructuring (if any) Not Applicable

I. Disbursement Profile

v

1. Project Context, Development Objectives and Design The Global Development Learning Network (GDLN) was launched by the Bank as a unique partnership of public, private, and non-government organizations, to build a virtual global community and to provide cost-effective learning opportunities for key stakeholders in the development process, in order to close the knowledge gap between developed and developing countries as new countries join the network. Currently, GDLN affiliated Distance Learning Centers (DLCs) are fully operational in all of the Bank’s operational regions. In the EAP region, DLCs currently exist in almost all of the larger nations as well as many of the smaller. Some are connected to national networks which provide even more distribution capacity to support distance learning.

The Indonesia GDLN Project was Bank financed through a Learning and Innovation Loan (LIL), as a pilot, to test the potential effectiveness and sustainability of the Global Development Learning Network (GDLN) in Indonesia. The project was also designed to expand access to development management information, strengthen the capacity of key- decision makers in both the public and private sectors, and to improve levels of regional dialogue on policy-related issues. In addition to the above, the project was also designed to develop appropriate technologies to link the Jakarta-based GDLN center to three regional university centers, representing the western, central and eastern regions. These linkages were intended to magnify the potential impact of the GDLN as a dynamic, potentially low cost development tool and provide an effective platform for regional distance learning and information exchange between key Indonesian institutions and the regional and global networks of GDLN.

1.1 Context at Appraisal The joint IBRD/IFC/IDA 2001 CAS (Document number: 21580-IND, last discussion prior to date of PAD was 1/20/2001) supports efforts to reduce poverty and vulnerability in a more democratic and decentralized environment. It articulates a strategy that specifically focuses on: (a) sustaining economic recovery and promoting broad-based growth; (b) building national institutions for an accountable Borrower; and (c) delivering better public services to the poor.

The project was designed to be an important communication tool to support the realization of the CAS objectives, in that it would directly contribute to improving development knowledge in both the public and private sectors. In addition, it would contribute to: (a) improving skill levels in the public service and other segments of society; (b) increasing knowledge sharing within Indonesia and internationally; and (c) developing consensus and national/regional dialogue. It was anticipated that the outcome of the LIL would be the basis for GDLN replication to additional universities within Indonesia.

1.2 Original Project Development Objectives (PDO) and Key Indicators The PDOs were not clearly stated in the text of the Project Appraisal Document (PAD), however, they were stated in Annex 1 of that document as part of the presentation of Key

1

Performance Indicators. For the purposes of this ICR, the PDOs stated in Schedule 2 of the Loan Agreement (LA) will be used. They are as follows:

The objective of the Project is to assist the Borrower in (i) expanding the access to information on development, through the strengthening of the capacity of decision makers in the public and private sectors and the improvement of regional dialogue on policy-related issues, and (ii) providing the latest communication technology for quality distance learning and information exchange between institutions in Indonesia and regional and global network distance learning networks.

Key Indicators The following are the key indicators as stated in Schedule 6 of the LA.

Indicator May May 2004 May 2005 Closing 2003 Date

Distance Learning Center GDLN x certification completed successfully Business plans reviewed bi-annually and x x x x modified to reflect progress and newly identified opportunities Financial management systems in x operation and effective Monitoring and assessment systems x x x producing quality input to bi-annual business plan reviews and Project management reviews Evidence of a trend that reflects an 1000 person- 2500 3225 increase in the number of development days training person-days person-days focused GDLN learning products training training (including: courses; training sessions; seminars; debates; awareness campaigns; and national, global and local dialogues on development issues) subscribed to by public and private professionals Evidence of a trend that reflects a clear x x x progression towards Distance Learning Centers becoming self-sustaining Quality and relevance of courses offered 75% of 80% of 85% of by the GDLN is evaluated satisfactory participants participants participants or better by participants Content provided by national and 45% 55% 65% international partners Distance learning facility utilization 30% 45% 55% rate

2

1.3 Revised PDO (as approved by original approving authority) and Key Indicators, and reasons/justification The PDOs were not formally revised. However, since there were long delays in the installation and certification of the equipment, the targets of the Performance Indicators (PI) were shifted to 2007 after the certification of the equipment was completed on January 2007 (ISR10, Annex 9). Some of the performance indicators were also reduced to reflect the reality that the project DLCs would have a more limited time for operation within the project implementation period. This decision to reduce the indicator targets, in effect, reduced the required impact and outcomes of the project.

1.4 Main Beneficiaries The Statement of Main Beneficiaries was not provided in the PAD or LA. However in the PAD, under the subsection, Participation (Section B.4), the following quote describes the intended beneficiaries to some extent, “the project is expected to increase policy dialogue through the extension of opportunities for Borrower officials, private sector professionals and civil society to participate and contribute to the development of the country.”

In addition, the PDOs (LA, Schedule 2) did identify decision makers in the public and private sectors as being main beneficiaries. Since the four project sites were established at universities, it is also reasonable to assume that university administrators, faculty, and students would be direct beneficiaries of the project.

1.5 Original Components Component 1 – Establishing, retrofitting and equipping: the main GDLN center at UI's Salemba campus (a section of a new building will be committed to GDLN) and the three selected sub-centers at the Universities of Hasanuddin (in Makassar), Riau (in Pekanbaru) and Udayana (in Denpasar)

The main Global Development Learning Network (GDLN) center at the University of Indonesia (UI) will be retrofitted and equipped to include two technology-based classrooms (synchronous and asynchronous) connected to GDLN Washington through two-way multimedia facilities for video, data, and voice exchange. The synchronous classroom will enable video-conferencing with real-time, video interaction between local course participants, remote course instructors and other international participants, and will also include facilities for simultaneous interpretation. Participants will have access to a computer screen sharing facility (T. 120 Standard) which can be used to support synchronous learning.

The asynchronous room will be used for asynchronous course delivery, which is Internet based. Both classrooms will be outfitted with 30 computer stations, (which include hardware, software, pedagogical tools, internet hook-up, and the installation and/or upgrading of electrical and telephone wiring). The Jakarta GDLN center will also include a reception area, offices, and support facilities for Distance Learning Center(s) (DLC) staff.

3

The three sub-centers will be retrofitted and partially equipped to include one multimedia classroom. Each sub-center will combine the asynchronous and synchronous functions into a single classroom, which will provide pedagogical tools for 15 participants. The rooms will be retrofitted to accommodate the required wiring for computer networking, support clean power delivery, and to enable a high quality video-conferencing. The delivery speed and all synchronous features will be identical to that of the main center in Jakarta (with the exception of reduced access speed for asynchronous course development, as the asynchronous server will be located at UI). However, the multimedia two-way network program exchange between the sub-centers and the GDLN center will be based on IP protocol (technology), which will significantly reduce the up-front investment costs. To ensure long-term sustainability, the sub-centers will be designed to make optimum use of the existing domestic connectivity and the communication infrastructure already provided by PT Telkom.

Component 2: Support for center and sub-centers operation. This component includes (a) financing of operation costs on a decreasing basis (100%; 75%; and 50%) over the four years of implementation; (b) technical assistance for training of DLC staff, periodic evaluation of their operations and management effectiveness, and the establishment of financial accounts and their annual audit; (c) funds for the sponsoring and development of local course content and partnerships; and (d) financing of technology review and upgrade.

1.6 Revised Components As described above the original project included two components: (1) establishing, retrofitting and equipping the main GDLN center and the three sub-centers; and (2) supporting GDLN operations on a decreasing basis over the project's four year implementation period.

During implementation, Component 2 was revised and split into two separate components (Components 2 and 3). Under this arrangement, the original Component 1 did not change; Component 2 included the provision of support for GDLN operations on a decreasing basis over the project's four year implementation period; and Component 3 included the provision of support for project management at the level of the Central Project Implementation Unit (CPIU).

The ICR found no evidence that the required technology review and upgrading was included in the revised components, or was implemented (Section 1.5 Component 2(d)). This is an important note in that one of the criticisms of this project was that it did not, at project completion, provide the latest and most suitable technology as required by PDO1.

1.7 Other significant changes Due to long delays and procurement problems, there were significant changes made to the project’s implementation arrangements and schedule. These delays resulted in the project receiving one loan extension that changed the closing date of the loan from

4

September 20, 2006 to December 31, 2007. The issues associated with these delays are discussed in more detail in Section 2.2.

Following the completion of the project, the project sites joined the INHERENT (Indonesia’s new national video-conference network for university development) to provide wider coverage of the national network through national networking. At the time of ICR preparation, discussions were underway to establish an Indonesia GDLN Association with the project DLC at the University of Indonesia, Jakarta as the hub linking Indonesia’s three regional DLCs with GDLN networks in the South East Asia region. This Association, through its connection to INHERENT would provide the possibility for linking the domestic system with the international GDLN network. These actions must be seen as positive as they provide significant potential for the future use and sustainability of the project sites; however, these actions did not take place during the project implementation period and are only obliquely mentioned as a possibility in the Risk Section of the PAD (page 10) - a one line reference indicating the intent that the project sites should join existing domestic and regional distance learning networks in order to enhance their potential for self sustainability. While INHERENT was not in the original design of the project, the GOI nevertheless has drawn lessons from the experiences of Bank-funded GDLN to develop a distance learning network that could serve the original objective of (i) expanding the access to information on development, through the strengthening of the capacity of decision makers in the public and private sectors and the improvement of regional dialogue on policy-related issues, and (ii) providing the latest communication technology for quality distance learning and information exchange between institutions in Indonesia and regional and global network distance learning networks. These developments should be considered a significant and positive factor in the development of the project DLCs, however, since these activities were not part of the Loan Agreement, the ICR did not consider the joining of INHERENT as a project outcome to be considered or evaluated.

2. Key Factors Affecting Implementation and Outcomes

The project was affected by several factors that plagued the project’s implementation and outcomes. The project went for almost four years with limited implementation progress, procurement or disbursement. In the end, these long delays resulted in significant shortfalls in the project’s performance and outcomes as required under the provisions of the Loan Agreement (LA). The following are several key factors that contributed to these long delays:

(a) The project suffered from unusual and difficult procurement problems throughout its implementation (see Section 2.1). These problems required long and difficult interactions between the Bank and the Borrower; (b) The Borrower and the University of Indonesia grew frustrated with the slow progress of the project, and when planned activities failed to be implemented counterpart funds were reallocated for use in other university programs during several project years;

5

(c) Bank (ISG) and contractor staff turnovers were seen to have a negative impact on project implementation; (d) Project documentation and Borrower comments (see Borrower’s ICR, Annex 9) indicate that the Borrower was not comfortable with the use of the Information Solutions Group (ISG) as the facilitator for the technical design and procurement of equipment. The Borrower believed that ISG was too authoritative in its dealings with the borrower and that some of the technical services, training, and advice given by ISG were less than satisfactory. As a result communications, cooperation, and implementation activities were often delayed and could be characterized as troubled between the Borrower and the Bank. This resulted in a much longer than expected procurement cycle, wrong decisions being made (i.e., selection of technology for connectivity between Jakarta and Washington), delayed disbursements, a loan extension, and unfulfilled expectations on the part of the Borrower (Borrower’s ICR and the Data and Findings from Local Interviews to Support Preparation of the Implementation Completion Report for GDLN LIL, 2008, Annex 9); (e) There was an unfortunate accident in Denpasar, Bali, that disabled one ISG engineer, with the result being the stoppage of significant staff training by ISG. This training had been formally agreed upon by both the Borrower and the Bank, but was not fully provided during the project period; (f) There were delays by contracted firms (the first was replaced during project implementation). These delays also contributed to additional delays in procurement, installation of the equipment, and the training of university staff to operate the highly technical equipment; and (g) There were multi-year delays in the required certifications of installed equipment and Jakarta-Washington connectivity (ISDN) at the project facilities by ISG. These certifications were planned for May 2003, but were actually provided by ISG on January 9, 2007. These certifications were a required condition before ISG would allow any video conference link through the Bank to the GDLN.

2.1 Project Preparation, Design and Quality at Entry Based on lessons learned. The concept and design of the GDLN centers were based on lessons learned during the establishment of previous GDLN centers in the EAP and Africa regions. The use of the LIL as a financing instrument to establish GDLN centers had precedent in other countries; for example, the Bank used LILs to support the establishment of a number of GDLN centers in Africa. The one noted difference was that in Africa, the LILs typically financed the establishment of a single GDLN center and they were for larger amounts (up to US$5.0 million), while in this project (US$2.6 million) the LIL was used to finance four centers that were linked together, and located in separate universities. Most of the funds were intended for equipment and operational costs on a declining basis over the duration of the project.

Borrower’s early concerns about procurement. During the design process, the technical staff from the Bank provided a clear explanation of the implementation process and the necessary procurement based on their previous experience in developing similar GDLN units in other places. However, during project preparation, the Borrower’s

6

preparation team came to understand that the Bank, ISG, would not allow substantial change in the technology and design specifications to accommodate the local characteristics of the project, especially when it came to the technical specifications of computer equipment (see consultant report, Data and Findings from Local Interviews to Support Preparation of the Implementation Completion Report for GDLN LIL, 2008, Annex 9). During this time, the Bank (ISG) also required that all technology equipment be first delivered to Washington for testing, simulation, and calibration, and then transshipped to the project sites. ISG had these requirements to insure compatibility of equipment and DLCs throughout the Bank’s network. However, this process left the Borrower with a perception that they were not in control of the project and in fact, had few options regarding the selection and installation of the equipment, even though they were the loan recipient, project manager and purchaser of the equipment. The Borrower continues to believe that there may have been other lower cost options available which would have also provided equal levels of compatibility with GDLN technologies used in the EAP region.

The Borrower’s concerns are given some credence when one understands that the GDLN network currently has other non-Bank member DLCs and often links to guest DLCs in all of its regions which were not established with ISG oversight. In these DLCs one finds various brands and types of equipment that were not staged, simulated or calibrated by the Bank before operation. The DLCs at Australia National University (ANU) and the University of the South Pacific (USP) are good examples in the EAP region. The ICR suggests that there might have been a more positive, yet equally effective approach to the selection, procurement, installation, calibration and certification of equipment than the one that was offered and strongly recommended by the Bank. It also questions the need for sole source contracting for such ‘plug and play’ equipment as primarily purchased for this project.

The ICR also notes that these issues, along with other procurement concerns have been the basis for many discussions, delays and some uncomfortable relationships between the Bank and Borrower (See PSR1, PSR2 and PSR3, Annex 9). They are seen by both the Bank and Borrower project teams as factors that negatively affected the implementation of the project.

Inadequacy of project documentation. The ICR also found that the early project documentation was lacking in several ways. The PAD did not use the standard project format, but instead used a purpose built format that may have been intended for use only with LILs. As it is written, it does not adequately provide the necessary baseline data required to support the preparation of the ICR. As an example, the PDOs were not stated in the main body of the PAD, but instead were stated in Annex A as a part of the matrix describing the performance indicators. The PDOs were also stated in a form that is difficult to measure in any meaningful way. They were never officially clarified and should have been. Furthermore, there were three PDOs stated in Annex A of the PAD, while there were only two approved PDOs stated in the Loan Agreement (Schedule 2).

7

When one looks at both sets of objectives, there is one apparent variance in the meaning of the PAD’s PDOs as compared with approved objectives found in the Loan Agreement (Schedule 2). In the approved objectives (Loan Agreement), it was declared that the project would provide the latest communication technology for quality distance learning and information exchange between institutions in Indonesia and regional and global network distance learning networks. In the PDOs stated in the PAD, and subsequently used in the ISRs, the meaning of the approved objective was less rigorous saying that the project would provide an effective platform for quality distance learning and information exchange. This is a subtle change, but it is important to note and is more fully discussed in later sections and in the risk assessment subsection below (item e).

It is also noted that some of the project’s performance indicators used the term ‘trend’ (toward improvements) as a measure of successful performance. The ICR would argue that such a measurement is generally too imprecise to be used in as a performance indicator.

Risk Assessment. The overall risk of this project was rated as substantial. The ICR agrees with this assessment. The PAD also provided an adequate estimate of potential risks and specific plans for the remediation of those risks. At project completion, almost all of these identified risks have materialized as factors that negatively influenced the implementation of the project and quality of its outcomes. Unfortunately, the ICR also found that the mitigation strategies, although discussed in project documentation and stated in the PAD, were not adequately implemented. The reasons for the lack of mitigation are difficult to determine. Perhaps some of these risks could have been remediated if the project had not suffered from the long implementation delays. Examples of these stated risks include:

(a) Sustainability – The PAD identified the risk that DLCs may not reach a level of cost recovery that was agreed upon in the project design (risk rated as substantial). The ICR found that, to date, there has been little if any cost recovery by the projects DLCs. It is also noted that the project design included a financing arrangement whereby the DLCs would receive operational financing provided by the project on a declining basis leading to self-sustainability at the completion of the project. This arrangement was never implemented due, in part, to the long delays in the project.

Although these project DLCs have not demonstrated the potential for any acceptable level of self-sustainability as originally planned in the project, it should be noted that as university supported media centers/DLCs, it is highly likely that all centers will be sustained indefinitely as part of the academic programs of the four host universities. The universities consider these DLCs to be valuable units that add capacity and increased access to learning within the respective universities and as outreach vehicles for university instruction at a distance. This sustainability is certainly not to be confused with self-sustainability as originally intended in the project design. However, it is reasonable to assume that the project DLCs, in some form, will be fully supported (sustained) by the universities for the foreseeable future. It is also reasonable to assume that there is potential for some income generation to be derived now that the DLCs are

8

finally operational and from the project DLCs’ relationship with the INHERENT network, however, the original assumption that the GDLN would be sustainable as an independent cost recovery unit is yet to be proven anywhere in the world.

(b) Availability of quality relevant content – The PAD identified the risk that the DLCs have substantial capacity to provide services, but this capacity may not be used due, in part, to the lack of access to relevant distance learning content, particularly content in Bahasa Indonesia (risk rated as substantial). The ICR found that the project DLCs continue to have limited availability of quality distance learning materials suitable for effective program use through their facilities. This has been an ongoing concern of all GDLN centers in the region and is expected to remain so for the foreseeable future. No significant effort was made to mitigate this risk during the project period as the DLCs were in very limited, informal operation until ISG certification in January 2007.

(c) Borrower commitment - The PAD identified the risk that the Borrower may not be able to establish a strong commitment to train/inform civil servants via non-traditional methods, and willingness of Borrower institutions, private sector, donors and individuals to participate in and pay for GDLN program and course offerings (risk rated as moderate). The ICR found that compared to the significant capacity of the project DLCs, few civil servants have been trained in the DLCs to date, however, the Borrower does not consider this risk to be a significant concern once the DLCs become fully operational, with access to quality content that is relevant to the needs of civil servants. In the view of the ICR, substantial risk remains that non-university organizations, particularly in the service areas of the three project DLCs outside of Jakarta, may not be as willing to participate and pay for GDLN services.

(d) Equipment management – The PAD identified the risk that the Borrower might not be able to maintain reliable management of GDLN equipment (risk rated as moderate). The ICR found that the Borrower continues to have a significant concern about the quality of training provided to support equipment maintenance and management. As a result, the quality of expertise located at the four project sites is considered less than satisfactory. It will take time and additional training to bring long term improvements to this capacity. The ICR found that the project DLCs are using various approaches to make this happen and there is reason to assume that eventually the necessary level of technical training will be fully met. The ICR did find that the mitigation of this risk was clearly unsatisfactory during the project implementation period.

(e) Obsolescence of equipment – The PAD identified the risk that the procured equipment would become quickly obsolescent (risk rated as moderate). As previously mentioned the ICR found that the Borrower considers the installed equipment to be less than the latest available as required by PDO1. Findings from other GDLN DLCs in the EAP Region (see recent Collocated GDLN Site Study Report (2008) conducted by EAPVP) suggest that they too have strong concerns about the inadequacy of the current technologies used by Bank/GDLN in the region. This supports the credence that must be given to the Borrower’s concern. Of course it can be argued that the equipment was state-of-the-art when the project was designed, but then, since the risk was clearly

9

identified in the PAD, one must then question the quality of mitigation that was initiated to address this risk. Related to this finding, the ICR notes that that the planned technology review and update in the original Component 1 (Section 1.5 Component 2(d)) was never implemented.

(f) Equipment operation and maintenance – The PAD identified the risk that the Borrower might not be able to maintain effective equipment operation and maintenance (risk rated as moderate). The equipment installed in each DLC is sophisticated and requires high technology support to maintain. Untrained maintenance and operation of the installed equipment can easily damage the high technology equipment which will then require expensive repair and possible replacement. The Borrower has indicated this in its own ICR, and it was also a finding of the consultant report, that there was insufficient training in equipment maintenance and operation provided by ISG and the contractors, even though, in ISG’s case there was a written agreement that they would provide that training (see PSR1). Furthermore, it is noted that the available trained staffs at the DLC sites are currently few in number, and if any were to leave the project, they may be difficult to replace and to retrain. This is particularly true in the three DLCs located outside the Jakarta area. If shortages of trained staff become critical, the DLCs will find it difficult to operate. This issue is a substantial factor in the potential sustainability of this project. International experience with other GDLN DLCs supports this conclusion.

2.2 Implementation As per the Loan Agreement, the project was expected to be completed by March 31, 2006, with the loan closing date being September 30, 2006. In a letter to the Borrower, dated December 16, 2007, the Bank agreed to extend loan closing date to December 31, 2007.

The Directorate General of Higher Education (DGHE) was responsible for project oversight and coordination. A Central Project Implementation Unit (CPIU) within the DGHE in the Ministry of National Education (MoNE) which was implementing the Development of Undergraduate Education (DUE) and the Quality of Undergraduate Education (QUE) projects was assigned with the additional task of implementing this Project and the coordination of the GDLN center and GDLN sub-centers. The GDLN center at the University of Indonesia at Jakarta and the three GDLN sub-centers at Hassanuddin University in Makassar, University of Riau in Pekanbaru, and Udayana University in Denpasar, were responsible for the implementation of the project activities related to the center and the three sub-centers.

This project is rather straight forward in design and the amount of investment is not large (US$2.6 million) when compared to other Bank projects. The project primarily involves the procurement of goods and materials, provision of training, and the provision of operational costs throughout the duration of the project. The overall number of staff involved in the project is also fairly small. Even so, the ICR found that project implementation was unsatisfactory. The following highlights some of the reasons for this rating:

10

Procurement of goods and materials. As stated in Bank documentation (PAD, PSR1), there were two options to be considered in procuring the projects high technology equipment and pedagogical tools.

Option 1: The Bank (ISG) would put in place an arrangement, whereby one of the current World Bank vendors, Texel Corporation, currently providing a similar service at World Bank Headquarters, would act as the GDLN vendor providing a number, but not all of the required services. This vendor was cleared for this task by the Bank's Corporate Procurement Office, by conducting a market study to compare with other competitors both in terms of quality and price. Under this arrangement, implementation responsibilities for the DLC were to be shared by the GDLN vendor, the ISG, and local contractors.

The GDLN vendor would provide the following services: • Procurement to include bridge financing so that equipment can be purchased and staged before payment is made; • Staging to be done in combination with ISG staff; and • Shipping of all technology related GDLN equipment.

Under this option, the Bank's ISG would provide the following services: • In addition to Facility Guide, which contains general specifications for power, data, furniture, and physical layout, ISG would provide customized drawings to illustrate the seating arrangement of furniture, position of data and power outlets, and computer room layout; • A connection diagram illustrating how all the pedagogical tools are interconnected; • Certification of all installation work ensure that it was done to specifications provided; • Certification of DLC's connection to the GDLN; and • Training of local technicians for operation of all DLC Equipment.

The local contractor would provide the following services: • Data wiring, including installation of outlets on the Personal Computer side and Patch Panel Connections to Cisco Catalyst Switch. Test end to end connectivity to ensure a good connection for each outlet; • Install projectors in ceiling, screens on wall, Public Address System, and codec; • Install specified software on each PC, and test all of these to make sure they are working as specified; • Install any servers specified with appropriate OS and application software; and • Connect all of above as specified by connection diagram above

The first PSR indicated that it was envisaged that this option would take 2 - 3 months to implement. This included procurement of all equipment, staging at Bank, and shipping to Country.

Option 2: The GoI could pursue the procurement of all technology related GDLN Equipment through ICB. The difference between this option and that outlined above is

11

that the Ministry would have to select a vendor that was able to provide all the services that the Bank vendor is currently providing including the services that ISG staff would provide during the staging process, specifically (i) simulation and (ii) configuration. The reason for this is that it would be prohibitively expensive to fly ISG staff to the staging site being used by the vendor.

The stated risks to this approach were that the Country and/or Organization may not be able to find a local vendor that could perform all of the above mentioned requirements, at prices in-line with projected costs.

To address this concern, the Bank and the Ministry would also need to agree on a protocol to quality assurance prior to any interface with the Bank's systems. The Bank's ISG would assist the Country and/or Organization in developing a RFP to reflect the technical needs and specifications required to establish and operate the DLC. However, the vendor selected by the GoI would have a contractual obligation to the Borrower to meet the specification and pass the technical tests required to link to the GDLN. The ISG would not be a party to the agreement.

Procurement approach selected. From the early design stage, the Bank (ISG) strongly supported the use of option one above. ISG took the position that GDLN equipment was highly technical and sensitive and it must be calibrated to the same specifications as the rest of the Bank’s equipment. Therefore, only ISG had the capacity to mobilize and assure that the equipment was set up properly. From the Borrower’s perspective, they felt that they had to trust the Bank to provide good advice in this regard since they had little experience and knowledge concerning the equipment that would be installed. Also from the Borrower’s perspective, there was no other practical choice since, early on, the Bank made it clear that it would not enthusiastically support option two. For these reasons, the Borrower agreed to use option one above, even though it was uncomfortable with the decision, believing that the procurement process was outside the guidelines typically used for Bank projects in Indonesia. It should also be noted that this special procurement approach was reviewed and approved by the Bank’s vice president for ISG at that time.

In late 2002, the Bank did an “about face”, deciding that the above procurement option would no longer be supported by the Bank as it was outside the Bank’s stated guidelines for the procurement of goods and services. This change came as a result of internal Bank discussions wherein, the above mentioned procurement procedure again came under question. ISG, in early January 2003, issued a memorandum indicating that the procurement process would have to be changed to a third ‘recommended’ approach, wherein, the Borrower would directly contract with the vendor to provide the above mentioned services. ISG would continue to provide technical support to this process, but would no longer be involved with the actual contracting process as it had been before. Again, it was the impression of the Borrower’s project team that the Bank would not be willing to actively support any other option. Given the delays that had already occurred, the Borrower was anxious to move the project ahead as soon as possible, therefore, it agreed.

12

The revised procurement process for the technical equipment and pedagogical tools was carried out in 2003 and the preparation of the video conference rooms in the four project sites was completed during that same year. During this time, staff changes in ISG and the project occurred. This resulted in some additional delays in project implementation.

Lease line vs. ISDN. During the initial design, the use of a leased telecommunications line to support GDLN program transmission had been approved by the Borrower and the Bank. This approval was based on specifications and recommendations by ISG engineers. In late 2003, the Borrower team discovered that the lease line did not have the capacity to meet the technical requirements of GDLN connection at the four sites. A cost effective option had to found or the project would have to be abandoned. This finding resulted in a serious delay in the implementation of the project. In particular, it delayed the field staging of the telecommunications equipment at each of the four project sites.

It was later found that the type of leased line needed for GDLN connectivity was not readily available in Indonesia, and where it was available, the connectivity cost was beyond the budget allocated within the project. After another substantial delay, ISG finally agreed to support the use of an ISDN connection between Jakarta and Washington DC so the field staging of the equipment could take place. It should be noted that ISDN connections are more costly options for GDLN connectivity and, for these reasons; they are not generally sustainable for high volume international GDLN programming in the project DLCs. As it stands, the project could not afford large scale participation in international GDLN programming where the cost of the ISDN connection must be born by the project centers. The Borrower has expressed its concern about this cost element, and plans to abandon the use of ISDN connectivity (where possible) through the Bank’s Washington Network Operations Center (NOC) and instead incorporate the use of more cost effective IP technologies (Internet) currently available in most GDLN DLCs.

As the field staging (installation of equipment) began to take place, the technical team in four university sites prepared to receive the equipment and support the installation process. A schedule was developed and shared among the four sites so the work could be coordinated in timely manner. About this time, an accident occurred in Denpasar involving a key ISG engineer. This engineer was no longer able to support the implementation of the project. This caused additional delays in the project’s implementation.

Lack of sufficient training support. According to the implementation plan, the installation of equipment should have been followed by technical training in maintenance and operation of the project DLC equipment; however this training did not take place at a level that was considered effective. It is understood that this may have been partially due to the above mentioned accident. Recent interviews with the Borrower (see consultant report, Data and Findings from Local Interviews to Support Preparation of the Implementation Completion Report for GDLN LIL, 2008, Annex 9) found that the transfer of the equipment to the Borrower and the training in its operation did not happen in a timely fashion, and in some cases it did not happen at all, even though it had been previously agreed upon by ISG as part of the selected procurement process. The

13

Borrower was able to gain access to some training through the vendor providing the equipment.

Delayed ISG certification of equipment and connectivity (between Jakarta and Washington). Once the sites were prepared, equipped and DLC staffs initially trained to implement the DLCs, it was found that there were unresolved technical and administrative issues between Jakarta and Washington (Bank headquarters) relating to GDLN connectivity. These issues were not resolved until the last year of the extended project period. The GDLN sites in Indonesia were finally certified by ISG in January 2007, (note that the implementation plan had called for certification take place in May 2003). This certification was critical as it was an absolute requirement of ISG before project DLCs could connect with the Bank’s NOC in Washington, a prerequisite to receive and transmit via the GDLN. The delayed certification, until after the original project completion date, was main factor for the failure of the project overall since all other activities are contingent on the network being operational and certified.

Support for operations. The project was designed to provide loan financing for DLC operating costs on a declining basis (yearly) so that at project completion, the project would be able demonstrate self-sufficiency. This did not happen. Due to the delays identified, the operation of certified DLCs did not commence until the extension period, after the original project closing date, and operation was for less than twelve months. There was no opportunity to implement a strategy of annually declining operational funding (from loan funds), therefore, GDLN sustainability has not been demonstrated in this pilot project as required by the Loan Agreement and PDOs/performance indicators.

Drawback of local funding by host universities. The many delays also contributed to the management decision by the host universities to withhold counterpart funds originally agreed upon in the Loan Agreement. This drawback of local funding over several project years required the Borrower’s project team to cancel many of the planned GDLN activities within the project period. These cancellations affected the overall outcomes of the project.

Other factors. There were several other factors that also affected the overall implementation of this project. They include (a) replacement of key Bank staff during the project; (b) replacement of key Borrower project staff including some of the rectors in universities hosting the project sites; and (c) replacement of the Bank vendor (Textel) with a second (Ciber) which created substantial delays in procurement process which were outside the control of the Borrower.

2.3 Monitoring and Evaluation (M&E) Design, Implementation and Utilization Monitoring and evaluation was conducted by a team from the Central Project Implementation Unit (CPIU) and by assigned missions from the Bank to review the progress status of the project. Ongoing dialogue was also maintained by the Bank’s task team located in the WBCO, Jakarta. PSR/ISRs and other project monitoring documentation have been prepared according to established Bank procedures throughout the project’s implementation period. The Bank’s review missions have been adequately

14

staffed and even though the project has had many long delays, the mission reviews were for the most part, timely and candid. Most of the implementation problems were clearly identified and discussed in the PSRs and ISRs and associated aide memoires. Sector manager review suggested a number of project changes and interventions, but there seems to have been little follow-up action taken by Bank management to get real results at the operational level.

This project would have benefited from a more rigorous monitoring and evaluaion framework at entry, including the design of more precise quantitative and qualitative performance indicators, and improved risk review procedures. As stated above, the project would also have benefited from more rigorous oversight by Bank management. It is also noted that the PSR/ISR ratings were often far more positive than their own implementation narratives would seem to support. The ICR also finds it hard to understand why the project’s ultra-long implementation delays and well documented Borrower concerns were allowed to go on without strong intervention by Bank management.

2.4 Safeguard and Fiduciary Compliance The ICR found no safeguard variances to note, however there were two fiduciary issues identified during the 2006 financial review (see Financial Review of Global Development Learning Network Project: January 1, 2006 to December 31, 2006, ICR Annex 9). These include:

• Variances computed for expenditures budgeted vs. actual expenses incurred during the implemented period, were not adequately analyzed for identifying and documenting the reasons for the variances. Identifying and documenting material reasons for variances would have assisted the project to exercise better control over budgeted expenditures; and • Some financial reports prescribed by the Bank’s Disbursement Letter were not prepared by the Project, and inconsistencies were observed in the information that was reported in the various reports. The Project should have ensured that all reporting was provided in prescribed formats while preparing the FMRs for submission to the World Bank.

The ICR did find that: • Bank funds were used for the purposes intended; • Sufficient and accurate supporting documents were maintained by the project for the SOE disbursements; • Expenditures submitted on SOEs were eligible for financing under the loan; and • The Project had adequate accounting and internal control systems to support the accurate preparation of SOEs for withdrawal of loan proceeds

All financial audits were completed as required with no major comments or issues noted.

15

2.5 Post-completion Operation/Next Phase The potential for sustainability and further growth in GDLN services beyond project completion seems possible and perhaps promising. This finding is based on the fact that the GDLN project has joined (late 2007) with INHERENT, Indonesia’s national university communication network. Although this network is new and yet to be fully developed nationwide, it is growing and as stated above, there is every reason to assume that it will be come fully established in the near future. Once established, INHERENT will link the various universities into a network that can share information, instructional content and programming both within the campuses and at a distance. The ICR found that the collaboration between GDLN and INHERENT at the DLC level has been strongest in the University of Indonesia, Jakarta and Udayana University, Bali. The collaboration is still being established at the other two universities, UNHAS and UNRI. Required budgets for GDLN and INHERENT, including operating cost, connectivity, and maintenance have been included as part of the university budget at UI, UNUD, and UNHAS. UNRI has yet to be fully budgeted.

Although this development is certainly a positive step, the ICR would argue that the potential of this relationship has yet to be fully realized. The emerging INHERENT network provides the project DLCs with an obvious opportunity, but only the future will tell if this relationship will mature to the point where the project DLCs will become viable operations, with well trained staff, strong access to appropriate content and programming, good facility utilization and an acceptable level of sustainability.

3. Assessment of Outcomes

3.1 Relevance of Objectives, Design and Implementation The ICR found that the PDOs were relevant to the intent of the project concept discussions and the Loan Agreement.

The project design, PDOs, and performance indicators, were based on the assumption that the DLCs will attempt to reach some acceptable level of self sufficiency through a business plan approach resulting in income generation from the sale of services and products. This is a standard approach with other GDLN projects in the EAP region, however, in the case of Indonesia, the design seems less relevant. The ICR found that the host universities have always considered the project DLCs to support units to their academic departments, and that they intend to focus their programming of the project DLCs on academics (with the support of INHERENT, a university network) rather than on commercial programming as expected in the project design. The possibility for commercial programming is accepted, but not given priority in the university environment.

This understanding raises a question about the project design’s expectations for sustainability. With the above understanding, it seems reasonable to assume that the potential for any type of income generation, from outside sources, may be insufficient to

16

sustain the project DLCs. It is more likely that the project DLCs will be integrated into each university as a communications/ICT/media unit with connectivity through LANs and the INHERENT network to support the academic departments of each university and will, in the end, receive most, if not all, of their funding from those host universities, not from outside sources.

The project design also assumed that there would be a high utilization of international GDLN content as a primary input to the DLCs program offerings. This has not been the case and, if one understands the academic focus of these DLCs, it seems likely that international content may not be the primary content of these DLCs in the future.

There are also three other compelling reasons to support this assumption. Since almost all of the international GDLN programming is provided in English, the project DLCs may find that it is less useful than local programming which can be custom designed and provided in Bahasa Indonesian. This may be a particular concern in the three DLCs located outside the Jarkarta area. In addition, local program content may be obtained at much lower costs from other academic institutions associated with the INHERENT network. International content may only be used as it relates to the universities academic needs and affordability. In addition, since most of the current GDLN content is not course based, but instead is comprised of short, non-credit seminars and workshops, the project DLCs may find that they do not have a wide audience for such programming within their host institutions and outside service areas.

3.2 Achievement of Project Development Objectives It has been stated that during project implementation, there were significant problems with equipment procurement, installation, commissioning and certification; coupled with difficulties in securing sufficient bandwidth. These problems have combined to cause profound delays in project implementation. There are obvious shortcomings in the capacity of these DLCs. All of the projects DLCs have a minimum number of highly qualified and trained staff. They also currently have low facility utilization rates, limited income generation, and a lack of available content to support full facility utilization. Clearly, from the perspective of the Loan Agreement, the PDO’s were not fully achieved within the project implementation period. This failure to achieve expected outcomes within the project period can be attributed mainly to the failure to achieve performance indicator 1, since without a certified operational network most of the other activities could not take place:

Performance Indicator 1: Distance Learning Center GDLN certification will be completed successfully by May 2003.

• The ICR found that certification was completed in January 2007, in the extension period and two years after the original scheduled project completion date.

Performance Indicator 2: Business plans reviewed bi-annually and modified to reflect progress and newly identified opportunities by May, of each year.

17

• Since the network was not operational until January 2007, the business plans were not prepared until May 2007, when a partial plan was made available. After project completion, a full business plan was prepared. There were no significant annual reviews of these plans during the project period,

Performance Indicator 3: Financial management systems were put in operation and effective by May 2003.

• ICR found that financial management systems were put into place on or before, May 3, 2003, however there were some shortcomings as noted in Section 2.4.

Performance Indicator 4: Monitoring and assessment systems were implemented producing quality input to bi-annual business plan reviews and Project management reviews by May of each year beginning 2004.

• The ICR found that monitoring and evaluation activities were conducted, however the baseline data and original review criteria was less than satisfactory. Furthermore, the performance indicators did not provide an adequate framework for effectively monitoring the progress of the project. In some of these indicators, the targets were either too imprecise to measure (i.e., PI5 and PI6 which use terms such as “trends”), or were somewhat irrelevant to the expected outcomes of the project (i.e., PI 6 which talks about the project DLCs becoming ‘self-sustaining’ although they are established as units of their host universities). In the PSRs and ISRs, issues were noted and discussed, but there was a marked absence of effective mitigation by Bank and Borrower project teams and management.

Performance Indicator 5: There is evidence of a trend that reflects an increase in the number of development focused GDLN learning products (including: courses; training sessions; seminars; debates; awareness campaigns; and national, global and local dialogues on development issues) subscribed to by public and private professionals. Targets included a planned service output of 1,000 training days in 2004, 2,500 training days in 2005, and 3,225 training days in 2006.

Although difficult to estimate due to inadequate data collection, estimates suggest that there were less than 1000 total training days (about 133 three-hour programs with an average of 20 participants in each program) provided during the entire project implementation period. Optimistic estimates suggest that only about 15% of planned performance has been realized during the project. The ICR has found Borrower estimates with even lower levels of estimated productivity (about 250 training days delivered). After the project close, the utilization of the GDLN-INHERENT network has continued to expand.

It was reported that, in the first quarter of 2008, there were 36 video conferences involving 852 connections to higher education institutions, (1720 connection hours), with an average of 24 locations connected in each video conference. Video conferences were conducted not only via in-country nodes but also international connections through the

18

GDLN gateway at the University of Indonesia, to the United States of America, Australia, and Thailand. The ICR notes that these large numbers likely reflect the overall INHERENT operation, not just the project DLCs, which do not have the technical or pedagogical capacity to support the stated level of connectivity.

Performance Indicator 6: There is evidence of a trend that reflects a clear progression towards Distance Learning Centers becoming self-sustaining.

As a method to demonstrate a trend toward self-sufficiency, a decreasing operational budget was agreed to by the host universities as part of the project design. Documentation of this agreement may be found in the Letter of Commitment as follows:

……Each university commits to allocate its self-generated fund (DIKS or DRK) for the GLDN operation (within their university environment) for 3 years beginning in 2002. The committed fund will be allocated at 0%, 25%, and 50% of the connectivity cost for the first, second, and third year of implementation. In addition, personnel expenditures will be supported by this fund according to the figures proposed in the business plan submitted by each university. The aforementioned fund will be transferred to the Center/Sub-Centers special account prior to each fiscal year contract………

This statement of agreement was issued in May 2002, by each university, but it was not renewed in subsequent years. From the universities’ perspective, the project DLCs were not performing according to the plan and hence the allocated budget was not justified. The withholding of these funds caused many operational difficulties in all of the project sites.

• There were no program offerings in 2004 or 2005 and only some local programs in 2006. While this is understandable given the context that the network was not certified to operate until January 2007, the ICR finds that there is no trend toward self-sustainability other than increased access to funding from host universities during the last year and after project completion. It is understood that some might want to argue that the fact that the project DLCs were included in the university budgets means that they are sustainable. If this argument is accepted, the ICR would question why the performance indicator and the plan for a decreasing allocation of operational funds were included in the project design.

Performance Indicator 7: Quality and relevance of courses offered by the GDLN is evaluated satisfactory or better by participants – 75% of participants in 2004, 80% of participants in 2005, and 85% of participants in 2006.

Since the network was not operational until January 2007, ICR found that that there were no program offerings in 2004 or 2005. In 2006 there were some local programs offered through DLC and INHERENT linkeage but the ICR found no record of participant evaluations.

19

Performance Indicator 8: Content provided by national and international partners will amount to 45% of total in 2004, 55% of total in 2005, and 65% of total in 2006.

• ICR found that no significant content was provided in 2004 or 2005 from any source, and in 2006, the project DLCs were involved in only a small effort with connections to three countries. Most content during this period was provided through national sources if at all, since there was no authorized connectivity with GDLN until the sites were certified in January 2007. Since the certification date, project DLCs have been involved in programs with Australia, Sri Lanka, USA, Japan, Philippines, Vietnam, Thailand, Timor Leste, Turkey, Pakistan and Nepal. Although there is a positive trend noted above, currently there is less than 20% facility utilization rate on the average in these project DLCs.

Performance Indicator 9: The project DLC’s facility utilization rates will increase to 30% in 2004, 45% in 2005, and 55% in 2006.

• Given that the network was not certified until January 2007, the facilities could not be formally utilized as reflected in the average facility utilization rate in 2004 and 2005 of less than 5%, and less than 10% in 2006. Twelve months after certification the utilization rate had doubled to around 20%, and, as the project DLCs become more integrated with the INHERENT network and gain more experience with the GDLN, there is reason to assume that their utilization rates will increase. However, regional experience suggests that it may be difficult for the project DLCs to increase their facility utilization rates above 50% on the average. Most GDLN DLCs in the region continue to have even lower average facility utilization rates.

3.3 Efficiency Baseline and comparative data was not collected or available on this pilot project. The unique differences between the Indonesia GDLN pilot and other existing GDLN DLCs made cost comparisons unproductive as well. For this reason no quantitative analysis of efficiency was attempted.

3.4 Justification of Overall Outcome Rating Rating: Unsatisfactory

When the PDOs and the associated performance indicators are reviewed to determine the level of project success, it can be said that the four project DLCs were established and were marginally operational at project completion, but there was little time to demonstrate the viability of those DLCs in any of the ways required by the Loan Agreement/PAD. If one looks at the performance indicators, it is evident that most were not met as required by the Loan Agreement/PAD. Some were totally abandoned, while others were accomplished with delays.

Regarding overall project outcomes, the project was designed as pilot that would demonstrate the use of the GDLN as a viable tool to support development priorities in Indonesia, however, the long delays and other issues noted in the ICR suggest that the

20

original project design cannot be considered viable as a future model for replication. As seen in the findings of this ICR, the procurement method and technology selected by the Bank was very much below optimal. The GOI has learned from these lessons and procured equipment competitively from the market and was able to select more up to date technologies. Experience has also been gained through the assistance of the GDLN which assisted GOI with the design of their own distance learning network through video-conferencing and internet, expanding access to information and sharing information both nationally and internationally.

Regarding quality at entry. As stated above, the project had significant design flaws that compromised the quality of project outcomes. Most notably, the design required the use of a commercial approach whereby self-sustainability would be gained from the use of well structured business plans, annual reviews of those plans, and significant revenues earned from non-institutional sources. In the project these plans and reviews were intended to provide baseline data for M&E. Unfortunately, the business plans for the project DLCs were not fully prepared until during and after the last year of operation, thus they were not reviewed during the project implementation period. Furthermore, when one looks at the priorities of the project’s host universities and their commitment to INHERENT, it seems obvious that the host universities intend to use the four project DLCs as support units for academic programming within and between their respective universities. This is understandable, but if they are successful in that effort, they may not have the capacity to support a significant amount of income-generating, commercial programming from outside sources. In addition, when one understands that these project DLCs currently have large participant bases of non-English speakers and that GDLN programming is currently provided primarily in English, these views are further reinforced.

The project design also featured an element whereby operational costs would be reduced on a yearly basis to demonstrate the project’s potential for sustainability. In this activity, it was assumed that income generation, from outside commercial sources, would, on an increasing basis, make up for the withheld loan funds. The ICR found that this design element was never implemented and that the full operational costs of these DLCs were integrated into the budgets of the host universities. This further reinforces the non- commercial argument presented in the previous paragraphs.

Regarding project preparation, the ICR found that the project teams did not fully assess and address the technical procurement approach that was needed in the project. The Borrower expressed concerns about the selected approach, but was given only one other seemingly impractical choice (to use traditional Bank procedures without sufficient support from ISG). These concerns were well documented by the Bank’s task manager and through the early PSRs.

Within the next year ISG again changed course and required that the Borrower accept a different procurement approach. Even after the second procurement approach was implemented, long delays and lack of follow through by the venders and ISG still plagued the project (i.e. lack of technical training provision and equipment certification).

21

These delays also forced long disbursement delays and the virtual collapse of the project’s overall implementation plan, with many performance indicator targets being missed or abandoned. Although the DLCs were eventually established and equipped, the equipment and connectivity to GDLN via Washington were not certified for operation by ISG until January 2007. This can be contrasted with the original project implementation plan wherein the target for certification was May 2003.

Furthermore, the critical plan to connect Jakarta and Washington by lease line was not properly researched by the Bank’s TA team, ISG, resulting in the provision of poor advice given to the Borrower and additional project delays. This resulted in more uncomfortable relations between the Borrower and the Bank. The Borrower was, in the end, forced to accept the use of an ISDN connection which was never intended in the project design as it is prohibitive in cost (outside the budget) and may, in the end, limit the amount of GDLN programming that is brought into the project DLCs from international sources. As a response to concern, the Borrower has decided to use IP technologies for future connections with INHERENT and its GDLN partners. This will give the project DLCs the capability of transmitting at substantially higher frequency (2-4 mps vs. 360 kps.) which provides better picture quality, and to skip future requirements to connect through Washington (Bank NOC) for its GDLN connectivity.

It must be said that all of these concerns relate to the project’s overall failure to reach many of the expected project outcomes and, as a pilot project, to demonstrate the viability the GDLN as a tool to support the development agenda as noted in the Bank’s CAS. For these reasons, the ICR rated the project’s quality at entry to be unsatisfactory.

Regarding project implementation, there were profound delays in project implementation and disbursement. Counterpart funds were withheld because the host universities were understandably frustrated with the slow implementation progress. In the end, the quality of the equipment procured for the project DLCs was questioned by the Borrower which stated that it did not consider the equipment to be the latest available and state-of-the-art, a fundamental requirement in PDO1.

The ICR did find that most of the project’s implementation issues were adequately documented in the PSRs, ISRs, and Aid Memoires left by the Supervision Missions. The long delays without Bank intervention also suggest that monitoring and evaluation was not effectively implemented by the Bank or the Borrower. The ICR also found that risk mitigation was not effectively implemented, even though Bank documentation clearly describes and accurately estimates those risks. Finally Bank and Borrower management did not aggressively move to address these problems in a timely fashion. Some of the delays were extraordinarily long (years). The ICR rates project implementation to be unsatisfactory.

3.5 Overarching Themes, Other Outcomes and Impacts The project DLCs have affiliated with the INHERENT network and are able to use both the INHERENT network and, with the limitations stated above, the GDLN to support their future programs. Although the current level of staff training is considered less than

22

satisfactory, future operations should allow for the provision of training and the use of new and lower cost technologies such as IP. As INHERENT grows in Indonesia, there is reason to assume that the project DLCs will have the opportunity to become more integrated with the other universities in the nation. This is a major goal of the host universities and must be seen as a positive outcome of the project. If the combined project DLCs/INHERENT network morph into the GDLN Indonesia Network, as has recently been suggested, there will be more potential that other universities can effectively use GDLN programming to support development in Indonesia. None of these potential outcomes were planned in the project, but they must be seen as potential unexpected outcomes that may help negate the obvious shortcomings of the project as originally agreed upon in the Loan Agreement.

(a) Poverty Impacts, Gender Aspects, and Social Development The potential programming provided through the four project DLCs may have an impact on poverty, gender and social development. This potential can be seen in recent GDLN programming in the EAP region where, AIDS, avian flu, malaria, forest governance, youth governance, and other development topics have been featured within the past year. In addition, with the project DLCs’ recent affiliation with the INHERENT network, it should have access to other forms of development programming offered in local languages and with a specific focus on the current issues of civil society in Indonesia. The actual impact that these program opportunities have is yet to be demonstrated by any of the project DLCs. To date, there have been only a few related programs offered to participants outside the universities.

(b) Institutional Change/Strengthening The introduction of the project DLCs within their host universities provides them with a powerful tool and significant potential to support future distance learning and outreach programming. It gives those universities access to other national and international sources of learning and information and it provides the potential for more interaction by the university with local NGOs, businesses and industries. However, the ICR notes that the use to which these DLCs will be put is yet to be fully determined or demonstrated. If the DLC’s technology is not up-to-date, maintained and/or upgrades as required, and if the DLC staffs are not well trained and managed, the DLCs will quickly lose their potential for institutional change/strengthening.

(c) Other Unintended Outcomes and Impacts (positive or negative) There were significant unintended outcomes. The GOI was able to draw lessons from the Bank funded project which assisted their development of the INHERENT network covering 151 centers throughout Indonesia. With this network in combination with GDLN as its global gateway, there is high likelihood of achieving the original PDO of (i) expanding the access to information on development, through the strengthening of the capacity of decision makers in the public and private sectors and the improvement of regional dialogue on policy-related issues, and (ii) providing the latest communication technology for quality distance learning and information exchange between institutions in Indonesia and regional and global network distance learning networks. It should be noted that the PDO is likely to be achieved 4 to 5 years after the completion of the Bank project.

23

3.6 Summary of Findings of Beneficiary Survey and/or Stakeholder Workshops A stakeholder workshop was held in March 2001, during early project preparation. Apparently the workshop was held to allow more comprehensive input into the development of business models for each project DLC. Based on the questionnaires distributed to the participants, the following market was identified as promising GDLN facilities users: a) University: regular students as well as part time employed students; b) Industries: oil and mining, manufacturing, renewable energy, financial industries, SMEs, etc. c) Businesses: stock exchange, banking, tourism, etc. d) Government: all ministries and agencies at the central government, and the regional governments; e) Non governmental organizations.

There were no other significant findings from this workshop.

4. Assessment of Risk to Development Outcome Rating: Substantial

The PAD presented a satisfactory assessment of risk; however the mitigation strategies, although reasonable in their presentation, were found to be, for the most part, unsatisfactory in their implementation (Section 2.1). The mitigation strategies were not fully implemented and the result was that many of the stated risks that have become project realities. The project delays certainly contributed to these shortcomings, but the stated risks should have been more effectively addressed early on as they became apparent problems within the project, and even later when there were opportunities to assess the projects progress and to adjust the project design and implementation plan.

Of particular importance, the project was designed as a pilot to demonstrate the viability of the GDLN as a tool to support the development priorities of the nation. The wrong application of the procurement method and selection of the technology has led to the unsatisfactory performance of the project during the project period. However, the GOI has learned these lessons and used competitive procurement methods and appropriate technology for its own funded INHERENT network to demonstrate the viability of using VCs and internet as a tool for expanding access to information national as well as globally for national and international stakeholders.

5. Assessment of Bank and Borrower Performance

5.1 Bank Performance (a) Bank Performance in Ensuring Quality at Entry Rating: unsatisfactory

The Bank’s performance must be rated as less than satisfactory during the lending phase of the project. After the Loan Agreement was signed, it is evident that the Bank team did

24

not reach a full consensus with the Borrower regarding the procurement process and the technical specifications for the equipment required in the project. This lack of consensus was borne out in later discussions, and was documented by PSR/ISRs and later Borrower documentation. These concerns proved to be a major factor in subsequent delays, discomfort in later Bank-Borrower discussions, and a required extension of the closing date.

The ICR also must add that Bank management did not provide sufficient oversight to the design and preparation of this project. The Bank’s management should have stepped in to make sure that the Borrower understood and was convinced that the proposed procurement process would work and that it was acceptable under both the Bank’s and Borrower’s procurement guidelines. The fact that the process was later abandoned by the Bank for those stated reasons seems to supports the Borrower’s original concerns.

The Bank took on the role of providing TA to support the design of the project DLCs. This was done in most if not all of the other DLCs in EAP. In this project the Bank did not provide accurate and professional technical advice regarding the selection and use of the leased tie line from Jakarta and Washington, a key technical requirement for the operation of the project DLCs. This inaccurate information resulted in the Borrower having to abandon the use of the tie line approach because it was found to be both too low in capacity to serve the needs of the project; and was not available to all of the project DLCs. The Borrower then had to accept the use of the ISDN connection approach, a cost prohibitive solution that was outside the project’s approved budgets. After project completion, the Borrower has used its own funds to more fully equip its DLCs to use lower cost IP technology as an alternative to the use of the ISDN technology.

It is also noted that, the project design was weak in a number of areas and based on key assumptions that proved to be irrelevant to the needs of this project. In addition, the performance indicators were not well prepared to effectively monitor the performance of the project and there was no attempt by the Bank to retrofit the design to make them more appropriate. (Sections 3.1 and 3.4).

During project implementation, the Bank also failed to provide all of the agreed upon training for operation and maintenance of the technical equipment. As a recourse to this situation, some additional training was secured from the Borrower’s equipment vender.

(b) Quality of Supervision Rating: Moderately Satisfactory

Fiduciary and safeguards policies were effectively monitored and evaluated during project supervision. There were 10 formal supervision activities and their findings are recorded in PSRs or ISRs accordingly. It is recognized that this project was difficult to supervise. The long delays and lack of progress over time made it difficult to maintain a timely review process. A review of the early PSRs reveal that the project supervision teams were generally well staffed and were fully aware of the risks associated with this project and the issues and shortcomings associated with the project’s implementation.

25

The early PSRs also documented the important concerns of the Borrower regarding the viability of the project implementation plan/design. These project documents encouraged substantial changes in the project design and implementation plan (see PSR 1), however, no formal changes were ever made. Instead the supervision teams continued to try to adjust the performance indicators and implementation plan to bring the project to full completion. In doing so, however these teams did not effectively address the major issues that that had been documented so well in the PSR/ISRs. Management should have stepped in early to change the project design in 2003 and aggressively addressed the reasons for the long delays and poor implementation progress. By 2005, the equipment had been procured and installed, there was little that could be done except to pressure ISG to install and certify the center to become operational. At the same time, the task team made significant efforts to develop linkages so that the PDO may be reached through a combined GDLN and GOI-funded INHERENT approach.

The ICR also found that the supervision teams made a significant effort to support the full implementation of the project through extensive dialogue with the Borrower. It was also found that the ratings given within the PSR/ISRs were inconsistent and probably too high for the reality of the project’s implementation. This situation was also noted by the project team leader in the later ISR6.

(c) Justification of Rating for Overall Bank Performance Rating: Unsatisfactory

The project design and preparation was found to be less than satisfactory for a number of reasons stated above. This finding is reinforced by the Borrower which also considers it to be so (see Borrower’s comments below).

During project implementation, the Bank’s supervision team did recognize the primary issues and shortcomings of the project and documented those concerns in the PSRs and ISRs. Even so, the ICR found that Bank management and project team follow-up was not quick to respond, or sufficient in action, to adequately address the identified issues. This resulted in limited changes in the project approach and delays that went far beyond what might be considered acceptable and typical in any Bank project, much less in a small and relatively less complex project such as this one. As a result, there was limited disbursement during the first four years of the project and the PDOs and performance indicators/targets were not substantially met. Most concerning is the finding that bad advice and slow delivery of agreed upon services by ISG may have been at least partially responsible for some of the incurred project implementation delays.

For these and the other reasons documented in this ICR, it was found that the Bank’s project teams from design to implementation should bear significant responsibility for the failures of this project. From this viewpoint, the ICR rated the overall performance of the Bank as unsatisfactory.

5.2 Borrower Performance (a) Borrower Performance

26

Rating: Moderately Satisfactory

The Borrower, early on, expressed concerns about the project design and implementation plan but found that there were few, if any practical options to the plan presented by the Bank’s technical team. It reluctantly accepted the Bank’s recommended procurement, staging and simulation approach. Later, the approach was abandoned for second approach which, again, failed to serve the project well. During this and later project periods, the PPIU actively sought to find ways to work around its problems with the procurement approach. The PPIU demonstrated flexibility and the ability continue to develop the project even when resources were not adequate and Bank and university actions negatively affected project implementation (i.e., lack of follow-through on training, bad advice regarding the use of the tie-line, the necessity to agree to use ISDN connections, extraordinary delays in equipment and connection certification, etc.) In the end, the PPIU was finally able to establish and staff operational DLCs at all project sites, and was able to obtain some training though its vender. Even though the sites do not yet meet the expected outcomes of the project, they do stand as an accomplishment, and the Borrower should be commended for that achievement in light the long delays and problems incurred during project implementation. Also, after project completion, the Borrower has continued to improve the potential of these project DLCs in ways that are certainly positive. The merging of the project DLCs with the INHERENT network is an example of this continued effort to make the project DLCs viable within the host universities. .

The host universities did withhold substantial counterpart funds that were agreed upon in the Loan Agreement, but there are extenuating circumstances in that the project had little disbursement for almost four years and there was a growing concern by these universities that the project might fail. Therefore, they chose to withhold some funding to reduce the financial risk to their institutions. Once the project implementation began to pick back up (2006 through 2007), the Borrower began to flow counterpart funds to the project again. It was also found that in 2003 the Borrower made an effort to adhere to the project requirement that operational funds be provided on a decreasing basis (annually) so that the project would be able to demonstrate the potential for self-sustainability from resources earned from outside sources. During the subsequent years this effort was abandoned. It is assumed that this came as a result of the project delays and lack of disbursement; the host universities decided that this element of the project was not implementable. Some of the host universities also witnessed a change of rectors. These senior staff changes resulted in some loss of direction as the PPIU made adjustments to support new management priorities and styles. The ICR also found that the Borrower’s project teams suffered somewhat from staff changes during project implementation and a severe lack of technical and GDLN/DLC management training.

For the most part, project documentation and adherence to the safeguard and fiduciary requirements of the project were satisfactory. In addition, the ICR found that the Borrower actively participated in M&E and was responsive to Bank concerns to the extent possible considering the state of project implementation. The ICR also found that

27

many of the negative factors affecting this project were beyond the control of the PPIU and the Borrower.

The ICR did note that the Borrower should have formally (and more forcefully) raised its specific and general concerns about the poor implementation progress of the project with senior Bank management. If that had been done in a timely manner, those actions might have stimulated more action from Bank management. This may have resulted in more effective project implementation and better overall outcomes. It is the opinion of the ICR that this project was an early on candidate for a redesign, reappraisal, or cancellation.

Due to the Bank’s lack of performance in this project and the finding that many of the factors that negatively impacted the project were beyond the control of the Borrower’s implementation team, the ICR found that Borrower performance was Moderately Satisfactory.

(b) Implementing Agency or Agencies Performance Rating: N/A

(c) Justification of Rating for Overall Borrower Performance Rating: N/A

6. Lessons Learned

• The Bank should put clients in the driver’s seat when designing projects. Client’s concerns were obviously unheeded during the design stage which led to the subsequent mismatch between technology chosen by the ISG and country’s technology environment. The idea of one-size-fits-all for the entire world, even for technology, did not work. • The Bank should recognize that when it enters exceptional relationships (outside typical Bank/client operations) with clients, special care should be taken to fully document all agreements with the client, insure full consensus of those agreements, insure absolute control over the quality of service delivery, and the full provision of the agreed upon services. Under such circumstances, the Bank takes on more responsibility for direct project implementation and for this reason; it must bear more responsibility for failures that may occur during that implementation process. • In projects such as this, Bank management should closely monitor the implementation of the project and should move as quickly as possible to re-structure and re-design the project when it was evident that the original design did not work. Multi-year delays should not be considered acceptable and if necessary, such delays should be reason enough to call for a redesign, re-appraisal, or cancellation of the project; • University based GDLN DLCs have different organizational philosophies and needs than more commercially focused DLCs in NGOs or WBCOs. Thus, the project designs to support the establishment of such DLCs should be adapted to reflect these differences. The self sustainability and revenue generation may or may not be a priority in GDLN affiliated DLCs that are university based.

28

• Consensus building is critical to effective project design, preparation and implementation; • In all GDLN projects, training and content development/acquisition is critical to project success; • Critical elements in GDLN operation, such as solutions for connectivity between Washington and the DLC require special care and consideration, including its practical application, cost, and availability; • For reasons of cost, GDLN should plan to distribute most if not all of its future programming through IP connectivity if available; • When host universities change rectors or other key decision-makers, delays can be expected since university objectives and programs may be reviewed and priorities adjusted to reflect the new administrations views; • There are advantages in cost effectiveness and efficiency when several similar projects are funded concurrently, with similar PDOs and performance indicators, and managed as a single program rather than as separate projects. This project is the only one in EAP where this was done; • The establishment of a strongly staffed, and well managed central project coordination unit (CPCU) must be considered a key contributing factor to the successful implementation of any project with widely disbursed operations; • If there are major procurement packages that might delay project implementation, it may be better to complete the procurement before loan effectiveness (using the governments own budget) or early during project effectiveness. This will require as much preparation as possible to be done during project appraisal (e.g. obtaining clearance for ToR, shortlist etc); • The use of Bank provided technical services including the prescription of technical equipment to be procured, methods of procurement, and training may be problematic in some project applications since the Bank’s direct provision of such services may not be in-line with previously established and agreed upon Bank-Borrower guidelines. Such inconsistencies in Bank policies and actions may lead to confusion and conflict between the Bank and Borrower. • The joining of national and international networks can strengthen the potential sustainability of individual distance learning centers and provide increased access to programming.

7. Comments on Issues Raised by Borrower/Implementing Agencies/Partners (a) Borrower/implementing agencies ICR comments on issues raised by the Borrower were provided in the above sections as each issue was discussed.

(b) Cofinanciers N/A

(c) Other partners and stakeholders N/A

29

Annex 1. Project Costs and Financing

(a) Project Cost by Component (in USD Million equivalent) Appraisal Actual/Latest Percentage of Components Estimate (USD Estimate (USD) Appraisal millions) ESTABLISHING FOUR 1.30 583,549.10 45% GDLN DLCS SUPPORT FOR OPERATION OF GDLN 1.32 850,781.80 64% DLCS

Total Baseline Cost Physical Contingencies Price Contingencies Total Project Costs 2.62 1,434,330.90 55% Front-end fee PPF Front-end fee IBRD 0.03 26,600.00 89% Total Financing Required 2.65 1,460,930.60 55%

(b) Financing Appraisal Actual/Late Type of Estimate Percentage Source of Funds st Estimate Cofinancing (USD of Appraisal (USD) millions) Borrower 0.98 161,346.00 16% International Bank for 2.66 1,460,930.60 55% Reconstruction and Development

30

Annex 2. Outputs by Component

Component 1: The original project closing date was December 31, 2006; at that time none of the project DLCs were certified for GDLN operation by ISG. This happened on January 9, 2007, when ISG finally certified the equipment and connectivity for GDLN operation. This certification enabled full GDLN connectivity and DLC operation.

At project completion the project DLCs had a facility utilization rate of about 20% on the average. The DLCs were able to operate for less than one year before project completion. During that time, it is estimated that less than 300 person days of training services were actually provided. This measure of productivity is far below the project duration target of 6,725 person days of training. The Borrower does have expectations that the utilization rates will increase for all project DLCs. This is due to the previously mentioned merger of the project DLCs with the university network in Indonesia (INHERENT). The extent to which the project DLCs will be active in the international GDLN has yet to be demonstrated, however.

Component 2: The funding for the operational costs of the DLCs and the PPIU were included under Component 2. The funds were not fully disbursed due to the long delays in the project and the limited duration of operations that the project DLCs were able to provide during the lifetime of the project. See Annex 1(a) for the final disbursement estimates for this Component.

31

Annex 3. Economic and Financial Analysis (including assumptions in the analysis)

Due to the lack of baseline data and the type and size of this project, no economic and financial analysis was undertaken.

32

Annex 4. Bank Lending and Implementation Support/Supervision Processes

(a) Task Team members Responsibility/ Names Title Unit Specialty Lending Task Team Leader/Senior Jerry G. Strudwick EASHD Task Team Leader Educator Tech. Assistance

Support/Procurement/

Installation/Staging/ Various ISG specialists Simulation/ Under the management of Certification/ Le Ann Vu Manager, ISG ISGVP Technical Training/ Supervision/ICR Novira Kusdarti Asra Financial Management Specialist EAPCO Faustinus Marius Ravindra E T Consultant EAPVP Corea William Hardi Consultant EAPCO Ratna Kesuma Operations Officer EASHD Colin David Lonergan GDLN Coordinator EAPVP Maria Ivia Martinez Consultant EAPVP Rimta K. Silangit Liaison Officer EAPVP Jerry G. Strudwick Sr Education Specialist HDNED Sonya Woo GDLN Coordinator EAPVP

(b) Staff Time and Cost Staff Time and Cost (Bank Budget Only) Stage of Project Cycle USD Thousands No. of staff weeks (including travel and consultant costs) Lending FY01 2 7.97 FY02 16 53.25 FY03 1 4.51 FY04 0.00 FY05 0.00 FY06 0.00 FY07 0.00 FY08 0.00 Total: 19 65.73

33

Supervision/ICR FY01 0.00 FY02 0.00 FY03 6 28.52 FY04 6 25.90 FY05 7 52.03 FY06 17 48.84 FY07 15 32.60 FY08 13 49.06

Total: 64 236.95

34

Annex 5. Beneficiary Survey Results None available

35

Annex 6. Stakeholder Workshop Report and Results The ICR found no workshop report, minutes or statement of results for the early stakeholder workshop held.

36

Annex 7. Summary of Borrower's ICR and/or Comments on Draft ICR The Borrower provided a full ICR, but did not provide a summary for inclusion into this Annex. The full version is documented in Annex 9 and is available in the project files.

In reviewing the Borrower’s ICR, the ICR found that the Borrower supported many of the findings of this ICR, namely the stated concerns of the Borrower, the reasons for poor project design, the reasons for poor project implementation and the overall shortcomings of the project. However, the Borrower’s ICR did rate a number of project elements from moderately satisfactory to fully satisfactory. The Borrower agrees that the project suffered from long delays and poor implementation, and that it did not meet the expected outcomes stated in the PDOs, but argues that the fact that project DLCs have been integrated into the INHERENT network demonstrates that the project DLCs are now successful and viable. This is not the view of the ICR which allows for the possibility that, in the future, the success and viability of the project DLCs may be demonstrated, but would argue that to date, the project has not been successful in demonstrating that viability.

37

Annex 8. Comments of Cofinanciers and Other Partners/Stakeholders N/A

38

Annex 9. List of Supporting Documents • PSR/ISRs 1 through 10 • Aide Memoires (Mission) • GDLN LIL PAD • GDLN LIL Loan Agreement • Loan Extension Agreement • Report – Financial Management Review, November 29, 2007 • ICR Consultant Report, January 2008 • Government ICR – April 7, 2008

39 95° 100° 105° 110° 115° 120° 125° This map was produced by the Map Design Unit of The INDONESIA World Bank. The boundaries, colors, denominations and GLOBAL DEVELOPMENT any other information shown on this map do not imply, on the part of The World Bank LEARNING NETWORK Group, any judgment on the legal status of any territory, PROJECT PROVINCES 15° or any endorsement or acceptance of such SELECTED CITIES AND TOWNS boundaries. INDONESIA PROVINCE CAPITALS NATIONAL CAPITAL MYANMAR MAIN ROADS VIETNAM PHILIPPINES RAILROADS PROVINCE BOUNDARIES 10° 10° INTERNATIONAL BOUNDARIES THAILAND Sulu Sea 135° 140°

Banda Aceh L A Y 5° A BRUNEI 5° 1 S Natuna Talaud Medan M Besar I Celebes Is. Tarakan PACIFIC OCEAN Pematangsiantar A Sea Simeulue 24 19 Morotai 2 SINGAPORE 23 Manado Nias Tanjungpinang Pekanbaru 25 Ternate Halmahera Waigeo 0° 3 Pontianak KALIMANTAN 26 Gorontalo 0° M Lingga 20 Samarinda Manokwari e Padang 30 Biak n Palu Sorong t 4 Balikpapan Peleng Obi a Siberut 5 Jambi Bangka 21 w Misool Yapen a Pangkalpinang Palangkaraya SULAWESI Sula Is. 32 Jayapura i SUMATERA Mamuju Ceram 9 Belitung 22 Fakfak IRIAN JAYA I s Palembang 27 Amahai . 6 28 7 Bandjarmasin 29 Kendari Buru 33 (PAPUA) Bengkulu Ambon Parepare Timika Puncak Jaya 8 (5030 m) 5° Makassar Muna Kai Enggano Bandar 11 Java Sea Baubau Is. Lampung Banda 31 JAKARTA Aru 0 200 400 Kilometers Serang Sea Is. 12Bandung Semarang Madura 10 13 Wetar PAPUA Surabaya Babar 0 100 200 300 400 Miles 16 Tanimbar JAWA Yogyakarta Lombok Sumbawa Alor Moa Is. 15 Bali Raba Flores NEW GUINEA 14 Merauke NEW GUINEA 95° 100° 105° Denpasar Mataram 18 Ende TIMOR-LESTE Arafura Sea 17 Sumba PROVINCES: 10° Waingapu Timor 10° Kupang 1 NANGGROE ACEH DARUSSALAM 12 JAWA BARAT 23 KALIMANTAN TIMUR 2 SUMATERA UTARA 13 JAWA TENGAH 24 SULAWESI UTARA 3 RIAU 14 D.I. YOGYAKARTA 25 GORONTALO 4 SUMATERA BARAT 15 JAWA TIMUR 26 SULAWESI TENGAH INDIAN OCEAN 5 JAMBI 16 BALI 27 SULAWESI BARAT 6 BENGKULU 17 NUSA TENGGARA BARAT 28 SULAWESI SELATAN 7 SUMATERA SELATAN 18 NUSA TENGGARA TIMUR 29 SULAWESI TENGGARA 8 LAMPUNG 19 RIAU KEPULAUAN 30 MALUKU UTARA 15° 15° IBRD 36232 JUNE 2008 9 BANGKA-BELITUNG 20 KALIMANTAN BARAT 31 MALUKU 10 BANTEN 21 KALIMANTAN TENGAH 32 IRIAN JAYA BARAT AUSTRALIA 11 D.K.I. JAKARTA 22 KALIMANTAN SELATAN 33 PAPUA (IRIAN JAYA) 115° 120° 125° 130° 135° 140°