Document of The World Bank Public Disclosure Authorized Report No: ICR00001589

IMPLEMENTATION COMPLETION AND RESULTS REPORT (IBRD-7190-PA, TF026846)

Public Disclosure Authorized ON AN

ADAPTABLE PROGRAM LOAN

IN THE AMOUNT OF US$24 MILLION

TO THE

REPUBLIC OF

FOR AN

Public Disclosure Authorized REFORM PROGRAM

IN SUPPORT OF THE

FIRST PHASE OF THE PROGRAM

June 14, 2010

Public Disclosure Authorized

Human Development and Social Protection Management Unit Country Management Unit for Argentina, Chile, Paraguay and Uruguay Latin America and the Caribbean Regional Office CURRENCY EQUIVALENTS (Exchange Rate Effective April 15, 2010)

Currency Unit = Guaraní (Gs) Gs 1.00 = US$ 0.000216 US$ 1.00 = Gs 4623

FISCAL YEAR January 1 – 31 December

ABBREVIATIONS AND ACRONYMS

ACE Parent Association (Asociación de Cooperación Escolar) APL Adaptable Program Loan CARE Steering Council for the Education Reform (Consejo Asesor para la Reforma Educativa) CAS Country Assistance Strategy CDE State (Department) Education Council (Consejo Departamental de Educación) CDS State (Department) Supervision Commission (Consejo Departamental de Supervisión) CRA Learning resource center (Centro de Recursos de Aprendizaje) CPS Country Partnership Strategy EGI Institutional Management Team (Equipo de gestión institucional) EIA Environmental Impact Assessment EMA Open (Eduación Media Abierta) GINA Integrated acquisition management procedure (Gestión Integral de Adquisiciones). GOP Government of Paraguay ICR Implementation Completion and Results Report IDB Inter-American Development Bank ISR Implementation Status and Results Report LA Loan Agreement LAC Latin America and the Caribbean Region M&E Monitoring and evaluation MEC Ministry of Education and Culture Secondary Education Improvement Project (Mejoramiento de la Calidad de la Educación MECES Secundaria) NGO Non-governmental organization PAD Project Appraisal Document PCN Project Concept Note PCU Project Coordination Unit PDO Project Development Objectives PEI School development plan (Plan Educativo Institucional) PSP Socio-Productive Project (Proyecto Socio-Productivo) PSR Project Supervision Report National Student Achievement Assessment System (Sistema Nacional de Evaluación de los SNEPE Procesos Educativos) TTL Task Team Leader UDE Department (State) Statistical Unit (Unidad Departamental de Estadística)

Vice President: Pamela Cox Country Director: Penelope Brook Sector Director: Evangeline Javier Sector Manager: Chingboon Lee Country Manager: Rossana Polastri Project Team Leader: Cristian Aedo ICR Team Leader: Cristian Aedo ICR Primary Author: Juan Prawda PARAGUAY EDUCATION REFORM PROGRAM FIRST PHASE

CONTENTS

DATA SHEET A. Basic Information B. Key Dates C. Ratings Summary D. Sector and Theme Codes E. Bank Staff F. Results Framework Analysis G. Ratings of Project Performance in ISRs. H. Restructuring I. Disbursement Graph

1. Project Context, Development Objectives and Design ...... 1 2. Key Factors Affecting Implementation and Outcomes...... 9 3. Assessment of Outcomes...... 14 4. Assessment of Risk to Development Outcome ...... 20 5. Assessment of Bank and Borrower Performance...... 21 6. Lessons Learned...... 26 7. Comments on Issues Raised by Borrower/Implementing Agencies/Partners ...... 27 Annex 1. Project Costs and Financing ...... 29 Annex 2. Outputs by Component...... 30 Annex 3. Economic and Financial Analysis (including assumptions in the analysis) ...... 42 Annex 4. Bank Lending and Implementation Support/Supervision Processes ...... 46 Annex 5. Beneficiary Survey Results...... 48 Annex 6. Stakeholder Workshop Report and Results ...... 49 Annex 7. Summary of Borrower’s ICR and/or Comments on Draft ICR...... 50 Annex 8. Comments of Cofinanciers and Other Partners/Stakeholders...... 63 Annex 9. List of Supporting Documents...... 64 MAP ...... 66 A. Basic Information PY - EDUCATION Country: Paraguay Project Name: REFORM PROJECT Project ID: P073526 L/C/TF Number(s): IBRD-71900,TF-54361 ICR Date: 06/24/2010 ICR Type: Core ICR REPUBLIC OF Lending Instrument: APL Borrower: PARAGUAY Original Total USD 24.0M Disbursed Amount: USD 22.8M Commitment: Revised Amount: USD 24.0M Environmental Category: B Implementing Agencies: Ministerio de Educacion Cofinanciers and Other External Partners:

B. Key Dates Revised / Actual Process Date Process Original Date Date(s) Concept Review: 07/16/2002 Effectiveness: 07/01/2004 07/01/2004 Appraisal: 02/18/2003 Restructuring(s): Approval: 07/31/2003 Mid-term Review: 03/06/2006 03/20/2006 Closing: 06/30/2007 12/30/2009

C. Ratings Summary C.1 Performance Rating by ICR Outcomes: Moderately Unsatisfactory Risk to Development Outcome: Moderate Bank Performance: Moderately Unsatisfactory Borrower Performance: Moderately Satisfactory

C.2 Detailed Ratings of Bank and Borrower Performance (by ICR) Bank Ratings Borrower Ratings Moderately Quality at Entry: Government: Moderately Satisfactory Unsatisfactory Moderately Implementing Moderately Quality of Supervision: Unsatisfactory Agency/Agencies: Unsatisfactory Overall Bank Moderately Overall Borrower Moderately Satisfactory Performance: Unsatisfactory Performance:

i C.3 Quality at Entry and Implementation Performance Indicators Implementation QAG Assessments Indicators Rating Performance (if any) Potential Problem Project Quality at Entry No None at any time (Yes/No): (QEA): Problem Project at any Quality of Yes None time (Yes/No): Supervision (QSA): DO rating before Moderately

Closing/Inactive status: Satisfactory

D. Sector and Theme Codes Original Actual Sector Code (as % of total Bank financing) Central government administration 20 30 Other social services 5 Secondary education 60 70 Sub-national government administration 10 5

Theme Code (as % of total Bank financing) Access to urban services and housing 14 30 Education for all 29 40 Education for the knowledge economy 29 30 Participation and civic engagement 14 Rural services and infrastructure 14

E. Bank Staff Positions At ICR At Approval Vice President: Pamela Cox David de Ferranti Country Director: Veronica Salatino Axel van Trotsenburg Sector Manager: Chingboon Lee Marito H. Garcia Project Team Leader: Mario Cristian Aedo Inostroza Cynthia Hobbs ICR Team Leader: Mario Cristian Aedo Inostroza ICR Primary Author: Juan Prawda

ii F. Results Framework Analysis

Project Development Objectives (from Project Appraisal Document) Improve management and efficiency of the system, and educational outcomes and equity in opportunity at the secondary education level.

Revised Project Development Objectives (as approved by original approving authority)

(a) PDO Indicator(s)

Original Target Formally Actual Value Values (from Revised Achieved at Indicator Baseline Value approval Target Completion or documents) Values Target Years Indicator 1 : Gross enrollment rate at the secondary level. 56% (At the end of Value APL I) quantitative or 44% 55.2% (2008) 59% Qualitative) (At the end of APL II) Date achieved 07/02/2001 06/30/2009 12/30/2009 Comments (incl. % The target for the end of APL I has been almost met. achievement) Indicator 2 : Net enrollment rate at the secondary level. 42% (At the end of Value APL I) quantitative or 33% 42% (2008) 44% Qualitative) (At the end of APL II) Date achieved 07/02/2001 06/30/2009 12/30/2009 Comments The target for the end of APL I has been met, however the number is somewhat (incl. % short of the rate expected by the end of the program. achievement) Indicator 3 : Repetition rate at the secondary level 1% (At the end of Value APL I) quantitative or 1% 0.6% (2008) 0.9% Qualitative) (At the end of APL II) Date achieved 07/02/2001 06/30/2009 12/30/2009 Comments The target for the end of the program has been surpassed. (incl. %

iii achievement) Indicator 4 : Dropout rate at the secondary level 9% (At the end of Value APL I) quantitative or 11% 10.7% (2008) 7% Qualitative) (At the end of APL II) Date achieved 07/02/2001 06/30/2009 12/30/2009 Comments The target has not been met. Even though the information shows a decrease in (incl. % the average dropout rate at the secondary level, the rate is still above the target achievement) expected at the end of APL I. Indicator 5 : Gross completion rate at the secondary level 79% (At the end of Value APL I) quantitative or 76% 69% (2008) 82% Qualitative) (At the end of APL II) Date achieved 07/02/2001 06/30/2009 12/30/2009 Comments The target has not been met. The actual outcome (2008 data) has been 69%, (incl. % meaning that the indicator moved in the opposite direction compared to the goal. achievement) Results of the SNEPE (2nd year of secondary): Indicator 6 : Language and Spanish literature Mathematics 54 40 (At the end of Value 51 APL I) quantitative or Data not available 36 60 Qualitative) 46 (At the end of APL II) Date achieved 07/02/2001 06/30/2009 12/30/2009 Unable to assess. This indicator was dropped for APL after a change in Comments methodology. 2006 SNEPE results became the baseline with new measurements (incl. % planned for 2010 and 2014. The application of the 2010 assessment has been achievement) postponed until 2011. Number of Students per section: Basic education 1st and 2nd cycles (grades 1 - 6) Indicator 7 : Basic 3rd cycle (grades 7 - 9) Secondary Education 23 30 Value 18 18 27 quantitative or 29 25 (At the end of Qualitative) 25 21 APL I) 28

iv 30 30 (At the end of APL II) Date achieved 07/02/2001 06/30/2009 12/30/2009 Comments The target has not been met. No increase in class size at any level has been (incl. % achieved, and in secondary education, there has been a reduction in the class size achievement) from 25 to 21 students.

(b) Intermediate Outcome Indicator(s)

Original Target Actual Value Formally Values (from Achieved at Indicator Baseline Value Revised approval Completion or Target Values documents) Target Years Model for selection and contracting of personnel designed, approved by MEC Indicator 1 : and in use in at least 3 Departments Model designed, Value Model currently approved and (quantitative No model implemented applied in 3 or Qualitative) nationwide Departments Date achieved 07/01/2004 06/30/2009 12/14/2007 Comments The target set at appraisal was surpassed. The model designed under the project (incl. % and approved by MEC is being used at the national level. achievement) Indicator 2 : Updating of a national education map completed Value (quantitative No updated map Map updated Map 100% updated or Qualitative) Date achieved 07/01/2004 06/30/2009 12/14/2008 Comments (incl. % The project accomplished the original target. achievement) Integration of the database (schools, human resources, statistics, SNEPE/student Indicator 3 : performance) completed Value No integration of the Database 100% Database 100% (quantitative database integrated integrated or Qualitative) Date achieved 07/01/2004 06/30/2009 12/30/2009 Comments (incl. % The project accomplished the original target. achievement) Indicator 4 : Baseline on student performance (2nd course) established Value No baseline on student Baseline Baseline (quantitative performance established established or Qualitative) Date achieved 07/01/2004 06/30/2009 04/20/2008 Comments The project accomplished the original target.

v (incl. % achievement) Indicator 5 : Accreditation and certification systems designed and approved by MEC Accreditation and Accreditation and Value certification certification (quantitative No system systems designed systems designed or Qualitative) and approved by and approved by MEC MEC Date achieved 07/01/2004 06/30/2009 12/14/2007 Comments The target set at appraisal was surpassed. The teacher certification model (incl. % designed under the project and approved by MEC is being implemented in the achievement) basic and secondary education levels. Indicator 6 : Social communication model agreed upon and in operation Social Value communication (quantitative No social communication 0 model designed or Qualitative) and in operation Date achieved 07/02/2004 06/30/2009 12/30/2009 Target not achieved. Despite having implemented a series of tasks, disseminated Comments materials and carried out training for communication facilitators under the (incl. % project, no evidence can be found that a social communication system was achievement) established. New initial teacher training curriculum approved by MEC and introduced in 5 Indicator 7 : public teacher training institutes Pre-service teacher Program approved training program Value by MEC and design completed, (quantitative No program introduced in 5 approved by MEC or Qualitative) public teacher and implemented in training institutes 13 institutions. Date achieved 07/02/2004 06/30/2009 12/30/2007 Comments There was an over achievement of the expected target. The pre-service teacher (incl. % training program is being implemented in 13 teacher-training institutes with 260 achievement) students enrolled. All of the 240 hour teacher training programs for upgrading of current teachers Indicator 8 : completed. All 240 hour in- All 240 hour in- Value service teacher service teacher (quantitative No programs training program training program or Qualitative) completed. completed. Date achieved 07/02/2004 06/30/2009 12/30/2008 Comments (incl. % The target set at appraisal has been achieved. achievement) Percentage of participants in the education leader component receiving a Indicator 9 : master#s degree Value 0 100% 98% (quantitative

vi or Qualitative) Date achieved 07/02/2004 06/30/2009 12/30/2009 Comments The target set at appraisal was achieved its stated target set at appraisal ensuring (incl. % that 98 professionals of MEC out of 100 that started completed their leadership achievement) program in two qualified . Indicator 10 : All of the 1st year secondary education texts designed Value 100% of text 87.5% of text (quantitative 0 designed. designed. or Qualitative) Date achieved 07/02/2004 06/30/2009 12/30/2007 Comments The target set at appraisal was achieved (see Annex 2 for an explanation). (incl. % Completed for 1st, 2nd, and 3rd grades for language (Spanish, Guaraní and achievement) English), Math, Social Sciences, Arts and . Indicator 11 : Operating CRAs and Science laboratories 75 CRAs and 25 Value science (quantitative 0 99% laboratories in or Qualitative) operation Date achieved 07/02/2004 06/30/2009 12/30/2009 Comments The target set at appraisal was achieved. 74 CRAs and 24 science laboratories (incl. % are operating. achievement) Quality improvement projects (PEIs) implemented in half of the 100 secondary Indicator 12 : schools selected for the program At least 50 PEIs implemented in Value equal number of 100 PEIs (quantitative No PEIs schools out of the implemented or Qualitative) 100 that were selected for this initiative Date achieved 07/02/2004 06/30/2009 12/30/2009 Comments (incl. % The project achieved twice of what was the expected target set at appraisal. achievement) Indicator 13 : Social community projects (PSPs) implemented or under implementation 100 PSPs Value implemented or 100 PSPs (quantitative No PSPs under implemented or Qualitative) implementation Date achieved 07/02/2004 06/30/2009 12/30/2009 Comments (incl. % The target set at appraisal was achieved. achievement) Indicator 14 : Parent association (ACE) participation in PEI design Value ACE participation Active ACE (quantitative 0 in PEI design in participation in 100 or Qualitative) 50% of the PEIs implemented

vii selected schools Date achieved 07/02/2004 06/30/2009 12/30/2009 Comments (incl. % The target set at appraisal was achieved. achievement) Indicator 15 : Scholarships assigned to low-income secondary students Value (quantitative 0 4,500 scholarships 4,500 scholarships or Qualitative) Date achieved 07/02/2004 06/30/2009 12/30/2006 Comments This output indicator was fully achieved. The scholarship has targeted low- (incl. % income students enrolled in secondary education and there has been achievement) transparency in their allocation. Indicator 16 : Mechanism established to guarantee sustainability of the scholarship program. Scholarship Value scheme as a Sustainability (quantitative No mechanism sustainable guaranteed or Qualitative) program Date achieved 07/02/2004 06/30/2009 12/30/2006 Comments Output indicator fully achieved. The GOP has financially and technically (incl. % sustained this scheme since 2007. In addition the GOP has increased the number achievement) of yearly scholarships. Indicator 17 : Existing alternative secondary education programs evaluated Value Evaluation study Evaluation not (quantitative No evaluation conducted. carried out or Qualitative) Date achieved 07/02/2004 06/30/2009 12/30/2009 Comments This outcome indicator was not achieved. The Alternative secondary education (incl. % program evaluation study was not carried out. achievement) Indicator 18 : Modular programs designed and approved by MEC and implemented System designed and implemented in Modular program 17 sites. Out of the Value designed and original 600 (quantitative No program initiated on a pilot students, 428 or Qualitative) basis in 20 dropped out, 76 locations. graduated and 96 continue their studies Date achieved 07/02/2004 06/30/2009 12/30/2009 Comments This outcome indicator was achieved but the modality has resulted to be highly (incl. % inefficient and not pertinent to the out-of-school population. MEC will not offer achievement) this modality during the next school year. Indicator 19 : Prototype schools constructed and fully operational Value 3 prototype school (quantitative 13.3% fully operational or Qualitative)

viii Date achieved 06/30/2009 03/01/2010 Comments This outcome indicator was not achieved. Two prototype schools built. One (incl. % school functioning since February 2009 with 245 students, while the second achievement) school started functioning in February 2010 with 146 students.

G. Ratings of Project Performance in ISRs

Actual Date ISR No. DO IP Disbursements Archived (USD millions) 1 10/21/2003 Satisfactory Satisfactory 0.00 2 05/17/2004 Satisfactory Unsatisfactory 0.00 3 12/14/2004 Satisfactory Satisfactory 1.00 4 04/13/2005 Satisfactory Satisfactory 1.00 5 11/07/2005 Satisfactory Satisfactory 2.56 6 11/22/2006 Satisfactory Satisfactory 6.81 7 06/22/2007 Satisfactory Satisfactory 10.24 8 12/18/2007 Satisfactory Satisfactory 17.16 9 04/03/2008 Satisfactory Moderately Satisfactory 17.91 10 05/21/2008 Satisfactory Moderately Satisfactory 18.90 Moderately 11 12/21/2008 Moderately Satisfactory 21.97 Unsatisfactory Moderately 12 06/21/2009 Moderately Satisfactory 22.53 Unsatisfactory Moderately 13 08/03/2009 Moderately Satisfactory 22.53 Unsatisfactory Moderately Moderately 14 06/24/2010 22.80 Unsatisfactory Unsatisfactory

H. Restructuring (if any) Not Applicable

ix I. Disbursement Profile

x 1. Project Context, Development Objectives and Design

1.1 Context at Appraisal

Macroeconomic situation. Compared with other countries in the Latin America and Caribbean (LAC) region at the time of project preparation, Paraguay registered some of the lowest economic growth patterns, highest income distribution inequalities and striking levels of poverty. The average per-capita income growth in Paraguay in the period 1980-2000 was only 0.4 percent of the average growth registered for the LAC region during the same time-period. Although Paraguay had improved income inequalities during the nineties, the income Gini coefficient at the time of appraisal (2003) was 58 percent, as compared to 54.9 percent for Chile, 51.3 for Argentina, 46.1 for Mexico and 44.9 for Uruguay for the same year. About 13.8 percent of the total population in Paraguay in 2003 was poor (earning US$2 or less a day) as compared to 8.5 percent in Brazil and 7.1 in Argentina.

Basic education reform background. In June 2001, the Government of Paraguay (GOP) was implementing the Basic Education Reform launched in 1994 with the financial and technical support of the World Bank (Bank) and the Inter-American Development Bank (IDB). This Reform aimed to expand mandatory free basic education to nine grades including six of primary followed by three years of lower secondary1, improve its quality, establish systems to measure student learning performance and improve sector planning. The reform was gradually implemented one grade per year. Thus, by 2003, the year this project was taken to the Board, the GOP was ready to undertake some improvements in secondary education (grades 10 to 12).

Significant gains were achieved before the start of the preparation cycle of this project. Net enrollments had expanded slightly in the first six grades of basic education from 92 percent in 1994 to 95 percent in 2002, and more significantly in grades 7-9 from 35 to 52 percent in the same time period. Net enrollment rates had also risen significantly in the level from 30 to 70 percent in the 1994-2002 period. Textbooks were provided by the GOP free of charge to all students in grade 7-9 for the first time in the country. A new national standardized student learning achievement system (Sistema Nacional de Evaluación de los Procesos Educativos – SNEPE) established baseline data to gauge future quality improvements. The Paraguayan Ministry of Education and Culture’s (MEC) Department of Planning and Statistics compiled the national education statistics as a way to strengthen the educational information management systems. Parents were encouraged to take active participation in the school through the parent’s association (Asociación de Cooperación Escolar – ACE).

Issues. At the time of project preparation, GOP had expressed its intention to continue improving the coverage and enrollment, especially for grades 10-12, the learning outcomes of students attending secondary education and the MEC’s institutional capacity.

Coverage and enrollment. The enrollment and completion gains in grades 7-9 of basic education increased demand for attending grades 10-12, which GOP could not appropriately address. Consequently, both gross and net enrollment rates of secondary education (grades 10-12) in Paraguay continued to be among the lowest in LAC and far behind its peer countries of Mercosur.

1 The first three grades of primary education are known in Paraguay as the first level of basic education while the next 3 grades (4, 5 and 6) are known as second level of basic education, and the last 3 grades (7, 8 and 9) are known as third level of basic education. The Reform made these nine grades of basic education compulsory. Grades 10 to 12, which were not compulsory, are currently known as secondary education. 1 Table 1. Gross and net enrollment ratios at secondary level (grades 10-12) in selected countries Country or region Gross enrollment rate (%) Net enrollment rate (%) Latin America Region 54.00 Paraguay 56.90 45.04 Argentina 93.70 76.02 Brazil 103.25 68.46 Chile 87.47 71.77 Uruguay 91.55 77.39 Colombia 70.94 54.27 Mexico 73.40 57.37 Peru 80.77 61.49 Source. Project Appraisal Document, Table 1, page 6.

In addition, only 50 percent of students completing basic education (grade 9) continued to secondary education with only 24 percent of those that started secondary education finishing this level (the secondary education completion rate in rural areas was 5 percent). Although there was gender parity in enrollment at the secondary education level, there was a stark contrast between the urban and rural enrollment (83 percent of total secondary education enrollment was urban and 17 percent rural) in a country like Paraguay with 50 percent of its population living in rural areas. The inequitable access to school mirrored the overall income inequality mentioned above.

Low coverage at the secondary level was due to a combination of demand- and supply-side factors. About 32 percent of 13 to 18 year olds did not attend school, with this share rising to 46.4 percent in the lowest income quintile and to 38 percent in the next income quintile. Lack of financial resources to cover annual fees, uniforms, transportation and other related expenses were found to be inhibiting increased demand. Lack of secondary education supply was an issue mostly in rural areas, where a shortage of these schools forced rural families to forego further schooling for their children or sent them to urban secondary schools, thus creating direct lodging and travel-related costs, which in many cases could not be financially sustained by these families. As a consequence of families sending their children to urban secondary schools, the capacity of these schools was exceeded, with many classrooms holding up to 70 students.

Quality and outcomes. At the time of the identification phase, Paraguay’s students learning achievement was low in general, and usually lower in rural areas. SNEPE test results for third graders in language and mathematics in urban areas averaged 47.8 and 45.3 percent pass rate respectively, while in rural areas it was 40.4 and 43.8 percent. Learning outcomes in grade 12 (measured through a non-standardized national student performance test - Pruebas Nacionales - a different testing instrument than SNEPE) were not too much different. Urban secondary education students scored consistently higher than rural students, girls higher than boys, private school students higher than those in public schools, and day shift students higher than night shift students.

Dropout rates were relatively high in 2001 – 5 percent as an average for grades 1-6, 8.5 percent as an average for grades 7-9 and 11 percent as an average for grades 10-12. According to a study by the Universidad Católica de Asunción (1999), students who did graduate from grade 12 showed crucial deficiencies in the ability to think critically and interpret ideas, thus lacking the skills to become productive citizens in the knowledge economy and the changing world.

Poor learning outcomes in secondary education were partially due to teaching and learning shortcomings mainly characterized by: (a) teachers delivering rote lessons with no room for participatory learning; (b) school environment not conducive to good teaching and learning and lacking basic teaching and learning materials, textbooks (for grades 10-12), laboratories and new education technology; (c) poorly stocked

2 and under-utilized school libraries, not connected to the Internet; and (d) few available facilities for individual study, group meetings and hands-on . was an issue, especially at the basic education level and in rural areas, where many entering students spoke only Guaraní. Few teachers had received formal bilingual training and the there were issues in appropriately translating the oral Guaraní language into written texts.

Many teachers had low qualifications, with the majority of the less-qualified working in the rural area. Only about half of the secondary education teachers were specifically trained for that level, and the training was of doubtful quality. In addition, the MEC had changed in basic and secondary education the subject-based curricula for competency-based area curriculum for which the majority of secondary education teachers were not trained to teach. Pre- and in-service teacher training was fragmented and disconnected and accounted for many of the deficiencies observed in teacher’s performance. The limited in-service teacher training provided in the secondary level was supply rather than demand driven, with training content determined at the central level totally detached from individual schools’ and teachers’ needs. School supervision at the secondary education level was focused on administrative rather than pedagogical issues. Each school supervisor was responsible for about 60 schools, visiting mostly urban schools a few times per year and rural schools only once a year. About 68 percent of school directors also taught, leaving little time for classroom observations and teacher support. The quality of the teacher training institutes was low and functioning without quality standards and accreditation procedures.

Institutional capacity. At the time of project preparation, the MEC was burdened with inefficiencies mainly characterized by: (i) lack of control over personnel recruitment and leaves; (ii) an inadequate legal framework to efficiently manage teaching staff appointment, to avoid unnecessary growth of teaching positions and teaching hours; and (iii) lack of institutional tools to properly plan and manage the human, financial and physical resources of the education sector. MEC’s administrative, planning and budgetary systems were totally uncoordinated working as independent units.

Although legal framework to decentralize the delivery of education services through MEC’s regional administrative departments was approved in 2000, MEC’s management model continued to be highly centralized at the time of project preparation and greatly uncoordinated between the central, regional and school education authorities. The top-down management style was replicated at the Departmental and school levels, whereby school principals made key decisions sharing little information with students, teachers or parents. ACEs, a growing voice in basic education, had not yet gained a place in secondary schools, especially with respect to planning, monitoring and motivation related activities.

The decentralization process was weak and poorly managed and financed. State education councils (CDEs) had been formed, but they did not receive the required resources to carry out their envisaged decentralized mandate.

Government’s strategy. The education sector has been at the forefront of the GOP’s political agenda since the arrival of the democratic process in Paraguay at the end of the eighties. In 1991, the democratically elected Government launched an ambitious education reform. As part of this reform process, a Steering Council for the Education Reform (Consejo Asesor para la Reforma Educativa – CARE) was established in 1991 to bring together all the key stakeholders from political, religious, civil society and government organizations in order to agree on key state educational policies and strategies, to be later included in the newly adopted 1992 Constitution. As a result, all the key strategies, programs and specific activities to be financed in the education sector to achieve the reform objectives were grouped into three main axes: (a) quality; (b) equity; and (c) efficiency.

One of the initial outcomes of the reform was for the MEC (through CARE) to contract with the Institute for International Development of the of Harvard and the Paraguayan Center for Sociological

3 Studies a series of diagnostic studies of the education sector (see Annex 9) to highlight key issues in the sector and propose remedial actions. Derived from these studies, a series of actions were undertaken by the MEC including, inter alia: (a) the provision of textbooks and learning materials for preschool and basic education (grades 1-9); (b) a curriculum revision for preschool, basic and secondary education; (c) the promotion of bilingual education with the teaching and learning of Guaraní; (d) the strengthening of parent participation in basic education; (e) the development and implementation by public schools of their annual school development plan (Plan Educativo Institucional – PEI); (f) the strengthening of the education statistical system; and (g) the installation of the SNEPE. Some of these initiatives were partially financed by the Bank and others by the IDB.2

The MEC announced in 2003 its long-term Strategic Plan for Secondary Education Reform for the period 2003-2015 through the publication and dissemination of the Plan Ñanduti (Plan Nacional de Educación para Todos 2003-2015) aligned to the GOP’s National Development Plan 2003-2008. The project was developed in accordance with the major objectives of the Plan Ñanduti, namely: (i) improving the quality of the teaching and learning, especially of language and mathematics at all educational levels, including secondary education; (ii) improving the equity in secondary education as it pertains to access and completion rates; and (iii) improving the efficiency of the education systems through more pertinent planning, better utilization of available human and physical resources, better and timely utilization of management information systems and by updating obsolete management procedures.

Accordingly, the GOP and the Bank agreed on an six-year Adaptable Program Loan (APL), to be carried out in two phases lasting about four years the first phase (from September 30, 2003 to June 30, 2007) and two years the second phase (from July 1, 2007 to June 30, 2009), to help design, implement and evaluate these policies and institutional reforms and provide the needed financial, human and physical resources (more detail on the second phase of the proposed APL is provided in section 2.5 of this ICR). The first phase of the APL aimed at working on policy reform issues and piloting processes, in particular in the areas of teacher training and certification, learning materials, access to secondary education of poor students and managing/planning in the education sector. The second phase aimed at expanding those activities of the first phase that proved to be cost-effective.

Rationale for Bank’s assistance. The rationale for Bank’s assistance was argued on the basis of the Bank’s long history of commitment to financial and technical assistance to the GOP and an ongoing relationship with the MEC through the implementation of the then ongoing Bank-financed MECES project. The collaborative approach developed in the preparation and supervision of MECES yielded increased cooperation between the Bank and a wide range of education stakeholders, including the teacher union and student association representatives, creating a conducive environment for a subsequent operation attuned to the ongoing education reform.

In addition, the Bank project team also drew on the extensive work and experience in supporting other secondary education projects elsewhere in LAC (Argentina, Brazil, Chile, Mexico and El Salvador) and in the world, thus providing best practice examples of initiatives at this educational level to improve quality. In particular, the Bank team offered expertise on secondary education reforms, school-based programs and continuous teacher training programs.

2The Paraguay Secondary Education Improvement project (Proyecto de Mejoramiento de la Calidad de la Educación Secundaria – MECES) to improve the access and quality of grades 7-9 was financed by the Bank through Loan 39410-PY. The following initiatives were financed by the IDB: (a) Paraguay Primary Education Improvement Program to improve the access and quality of grades 1-3; (b) Strengthening of Bilingual Education to support teacher training and provision of learning quality inputs for basic education, especially in the Guaraní language; and (c) Program to Strengthen Basic Education Reform to continue improving the access, retention and completion rates as well as the quality of grades 1-6. 4 1.2 Original Project Development Objectives (PDO) and Key Indicators

Attuned to the Plan Ñanduti and to MEC’s Strategic Plan for Secondary Education Reform for the period 2003-2015 the objectives of the project were to improve the Borrower’s: (i) management and efficiency of the education system; and (ii) educational outcomes and equity in opportunity at the secondary education level.

The achievement of these objectives was to be measured by the following outcome indicators shown in Table 2 below.

Table 2. Agreed outcome indicators and targets for APL I and II Indicator Year 2001 2006 2009 (baseline) (end of APL I) (end of APL II) Gross enrollment rate at the secondary level 44% 56% 59% Net enrollment rate at the secondary level 33% 42% 44% Repetition rate at the secondary level 1% 1% 0.9% Dropout rate at the secondary level 11% 9% 7% Gross completion rate at the secondary level 76% 79% 82% Learning outcomes in Spanish and Math in grade 2 of as measured by the National Student Achievement Assessment System (SNEPE) Spanish Mathematics 51% 54% 60% 36% 40% 46% Number of students per section:

Basic education - 1st and 2nd cycles (grades 1-6) 18 23 28 Basic education - 3rd cycle (grades 7-9) 29 30 30 Secondary education 25 27 30 Source. Project Appraisal Document, page 4.

Of these indicators, the only ones monitored during project implementation and reported in the ISRs were the net enrollment rate and the gross completion rate at the secondary level. However, the ISRs monitored progress on the following trigger indicators for moving to Phase II of the APL except the one concerning scholarship which was converted into an intermediate outcome indicator and reported accordingly in the ISRs:

• Model for selection and contracting of personnel designed, approved by the MEC, and in use in at least 3 states (Departamentos); • Number of students per class increased from 18 to 20 in the first two basic education cycles and from 25 to 27 at the secondary level; • Certification and accreditation policies designed and approved by the MEC; • Design of an initial teacher training finalized, design and implementation plan approved by the MEC, and new program introduced in 30 percent of the official training institutes (5 institutes); • 50 percent of the PEIs in place in selected schools (100 schools); • At least 4,500 scholarships assigned, a strategy established and approved by the MEC to assign funds exclusively for scholarships, thereby ensuring sustainability of the program; and • Existing alternative secondary education programs evaluated and modular program designed, design and implementation plan approved by the MEC and program initiated on a pilot basis in 20 locations. 5 1.3 Revised PDO (as approved by original approving authority) and Key Indicators

The Project Development Objectives (PDOs) were not modified during project implementation. However, for the reasons explained in Section 2.3 of the ICR, the outcome indicator concerning learning outcomes in Spanish and Mathematics in grade 2 as measured by SNEPE was removed from the list. The SNEPE findings of the measurement carried out in 2006 were now made the new baseline, expecting to measure changes with the next (comparable) measurement to take place at the end of 2010, outside the scope of this first phase of the APL.

1.4 Main Beneficiaries

According to the Project Appraisal Document (PAD) the project aimed at providing about 22,400 additional poor students with the opportunity to access and complete good quality secondary education in addition to the ongoing secondary education enrollment at the time of project preparation of about 167,000 students. The additional supply to be provided and financed by the project was to include 9,000 scholarships to attend the regular secondary education modality, 5,000 places in new classrooms/secondary schools through the construction of three macro-secondary schools and 8,400 through programs. The project design estimated that labor earning over the student’s working lifetime would increase by approximately 35 percent for those graduates of secondary education.

Changes in the teacher training programs and better quality control of teacher training institutes through the new accreditation system included in the project aimed at improving the quality of all new teachers entering the system during project implementation (about 260) and of approximately 12,000 public teachers working in schools at the time of project launch.

The project also intended to benefit the Ministry of Education by generating savings through a more efficient allocation of students across schools in the education system as a whole. Accordingly, the project design estimated that by increasing the number of students per section in the first two cycles of basic education (outside the scope of the project) from 18 to 28 and from 25 to 30 in third cycle of basic education (also outside the scope of the project), the potential total savings would be equivalent to 11.7 percent of the total budget of the MEC, equivalent to about US$44.8 million in teachers’ salaries in one year. Even though the first three-levels of basic education were outside the scope of the project, the above-mentioned savings were intended to be generated by the strengthening of micro-planning activities carried out by the Planning Unit of the MEC financed by the project. These strengthening activities aimed at rationalizing existing human and infrastructure resources at all levels of education and making better allocation of fresh ones.

The project design also intended to benefit the planning units of the Ministry of Education at the national and state levels as well as project secondary schools through better monitoring and supervision at the state level, improved planning and management of resources at the school level and stronger parent and community participation planning and monitoring resulting in greater efficiency in school management and improved accountability. Accordingly, the project intended over the six-year period of the two phases to benefit 200 schools to produce their PEIs, 400 schools to elaborate their Social Community Project (Proyecto Socio-Productivo – PSP) and to train parents in 1,200 ACEs to participate in the preparation of their school’s PEI and PSP.

1.5 Original Components

6 The following four components comprising 7 sub-components and 17 distinctively different large key activities were included in the project’s original design as fully explained in the PAD Project Appraisal Document (PAD), report No. 25630 dated June 25, 2003:

Component A – Improved tools for planning and management of the education system (total cost US$8.86 million; US$8.52 million from the loan). This component, representing 33.4 percent of the total project cost covered two dimensions – one dealing with the strengthening of the MEC’s management capacity and another one pedagogical in nature. Within the first dimension, the project aimed at financing the following key activities: (i) the enhancement of macro and micro-planning, management and supervision capacity; (ii) the design and implementation of a certification system for MEC professionals, including administrative staff and teachers; (iii) the design of an accreditation system for teacher training institutes; (iv) the expansion of the student performance assessment system (SNEPE) for secondary education in the areas of language, mathematics and critical thinking; and (v) the implementation of a national education communication plan to reach a diversity of stakeholders.

Within the pedagogical dimension, the project aimed at financing the following key activities: (i) the design and implementation of a continuous pre-service teacher training system; (ii) the provision of in- service teacher training to selected secondary education teachers to improve teaching and learning outcomes; and (iii) the implementation of a graduate leadership program for selected key public education leaders, authorities and managers.

This Component intended to finance, inter alia: (a) technical assistance; (b) training; (c) acquisition of necessary equipment, furniture and software for upgrading of the CDEs, the Departmental Supervision Commissions (CDSs) and the Departmental Statistical Units (UDEs) within the MEC; (d) provision of software and printing materials for the operation of the improved administrative model, including publication of financial and human resource management handbooks; (e) technical assistance to design, validate and apply learning assessment tests, acquisition of equipment to score the tests, training and dissemination of key tests’ findings to parents, students, teachers, principals, supervisors and authorities; (g) acquisition of vehicles for the central MEC, the Departmental Education Councils and the Project Coordination Unit (PCU); (h) workshops; and (i) funds for the design of printed and media messages.

Component B – Secondary School Improvements (total cost US$8.4 million; US$7.57 million from the loan). This component, representing 31.7 percent of the total project cost, aimed at financing the following key activities: (i) the provision of quality didactic materials and learning equipment to secondary schools, including training; and (ii) community-associated management through the design and implementation of PEIs to carry out demand-driven investments for institutional planning purposes at the school level as well as PSPs consisting of investments for productive purposes at the school level.

This Component intended to finance, inter alia: (a) the acquisition of newly designed textbooks for secondary education; (b) the acquisition of learning resource centers (Centros de Recursos de Aprendizaje – CRAs) to provide a place for research and group activities as well as a learning material reference center for teachers in 75 schools including visual and information equipment, library books, maps and operational handbooks; (c) the acquisition of 25 science laboratories, including the training of teachers; (d) the provision of funds (grants) to finance the preparation and implementation of PEIs and PSPs; and (e) the provision of technical assistance and training for ensuring parent participation in the design of their school’s PEI and PSP as well as their participation in the ACEs.

7 Table 3. Original expected costs and sources of financing at appraisal Components Total Bank GOP % financed by indicative financing counterpart the Bank Costs (US$M) (US$M) (US$M) (1) (2) (3) (2)/(1) 1. Improved tools for planning and 8.86 8.52 0.34 96.1 managing of the education system 2. Secondary school improvements 8.40 7.57 0.83 90.1 3. More equitable access to secondary 7.80 6.82 0.98 87.4 education 4. Project management 1.20 1.09 0.11 90.8 Total project costs 26.26 24.00 2.26 91,4 Front-end-fee 0.24 0.00 0.24 0.0 Total financing required 26.50 24.00 2.50 90.50 Source: Section C.1 of the PAD.

Component C – Mechanisms to Provide More Equitable Access to Secondary Education (total cost US$7.8 million; US$6.82 million from the loan). This component, representing 29.4 percent of the total project cost, aimed at financing the following key activities: (i) a sustainable scholarship program for extremely poor students; (ii) the design and implementation on a pilot scale of alternative secondary school programs for rural and working youngsters; and (iii) the construction of prototype secondary schools in marginal urban areas.

This Component intended to finance, inter alia: (a) the provision of technical assistance for the design and implementation of the scholarship system to be implemented in six Departments (Alto Paraná, Caaguazú, Central, Guairá, Itapúa and San Pedro) to benefit low-income students; (b) scholarship funds; (c) operating costs associated to the provision and monitoring of the scholarships; (d) the provision of training to about 100 education professionals to carry out the application of indirect income tests and home visits; (e) the provision of technical assistance for the design and implementation of an alternative secondary education program to provide flexible opportunities to out-of-school low income students demanding secondary education; (f) training; (g) printing and distribution of learning modules; and (h) civil works for 3 prototype secondary schools in the Departments of Caaguazú, Fernando de la Mora and Luque.

Component D – Project Management (total cost US$1.2 million; US$1.09 million from the loan). This component, representing 4.5 percent of the total project cost, aimed at financing the following two key activities: (i) the management of the project by a PCU, including the required annual audits, office equipment, operating costs and training of the PCU staff; and (ii) the project monitoring and the carrying out of tailored-made impact evaluation studies.

1.6 Revised Components

The original components were not revised during project implementation. Nevertheless, even though Schedule 2 to the Loan Agreement (LA) does not include civil works for the construction and/or rehabilitation of CDEs (including their CDSs and the UDEs), the Bank approved the construction of 8 CDEs in the cities of Asunción, Caacupé, Caazapá, Concepción, Encarnación, Filadelfia, San Lorenzo and Villarrica during the implementation cycle.

8 1.7 Other significant changes

As detailed elsewhere in the ICR, one of the three prototype schools included in the original design (the one located in Fernando de la Mora) was dropped from further consideration due to lack of available financial resources once the original architectural design for each of the three prototypes was radically modified by the pedagogical units within MEC resulting in significant increases in construction square footage and unit cost. In addition, only 5 of the 8 intended CDEs construction projects were financed by the loan (Caacupé, Caazapá, Concepción, Encarnación and Filadelfia).

The original closing date of June 30, 2007 was extended three times for a total of two and a half years.

2. Key Factors Affecting Implementation and Outcomes

2.1 Project Preparation, Design and Quality at Entry

This and the following sections in the ICR were written based on: (i) a thorough review of the project files; and (ii) the findings of the supervision missions carried out in November 17-21, 2008, March 23-27, 2009 and June 22-25, 2009 and the ICR field mission undertaken on December 1-4, 2009, which included a visit to the construction sites of the prototype schools in Luque and Caaguazú and the CDE of Caacupé.

The ICR rates Quality at Entry as moderately unsatisfactory, based on current standards for quality at entry, which were not in place at the time of project preparation. A summary of the ICR’s assessment on these topics follows, while sections 2.2, 2.3 and 5.1 (a) of the ICR provide more detail on the reasons for this assessment.

The strategic relevance and approach was appropriate based on: (i) the findings of the studies commissioned by CARE to launch the education reform in 1994; (ii) the lessons learned from the then ongoing Bank-financed MECES and other education sector operations financed by the IDB; and (iii) the recommendations of experts from others countries, in particular Argentina and Chile that were undergoing an education reform.

The selection of an APL as the lending instrument to gradually achieve GOP’s secondary education sector medium-term vision attuned to the ongoing education reform was also an appropriate decision.

Project preparation included effective policy dialogue with the GOP and an extensive consultation process, including key members of the Education Commission of Congress that were to play a key role during the implementation cycle, thus ensuring in the process, GOP’s ownership of the project design.

Relations with other agencies. Although no other multilateral or bilateral institution was included in the financing of this project, the Bank team worked closely with the IDB, which was heavily involved in supporting basic education in Paraguay. In addition, variety of qualified non-governmental organizations (NGOs) participated in other activities of the preparation and design.

Outcome and output indicators. The project’s results framework as set out in the PAD was appropriate and well defined for the series as a whole. However, only two outcome indicators (out of six) were monitored during implementation of Phase I,3 so that there is a disconnect between ICRs and ISRs ratings.

3 As mentioned earlier, the ISRs monitored progress on two PDO indicators (net enrollment rate and gross completion rate at the secondary level) and the trigger indicators for moving to Phase II of the APL. 9 Technical, financial and economic aspects. While recognizing that most of the approaches included in the project design, except the ones concerning the prototype schools and alternative secondary education, were well-tested elsewhere and proven effective, the project design was overly ambitious covering too many fronts (17 key activities grouped in 7 sub-components and 4 components). This complexity in the design caused significant delays in project implementation of some activities, like the civil works of the prototype schools and the acquisition of the textbooks, learning materials for the CRAs and science laboratories, alternative secondary education as well as the tools for planning and managing the education system.

The ICR detected four additional technical shortcoming fully explained in section 5.1(a): (i) lack of articulation among project activities; (ii) inclusion of sub-components – such as alternative secondary education option, new model of teacher training and social communication – in the absence of a technical and operational description at the time of Board approval; (iii) projected unit cost estimates that turned out to be significantly below actual costs, for example in the case of management re-engineering for the MEC, learning inputs and civil works (for the CRAs and CDEs/UDEs); and (iv) lack of a cost-benefit analysis to justify the construction of three prototype secondary schools.

On a more positive note, the ICR highlights the extreme care taken during project preparation to ensure the appropriate poverty-related targeting of the scholarship scheme and the mechanism to ensure transparency and accountability in the selection of candidates and in the allocation and management of the corresponding financial resources.

The implementation arrangements for this project were appropriate entrusting: the PCU with the fiduciary responsibilities and provision of technical assistance and MEC’s frontline units with the actual operation of the different project activities.

The project design complied with almost all the Bank’s safeguard policies in existence at the time of project appraisal, except with the Indigenous Peoples Plan (to benefit the indigenous population enrolled in some project schools), which was not included during the preparation phase. The fact that the in Paraguay makes the teaching of Guaraní mandatory in primary and secondary schools mitigated this project preparation oversight, thereby ensuring that the interests of indigenous communities were taken into account during project implementation. Procurement and financial management assessments were carried out, a first-year procurement plan was made available and a thorough social assessment was undertaken highlighting key issues, like the high opportunity costs for poor students to continue with their secondary education, which were appropriately addressed by project design. Environmental guidelines were provided for the prototype schools.

Risk assessments. The overall risk assessment for this project was adequate. The project highlighted some potential critical risks and proposed some mitigating measures as detailed in section 5.1(a) of the ICR. One risk that was missed was possible delays in Congress approval (see section 2.2).

Lessons learned. Lessons learned from the Bank-financed MECES program and from other education projects worldwide, detailed in section 5.1(a) of the ICR, were incorporated in the project design.

Weighing the above strengths and weaknesses of project design, in particular the lack of cost-benefit analysis to justify the inclusion of the large civil works, the lack of a technical and operational description of some key activities and the lack of implementation readiness at the time of Board approval, the ICR ranks the Quality at Entry of this project as moderately unsatisfactory.

10

2.2 Implementation

The following factors significantly delayed the physical and financial implementation progress of the project and affected achievement of results. First, the loan effectiveness required Congress approval that took almost 10 months after the signing of the loan in September 2003. Once the loan was approved by the Paraguayan Congress in mid-2004 the loan could not be disbursed until overdue payments on other loans needed to be resolved. The Special Account was finally opened towards the end of 2004 and the first allocation of one US million dollars was made by the Bank.

Second, the project was not ready for implementation at the time of loan approval, in particular: (i) several key activities under the project (pre-service teacher training, the alternative secondary education) were not defined; (ii) there were no technical and pedagogical specifications for the acquisition of textbooks, science laboratories and other learning resources for the CRAs, or technical specifications for the reengineering process; and (iii) there was no agreement between the architects and the pedagogical units within the MEC about the prototype schools resulting in a significant change of the original appraised designs including a noteworthy increase in costs and square footage and the deletion of one of the 3 original construction sites.

Third, the PCU was not effective in implementing required procurement activities, especially during the early (2004 to March 2006) and latest stages of the implementation cycle (2008-2009).

Fourth, project implementation slowed down for some months (August 2008 to March 2009) after the new government took office (see section 5.2(b) of this ICR).

It took the MEC about 33 months after Board approval in July 2003 to address the above-mentioned implementation issues and get all the project activities under implementation. As a consequence, from the mid-term review of March 2006 until the end of 2007 the physical and financial implementation progress improved with total documented disbursement amounting to US$16 million (67 percent of the total contracted loan) during these 21 months. One project activity – the scholarship program – was successfully completed. The local counterpart teams coordinated better among themselves and began demonstrating greater cost-effectiveness in their work.

From 2008 onwards it became clear that the project was running out of funds and difficult trade offs (further explained below) had to be made by the MEC. Several reasons explain this financial situation. First, the local currency (Guaraní) appreciated: Guaraní was worth 6,900 units for one US dollar at appraisal, reaching its highest appreciation in October 2008 of 3,930 to one US dollar and falling back to 4,780 at the closing of the loan in December 2009. As the loan was denominated in US dollars, but by law expenditures had to be made in local currency, the appreciation already reduced the purchasing capacity by about one third. Second, the reallocation of loan proceeds to the civil works category of expenditure to cover the additional requirements of the new design of the prototype schools (going from US$3.8 to US$4.8 million) reduced the availability of financial resources for other ends such as to acquire learning and multimedia goods, computing equipment for the management reengineering of MEC, school furniture and to select qualified consultants or firms to carry out some evaluation studies. Third, the original cost estimation in the project design included only textbooks for the of secondary education but during implementation the acquisition scope was increased to also cover the second and third grades of the common curricular plan. Finally, the projected unit costs at appraisal were below the real ones, partly due to local and international inflation.

After the hand-over of project management to a new administration in August 15, 2008, the implementation process slowed for some months. This situation hindered the ongoing construction process of the two prototype schools and the management re-engineering process. At the end of 2008 the 11 new administration cancelled the international acquisition process worth approximately US$500,000 of computers for the prototype schools and the MEC’s management reengineering that was started by the previous administration. Key technical and fiduciary staff at the PCU was replaced by Ministry of Education staff which required initial training and close supervision by Bank procurement staff to implement planned procurement activities.

2.3 Monitoring and Evaluation (M&E) Design, Implementation and Utilization

The project’s monitoring and evaluation (M&E) arrangements as well as the baseline data were ready at the time of Board approval. The results framework included seven outcome and nineteen output indicators that the MEC/PCU was able to monitor and report during the implementation cycle.

Outcome indicators were intended to measure the attainment of the three parts of the PDO: education system management and efficiency; educational outcomes (quality of education); and equitable access to secondary education. In particular, improvements in management and efficiency of the education system were expected to be manifested by increased class sizes, improved gross completion rates and lower repetition and dropout rates. Educational outcomes were to be measured directly through math and language SNEPE tests. Finally, equity in access to secondary education was to be achieved by increasing enrollment rates and specifically by providing targeted scholarships to poor students, thus increasing enrollments for this group.

These indicators were expected to be measured at the end of the entire APL program. For APL I, in addition to tracking progress towards these outcome indicators, the project was expected to report on the attainment of APL II triggers and on the delivery of specific outputs by component.

While in general the M&E framework was reasonably well defined and appropriate, it was too complex and included too many indicators. There were also discrepancies between the indicators listed in the main text of the PAD, Annex 1 of the PAD and those tracked in ISRs, which further complicated M&E arrangements. Finally, one of the key indicators – learning outcomes as measured by SNEPE – had to be dropped during implementation when it turned out that the 2001 learning outcomes exams administered by SNEPE (under a subject-based curriculum) and intended to be used as a baseline for the project were not comparable to the exams to be administered in 2006 and then again in 2010 (under a competency- based curriculum). For this reason, this baseline was considered inappropriate and replaced by the findings of the 2006 learning assessment. The indicator was dropped as the second measurement to assess changes against the baseline (the one to be carried out at the end of 2010) was outside the project’s time- horizon. No alternative indicator was proposed to replace it, thus leaving the project with no tool to measure the attainment of one of the parts of the PDO.

Finally, the project design intended to carry out several independent evaluation studies and/or thorough monitoring follow-up activities of the pilots being tested under the project. With the exception of the impact evaluation study carried out for the scholarship scheme, whose findings were reported in the sector work (World Bank 2007), and some follow-up on the use of textbooks and CRAs, none of the other key activities was assessed with respect to the attainment of their particular goals. Accordingly, there is a lack of hard evidence to ascertain if some of the following project activities had an immediate observable impact and/or if they had attained their expected goals: (i) the leadership program; (ii) the pre and in- service teacher training; (iii) the use of school mapping to make a better allocation of scarce resources in secondary education; (iv) the community-based strategies including the PEIs, PSPs and ACEs; and (v) the alternative secondary options. Lack of financial resources prevented the carrying out of these evaluation studies.

12

2.4 Safeguard and Fiduciary Compliance

The project implementation process generally complied with the Bank’s procurement and financial management requirements. Main issues arising during implementation concerned: (i) poor documentation quality; and (ii) duplication of review tasks, confused roles within the MEC, cumbersome coordination mechanisms and accountability channels making the existing administrative arrangements in place at the early stages of implementation neither efficient nor conducive to expeditious and timely decisions thus causing significant procurement delays. These were satisfactorily addressed by the MEC.

Although at the time of project approval the Bank agreed to a first-year procurement plan and judged the procurement implementation capacity as adequate, several issues arose in practice. First, no approved final project’s Operational Manual, including a procurement section, was available at project approval, although a draft had been discussed with the Bank. It took almost 3 years after Board approval to have the technical and pedagogical specifications ready for the acquisition of learning goods, the selection of a qualified firm to undertake the MEC’s reengineering task and the final design for the construction of the prototype schools. The construction and/or rehabilitation of 72 classrooms to house the different size CRAs was a procurement activity started on time.

MEC/PCU did not select a qualified construction oversight firm (fiscalizadora) in a timely manner, which delayed loan disbursements against eligible expenditures incurred in the construction of the two prototype schools and the 5 CDEs. This firm was contracted towards the end of 2007 and construction was accordingly delayed.

On the positive side, the PCU’s procurement unit was the recipient of the Ministry of Finance’s prize for good governance and transparency, among other things, for having implemented the nation system-wide integrated acquisition management procedure (Gestión Integral de Adquisiciones – GINA). This system is a digital back-up of all the acquisitions of goods, services and works undertaken during the life of the project, including documents from the original request to initiate the acquisition to the final payment to the supplier. This system produces a series of decision-making reports, among others the annual and revised procurement plan and a supplier’s compliance tracking record.

The Bank was provided in a timely manner with the quarterly financial management reports and with the unqualified annual audit reports. All the reports were found to be satisfactory.

Environmental and resettlement assessments were carried out for the two prototype schools built under the project but not for 5 CDEs. For the latter case, a retroactive assessment undertaken by the Bank’ Team found that no environmental issues were triggered during the construction of the CDEs. Despite the fact that no Indigenous People Plan was prepared for this project, the MEC satisfactorily complied with the corresponding safeguard including learning materials (textbooks) and teacher training in Guaraní, mostly as a result of a national bilingual educational policy in existence in Paraguay long before the launching of the project.

2.5 Post-completion Operation/Next Phase

All the triggers to move to the second phase, except one (increase in number of students per class), were satisfactorily attained. Preparation of the second phase started in 2008, but after the change in government MEC decided not to request a second loan and the preparation activities for the second phase were suspended.

13

Nonetheless, the following project activities are being sustained by the MEC with their own fiscal resources:

(a) In the area of improved tools for planning and management of the education system supported by the project under component A: (i) the educational statistical management information system is an ongoing activity within the MEC that has made available the official statistics for the school year 2008 as well as the preliminary information for 2009; (ii) the management re-engineering process that was installed by the project is also an ongoing and, gradually, expanding activity within the MEC; (iii) four of the five CDSs completed before the closing of the loan are fully functional and providing the expected administrative services (see Annex 2 for details on the fifth, partially completed CDS); (iv) the model to accredit and certify teacher training institutes is being currently applied to 20 teacher-training institutes; (v) the pre-service teacher training developed under the project is being utilized in 5 institutions benefiting 260 students expected to graduate during the second semester of 2010; (vi) the in-service teacher training developed under the project is being provided to about 1,780 participants during 2010; and (vii) SNEPE has been institutionalized within the General Directorate of Evaluation of Education Quality within MEC, with an expected learning assessment measurement in secondary education at the end of 2010 and a subsequent one in 2014.

(b) In the area of mechanisms to provide more equitable access to secondary education supported by the project under component C: (i) the scholarship scheme instituted by the project is a recurrent annual activity sustained by the MEC with 10,060 scholarships granted in 2009 financed completely with fiscal resources; and (ii) the two prototype schools are functioning, albeit at a reduced enrollment as envisaged by the appraised design (the school in Caguazú started operations in early 2009 has 133 in first grade and 112 in for a total of 245 students, and the school in Luque started operations in March 2010 with 143 students in first grade).

The MEC’s authorities informed the ICR team that at least during 2010 the activities financed under component B (textbooks, learning resource centers, science laboratories and the management community- based strategies) are not going to be financially sustained and that the alternative secondary education modality will be suspended once the current student cohort completes their remaining modules at the end of 2010.

3. Assessment of Outcomes

3.1 Relevance of Objectives, Design and Implementation

Project’s Development Objectives were relevant at the time of project design and continue to be relevant now.

At the lending stage. The project PDOs were consistent with those outlined in the Bank’s Paraguay Country Assistance Strategy (CAS) discussed by the Board on March 27, 1997, covering the period until December 2003, including: (i) to increase coverage and quality of secondary education to augment the supply of qualified labor; (ii) to improve quality and efficiency of secondary education; (iii) to increase the MEC’s capacity for information-based decision making; and (iv) to promote decentralized participation of the education community. In addition, the project design was in line with the CAS’s recommendation to focus on poverty issues, in this case, equity in access to quality secondary education.

The project design was also attuned to the Country’s Poverty Reduction Strategy (Estrategia Nacional de Reducción de la Pobreza y la Desigualdad, November 2002), prepared with the support of the Bank. This strategy also highlighted the important relationship between education opportunities and poverty reduction, noting that the extreme poor are those with the least number of years of education and with the 14 most limited access to education. As the greatest number of poor and extreme poor population resides in the rural areas, the proposed operation was also consistent with Poverty Reduction Strategy by assisting the GOP in its efforts of increasing net enrollment rates in secondary education, especially in the rural areas. With these goals in mind, the proposed Bank-financed operation aimed at: (a) improving human capital through provision of education; and (b) strengthening the social capital of the poor by offering greater opportunities for participation in the local education community.

Finally, the PDOs were also consistent with the Millennium Development Goals of eradicating extreme poverty and increasing literacy rates of 15-24 year olds, thereby increasing the country’s possibilities of sustainable economic growth and more equitable income distribution.

At the closing of the loan. The project’s PDOs are also consistent with two of the three objectives set forth in the current Country Partnership Strategy for the Republic of Paraguay (Report 48087-PY), dated April 7, 2009 and covering the period 2009-2013. Accordingly, this CPS sets the direction of Bank support to help the GOP achieve:

(i) Strengthened state institutions to improve policy making, public services provision, and the rule of law; (ii) Improved investments in the social sectors and focusing public spending on the extreme poor; and (iii) Fostered growth with a focus on employment generation and on improving income distribution, without destabilizing the country’s macroeconomic framework or the environment.

3.2 Achievement of Project Development Objectives

Assessment of PDO achievements. This ICR recognizes that the PDO and outcome indicators selected to measure it at the time of project design assumed that both APL phases would materialize. In reality only the first project was completed, and the second stage that intended to scale up its impact to the national level was not carried out. Therefore, it is hard to claim that activities supported by the project had a direct causality link to the intended outcomes expected by the end of APL II. Nevertheless, no separate outcome indicators had been defined for APL I, and only intermediate (output) indicators were tracked during project implementation. This design flaw has a broader implication for the evaluation of the project. The limited attribution between intermediate project outcomes and expected program outcomes makes the process of establishing linkages of causality artificial. This ICR has opted for giving more weight in the rating to the results outcomes as opposed to the intermediate outcomes because (a) outcome indicators better reflect the purpose and intention of the project, and (b) it took as long to complete the first phase as it should have taken to implement the entire program.

Using the data collected during project implementation as well as information from other sources, the following conclusions can be drawn with respect to the achievement of project’s development objectives:

Improved Management and Efficiency of the Education System: not achieved. Indicators that were intended to measure improvements in management and efficiency of the education system are increase in class size, increased gross completion, lower repetition and lower dropout rates. Specifically, between 2001 to 2009 class size in the 1st and 2nd cycle of basic education was supposed to increase from 18 to 28 students, in the 3rd cycle from 29 to 30 students, and in secondary education from 25 to 30 (revised to 27 during implementation). In practice, no increase in class size at any level has been achieved, and at least for secondary level, there has been a reduction in the class size from 25 to 21 students, largely due to union pressures. Gross completion rate at the secondary education level was expected to increase from 76 percent in 2001 to 82 in 2009. The actual outcome (2008 data) has been 69 percent, meaning that the indicator moved in the opposite direction compared to the goal. Even though the average dropout rate has 15 decreased from 11 percent in 2001 to 10.7 percent in 2008, the rate is still above the target for the end of APL I (9 percent). The only indicator where the target has been surpassed is the reduction of the repetition rate which decreased from 1 percent in 2001 to 0.6 percent in 2008 versus a 0.9 percent in the PAD as a target for the end of program.

It should be noted that although the outcome of greater efficiency has not been achieved, outputs meant to lead to that outcome have been successfully delivered. In particular, alternative secondary education programs have been designed and approved by MEC and implemented in 20 sites, but the modality turned out to be cost-ineffective. Only 172 students out of the original 600 that started this modality in 2006 have remained in 17 sites (3 sites have dropped out). From the surviving cohort, only 76 had graduated in December 2009 – January 2010 period, which means that it took them more than three and a half years to complete their secondary education, about 6 months longer than in a regular school. It is expected that the remaining 96 students will complete this level during 2010 (taking them more than 4 years). MEC decided not to continue offering this alternative.

Improved Educational Outcomes: not possible to evaluate. As mentioned before, this outcome indicator was dropped for APL I after the change in methodology made original baseline data incompatible with subsequent assessments. It was decided to use the 2006 SNEPE results as the new baseline to be compared with the next measurement due to take place in 2010 and then again in 2014. By early 2010, MEC had completed the validation of the assessment instrument that will be administered nationwide at the end of 2010 to all graduates of secondary education (grade 12), while the sample – based assessment to grade 11th has been postponed until 2011.

However, all of the outputs intended to bring about better education quality have been achieved. In particular, MEC has designed and approved a model for the selection and contracting of personnel and is using in not just in 3 Departments as originally envisaged, but nationwide. MEC has also surpassed the target on designing certification and accreditation policies, which are now applied to 20 pre- and in- service teacher training institutions. With respect to teacher certification, MEC is currently implementing the approved model nationwide in both basic and secondary education levels, albeit with different degrees of progress (moving faster in basic than in secondary education). A new General Directorate has been established within MEC by the new administration (Dirección General de Evaluación de la Carrera Magisterial then recently renamed Dirección General de Desarrollo Profesional del Educador) to manage the implementation of the above-mentioned activities. MEC has also designed and approved initial teacher training program that is now being implemented in 13 teacher-training institutes (8 more than original agreed at appraisal) with an enrollment of 260 students. The goal on quality improvement projects (PEI) has also been surpassed: not just 50 but all 100 project secondary schools have implemented their PEIs.

Greater Equity of Opportunity in Secondary Education: achieved. Indicators that can be used to gauge progress on the equity front are changes in the net enrollment rate and incremental growth of the number of secondary students from poor families. The target on the net enrollment rate has been met: this indicator increased from 33 percent in 2001to 42 percent in 2008, which has been the target for the end of APL I. This number is somewhat short of the 44 percent expected by the end of the program (2009). In addition, the scholarship scheme supported by the project has proved to be a highly cost-effective and well targeted intervention. The project provided 4,500 scholarships annually to poor secondary education students. Furthermore, MEC has sustained this activity with its own fiscal resources since 2007, granting 4,518 scholarships for secondary education in 2007, 3,000 in 2008 and almost tripling the number to 10,060 in 2009. The size of the scholarship has remained stable at about US$260 equivalent.

On balance therefore, the project’s PDO achievement is rated moderately unsatisfactory.

16

Assessment of the intermediate output indicators. The number of intermediate output indicators reported to Bank management at the end of each supervision mission since the beginning of project implementation only included the - number of scholarships granted by the project.

Table 4 below includes the status of the 19 original output indicators. From the findings indicated in this table, the ICR rates the achievement of the original output indicators as moderately satisfactory.

Table 4. Status of intermediate outcome indicators

Component Output and PAD target Unit Status as of December Achievement 2009 (%) New selection and hiring A. Improved model applied in three states* Model Model applied nationwide 100 tools for National education map 100 planning and percent completed Educational map Completed 100 managing the Database (schools, human education resources, statistics, student Completed 100 sector certification) 100 percent Integrated database integrated Baseline on student Baseline Completed 100 performance Accreditation and Accreditation and certification systems designed certification systems Completed 100 and approved by MEC* Social communication model Social communication agreed upon and in operation model Not completed 0 New initial teacher training New initial teacher training Approved by MEC. curriculum approved by MEC curriculum Implementation in 13 public 100 and introduced in 5 public teacher training institutions teacher training institutions* about to start All of the 240-hour teacher 240-hour in-service teacher training programs for training program Completed 100 upgrading of current teachers completed All 100 participating education leaders with Leaders with a master’s 98 completed 98 master’s degree program degree completed B. Secondary All the 1st-year secondary Completed for 1st, 2nd and school education texts designed Text designed 3rd grades for language improvements (Spanish, Guaraní and English), Math, Social 87.5 Sciences, Arts and Physical Education (with the exception of the science textbooks) At least 75 CRAs and 25 Operating CRAs and science 72 CRAs operating. science laboratories in laboratories 24 Science laboratories operation acquired in the process of 96 being distributed to the schools 100 PEIs in operation in PEIs in operation Completed 100 participating schools* 100 PSPs implemented or PSPs implemented or under Completed 100 under implementation implementation ACE participation in PEI design in 50 percent of the Active ACE Completed 100 selected schools

17

C. Mechanisms Scholarships assigned to at Assigned scholarships Completed 100 to ensure least 4,500 students* equitable A mechanism established to Scholarship scheme as a access to ensure sustainability of the sustainable program Completed 100 secondary program education Existing alternative secondary education program Evaluation study Not carried out 0 (SUMANDO, Fe y Alegria) Modular program designed Designed completed and and initiated on a pilot basis Modular program implemented in 20 sites. Out in 20 locations of the original 600 students 44 that started in the summer (factoring 308 out of 2006, only 172 remain in 17 600 initial students sites, of which 76 graduated and 17 out of 20 in Dec 2009-Jan 2010. MEC original sites) will not offer this modality to a new cohort. Three prototype schools One prototype school Estimated to be 13.3 constructed and fully functioning since February percent in February operational using new Prototype school fully 2009 has 133 students in 1st 2010 (250 in one secondary curriculum operational grade and 112 in 2nd grade school and 150 for a total of 245, while the students in the other second school started out of a total functioning in February expected capacity of 2010 with 146 students in 3,000 students in 1st grade both schools two shifts) * Indicators that are incorrectly also utilized as outcome.

Assessment of the triggers to move to Phase II

(a) The following triggers were achieved fully or even surpassed: (i) the model for selection and contracting of personnel designed, approved by the MEC, and in use in at least 3 states; (ii) the certification and accreditation policies designed and approved by the MEC and implemented; (iii) the design of an initial teacher training finalized, design and implementation plan approved by the MEC, and new program introduced in 30 percent of the official training institutes (5 institutes); (iv) 50 percent of the PEIs in place in selected schools (100 schools); and (v) at least 4,500 scholarships assigned, a strategy established and approved by the MEC to assign funds exclusively for scholarships, thereby ensuring sustainability of the program;

(b) The following trigger was only partially achieved: (i) existing alternative secondary education programs evaluated (never done) and modular program designed, design and implementation plan approved by the MEC and program initiated on a pilot basis in 20 locations (implemented in 17 of the original 20 sites, but the model is rated unsatisfactory); and

(c) The student classroom ratios moved in the opposite direction than expected at appraisal for reasons described elsewhere in the ICR.

Accordingly, the ICR rates the attainment of the agreed Phase I triggers moderately satisfactory.

3.3 Efficiency

The Net Present Value (NPV) was US$3.8 million and the estimated Economic Rate of Return (ERR) was 15 percent, considering a 10-year time horizon since the beginning of the project. The estimated NPV and ERR for a 20-year time horizon since the beginning of the project were US$ 25.4 million and 33

18 percent, respectively. The Cost-Benefit Analysis was re-estimated to calculate the NPV and the ERR using end of the program actual values.

The estimates indicate a negative NPV of US$ 1.4 million and a revised ERR of 8 percent. Under this scenario (10 years) the project could not be considered a good social investment. However it is important to highlight two aspects. First, the Cost Benefit Analysis model used at project preparation did not quantify the benefits associated with a more educated labor force. These benefits may have two types of effects: (i) productivity and growth effects above the direct income increases that beneficiaries are receiving, and (ii) lower risks of unemployment that more educated people face. Additionally, intangible benefits were not included (e.g. aesthetic appreciation, better health, and better voting behavior and social behavior that research has linked to increases in education). If these benefits would have been quantified and included in the analysis at appraisal the expected direction of the effects would be an increased NPV and an increased ERR.4 Second, a longer time horizon would change this result. For example, using a 15 year time period since the beginning of implementation would make the project to break even5 using end of program values and considering a 20 year time period the estimations indicate a revised NPV of US$600,000 and a revised ERR of 11 percent.6

Based on the above, the ICR rates the efficiency of this project as moderately satisfactory.

3.4 Justification of Overall Outcome Rating

Given that only one of the three development outcomes of the project has been achieved (greater equity of access to secondary education), one outcome (learning outcomes) is impossible to evaluate, and one (management and efficiency of the system) has actually deteriorated, the overall outcome is rated moderately unsatisfactory.

An important caveat to this rating is that the objective was to be achieved as a result of two APL phases, while in practice only one of the operations materialized. Nevertheless, given the fact that it took nearly as long to complete the first phase as was envisioned for the entire program, it seems appropriate to evaluate the project against the original program outcome indicators. If favorable data were available on changes in education outcomes, it would have been possible to rate the project outcome as moderately satisfactory.

3.5 Overarching Themes, Other Outcomes and Impacts

(a) Poverty Impacts, Gender Aspects, and Social Development

Out of the three key activities included in the project design under Component C to improve equity in access, retention and completion in secondary education, only the scholarship program was fully and successfully implemented and targeted. Two other project interventions also aimed at reducing poverty - the prototype schools and the alternative secondary model. The prototype schools, although completed, are recently in operation with about 10 percent capacity of their original design, while the alternative secondary models has not yet completed the graduation of its present student cohort and presents efficiency problems (high dropout rates and of significantly longer duration than the school-based modality). Given that the total number of scholarships provided to poor students annually represents only

4 The analysis conducted in this report did not make an attempt to value these benefits. 5 That is the project will have a NPV = $0 and an ERR = 10 percent. 6 The selection of the time horizon is an option of the analysis but it is expected that the benefits of these intervention are closer to 15 years since the beginning of the implementation of the project rather than the 10 year considered in the PAD. 19 about 5 percent of the total enrollment in public secondary schools,7 the impact of this intervention in alleviating poverty is moderate.

(b) Institutional Change/Strengthening

Even though the original project design was ambitious with respect to making structural institutional changes with the re-engineering and support for the decentralization of the MEC, in reality only modest changes were accomplished with almost no visible impact in the short term. However, the new selection and recruitment of teachers, the certification of teachers and the accreditation of pre and in- service teacher-training institutes will, in the medium to long-term horizon, improve the quality of the teacher cadre. The availability of updated statistical education information, school-mapping and learning outcomes could also improve the decision-making process to better allocate the limited financial, human and physical resources within the education system.

There is no hard evidence that the graduates of the leadership program have made an impact in promoting constructive changes within their working environment. There is a long way still to go before management reengineering will bear fruit. Finally, there is no institutional strategy to concentrate teachers’ working hours in fewer secondary schools, so that they can have more time to participate in strengthening teaching and learning activities in the classroom implemented at the school level.

(c) Other Unintended Outcomes and Impacts (positive or negative)

Not applicable.

3.6 Summary of Findings of Beneficiary Survey and/or Stakeholder Workshops

Not applicable.

4. Assessment of Risk to Development Outcome

The project includes three development outcomes: enhanced education system management and efficiency (increased class sizes, improved gross completion rates and lower repetition and dropped out rates); improved educational quality (as measured by standardized tests); and equitable access to secondary education (as measured by increased enrollment of the poorest).

The Development Objectives were relevant at the time of project design and continue to be relevant now for the current administration. At such, efforts will be continued to achieve these outcomes.

However, there are some risks: first, the reengineering program completed at the end of 2009 will be gradually scaled up to the rest of the Departments in the country. However it will take some time and additional fresh financial resources and hardware before MEC reaches this threshold to have an impact on the education system management.

Second, the standardized learning assessment SNEPE to measure impacts on quality, a crucial monitoring and evaluation task, is being institutionally, technically and financially sustained by the MEC by housing it in the recently created Dirección de Evaluación de la Calidad Educativa (DECE) within the General

7 A rigorous impact evaluation study conducted on the scholarship program (World Bank, 2007) showed that the program increased the probability of beneficiaries’ attendance and graduation from secondary education as compared to a carefully selected group of non-beneficiaries. 20

Directorate of Planning. This unit has been strengthened with qualified staff to write and validate test items as well as to process and analyze the field information. The MEC is still dependent on external financing is to conduct the administration of tests and dissemination of results. In terms of inputs to produce quality of education, the Paraguayan Congress did not approve in the 2008, 2009 and 2010 Budget Laws financial resources for the reprinting of the textbooks acquired under the project neither for the acquisition of other learning goods (laboratories, learning resource centers).

Third, in terms of equity of access the scholarship scheme is currently funded by the GOP and the authorities are committed to maintain the funding for this program.

Accordingly, the ICR rates the risk of this activity as moderate.

5. Assessment of Bank and Borrower Performance

5.1 Bank Performance

(a) Bank Performance in Ensuring Quality at Entry

Rating: Moderately unsatisfactory

The Bank provided effective project preparation support to the GOP through a highly qualified and experienced core Bank team. When specific technical issues required attention, the Bank team drew on senior expertise from across the Bank and from outside the Bank. There was only one task team leader assigned to this operation during the entire lending cycle (from pre-identification to Board approval) thus providing continuity during the preparation phase. As shown in Annex 4 of the ICR, a significant number of professionals were involved in the preparation cycle.

The quality at entry of this project is rated as moderately unsatisfactory because of the following:

On the positive side the analysis performed for this report indicates that: (i) the strategic relevance and approach is considered appropriate because it was nurtured by a vast array of useful information stemming from different sources; (ii) the decision to support the GOP’s public secondary education system with this operation was considered an appropriate strategic decision by the Bank at the time of project preparation; (iii) an APL was considered an appropriate financial instrument; (iv) the GOP had full ownership of the project design; (v) the Bank Team demonstrated client orientation; (vi) the Bank team developed good working relationships with the IDB, who was supporting other levels of the education system, and elicited the participation of several NGOs in capacity building strategies for the implementation of some project activities; (vii) the extreme care taken by the Bank team during preparation to ensure appropriate poverty targeting of scholarship scheme and transparency and accountability in the selection of candidates; (viii) the implementation arrangements included for this project were appropriate; (ix) there was a compliance with Bank’s safeguard policies; (x) the Bank team presented appropriate risk assessment highlighting some potential risks and proposed some mitigating measures; and (xi) the project introduced appropriate lessons learned.

On the downside: (i) the short time horizon of the APL, the limited amount of financial resources allocated to the first phase and the restricted geographic scope of the project covering only 8 out of 17 Departments of the entire country, and within each project Department, only few secondary schools, prevented this operation to have a significant impact in achieving the policy reforms sought by the GOP in secondary education; (ii) the project design was overly ambitious covering many fronts; (iii) some components, like the alternative secondary education, were included in the design lacking a technical and

21 operational description at the time of appraisal; (iv) there was a high degree of lack of articulation among the different activities financed by the project; (v) the poorly defined results framework; (vi) a major drawback during preparation was the lack of a cost-benefit analysis or an assessment of potential demand to justify the inclusion of three large prototype schools to be financed by the Bank loan;8 and (vii) implementation readiness conditions were not fully present at the time of Board approval causing implementation delays.

As stated above the decision to build the large prototype schools has not been based on good analysis. Smaller secondary schools, holding a maximum of about 500 students and with about one sixth the square meters of the prototype schools9 would have been a more efficient way to improve access and retention, and learning outcomes. In addition, high cost of these schools effectively prevented their replication and expansion elsewhere in the country after the closing of the loan. This alone should have been a sufficient reason never to include this intervention, as it defeats the principle of sustainability and scaling up of successful interventions. Finally, the cost of annual maintenance estimated at about 2 percent of the total cost of construction (US$ 50,000 per school per year) was not considered in the design.

Rating criteria. Weighing the relevant achievements and shortfalls of the Bank team at the preparation stage, the ICR rates the Bank project preparation performance as moderately unsatisfactory.

(b) Quality of Supervision

Rating: moderately unsatisfactory

The project had three Task Team Leaders (TTLs). Continuity was provided during the entire preparation and implementation cycle, ensuring a smooth and effective change of command from the Bank’s side.

The Bank carried-out on-average about one supervision every five months and supervision was adequate and effective. Supervision missions included the appropriate skill-mix to carry out their envisaged terms of reference as attested in Annex 4 of the ICR. The project files also confirm the support provided by the various missions to the counterpart team including: (i) a procurement and financial management-training workshop during the launching mission in March 2004 to familiarize the MEC/PCU teams with Bank procedures; (ii) the identification of a set of qualified consultants to provide technical assistance in the redirection of the alternative secondary modality; (iii) advice to the MEC/PCU team to better articulate all the components seeking improved synergy and the usage of cost-benefit consideration of the different project activities; and (iv) provision in May 2008 of a qualified civil engineer specializing in structures to assess the collapse in part of the ceiling of one of the prototype schools and to recommend ways to address this issue.

Supervision missions were candid in raising issues, and in some occasions, also flags, like for example: (i) the inappropriateness of the original alternative secondary model; (ii) the cancellation of one of the three school prototypes; (iii) the weaknesses of the in-service teacher training model; and (iv) the unresolved safety issues during the construction of the prototype schools. The reporting during the early stages of implementation lasting up to mid-2006 focused mainly in inputs and processes, while from mid- 2006 onwards, and especially once the sector work was completed and discussed with the MEC in 2007, the reporting was more holistic in nature focusing on results. Accordingly, supervision missions tried to strike a balance by looking at intermediate and final outcomes with inputs and processes. Likewise, some of the early supervision reporting is based on expectations (like for example with respect to the alternative secondary option up to and including 2006, the in-service teacher training up to early 2007, the prototype

8 During implementation only two of these prototype schools were built. 9 The size of the prototype schools built was approximately 6,600 square meters. 22 schools and the leadership program), while the reporting in the latter stages of the implementation cycle is based on more solid evidence and facts of the grounds.

On the downside, nothing was done by the TTLs to correct the project’s inadequate results framework during phase I.

Another major drawback was the 2006 approval of the increase in the size of the schools from 3,700 to 6,600 square meters (78.4 percent increase in square footage).10 Again no evidence can be found of a cost-benefit analysis or assessment of potential demand to justify such an increase. This lack of assessment is evidenced by the current underutilization of the schools (only 245 students in the prototype school in Caaguazú and 146 students in the school in Luque).

The construction of the two prototype schools built with loan proceeds has been the source of constant and significant criticism to the Bank, especially from the Paraguayan press and the newly elected government who sees these schools as “white-elephant projects” not meriting the financial investments allocated to them. During 2008 and 2009 the Ministry of Education had to address safety issues in both prototype schools to rectify omissions in their design. The Ministry also designed and distributed brochures for a safe use of the schools.

Weighing the above, Bank supervision performance is considered as moderately unsatisfactory.

(c) Justification of Rating for Overall Bank Performance

Rating: moderately unsatisfactory

The ICR rates the overall Bank performance as moderately unsatisfactory because most of the project preparation gaps were satisfactorily addressed by the MEC during implementation.

5.2 Borrower Performance

(a) Government Performance

Rating at the lending stage: moderately satisfactory

The project preparation was executed by line staff from MEC complemented with technical assistance from the MEC’s PCU. The political context provided needed stability of the technical counterpart teams that prepared, and then implemented Phase I until the change of government in August 2008.

The Bank team interacted with highly qualified technical counterpart teams, most of which were also involved in the implementation of the previous Bank-financed project in education (MECES, Ln. 39410 – PY) and thus familiar with Bank-financed operations. These teams had clarity about the key issues affecting secondary education as well as with respect to the relevant interventions to address them. In addition, these teams were very receptive to the recommendations provided by international experts that attended various thematic workshops during the preparation cycle.

On the downside, the MEC/UCP teams responsible for the various components tended to work in isolation, precluding the needed coordination among project components, as was the case with the

10 Because of the replicability of the model, the maintenance cost and funds availability to fund the increased cost per school, the number of prototype schools was reduced from three to two in December 2006. 23 prototype school buildings explained before. Another weakness that surfaced during project preparation concerns the alternative secondary education activity which was included in the design without a thorough conceptualization by the MEC/PCU.

Weighing the strengths and the weaknesses at the preparation stage, the ICR rates the Government performance during the preparation stage as moderately satisfactory.

Rating at the implementation stage: moderately satisfactory

The Executive and Legislative branches of the GOP were fully supportive of this project until August 2008, at which time a new elected government assumed responsibility. The loan was signed on September 10, 2003, about one month after Board approval. The Paraguayan Parliament approved the loan about nine months after the signing which is a remarkably short time for Paraguay. In addition, Congress approved the regulations to the Teacher Statue enabling teacher certification as intended by the project.

The implementation cycle enjoyed the benefit of a stable administration at both the MEC and the PCU until the change of government in August 15, 2008. This situation of stability trickled down to the rest of the frontline units within the MEC involved in project implementation, as well as with the key staff at the PCU.

On the downside, outstanding repayments by the GOP on another Bank-financed operation not related to this project (the Eight Highway project) resulted in a 3 months delay in disbursements.

Another GOP shortcoming concerns the lack of Congress approval to MEC’s request to replenish secondary education textbooks and learning materials in the 2008, 2009 and 2010 budgets.

Rating: For the above reasons, the ICR rates the Borrower performance at implementation as moderately satisfactory.

(b) Implementing Agency or Agencies Performance

Rating: moderately unsatisfactory

This project was implemented by the MEC with support from the PCU. During the entire implementation cycle there were effective linkages and excellent working coordination between the MEC, in charge of project’s pedagogical-related matters, except the infrastructure portion, and the PCU in charge of the fiduciary aspects and provision of qualified technical assistance.

There was strong and decisive leadership from senior level officials in the MEC, in particular the Minister and Vice-Minister of Education and the heads of the following MEC’s frontline units which became key stakeholders in the project’s implementation cycle: (i) the General Directorate of Planning (in MEC’s reengineering through improved tools for planning and management); (ii) the General Directorate of Secondary Education (championing the entire pedagogical scheme); (iii) the General Directorate of Educational Development; (iv) the Directorate of Evaluation (SNEPE); (v) the General Directorate of Higher Education (in the Pre and in-service teacher training and accreditation of teacher training institutes); (vi) the Directorate of Curriculum (in the establishment of technical specifications for the learning resources and textbooks); (vii) the Directorate of Scientific Upper Secondary (in the alternative secondary education modality); and (viii) the Directorate General of Administration and Finance (in the teacher certification and recruitment processes).

24

The PCU mirrored the stability that was present in MEC during the entire implementation cycle until August 15, 2008. From loan effectiveness until September 2008, the PCU had only two coordinators. The tenure of the first one lasted from January 2004 to the end of May 2007, and the second one from June 2006 until September 2008. This situation benefited the entire implementation process and contributed to a learning curve, which, in time, resulted in a more consistent financial and physical implementation pattern. A third PCU coordinator was nominated by the new administration in December 2008 with the task to complete the first phase by December 30, 2009.

The education authorities of the new administration decided to streamline and overhaul the technical staff at the PCU. With the exception of the PCU’s financial coordinator, key PCU staff was replaced by Ministry of Education personnel. Partly because of these changes, the implementation of some activities, in particular civil works and the procurement of school furniture and learning goods for the two prototype schools as well as for the hardware required in the MEC’s reengineering process, suffered delays. As a consequence, the civil works at the prototype schools were completed and the schools partially equipped only at the end of 2009 (see Annex 2 for further details). This required an exceptional third extension to the project closing date.

All the supervision missions benefited from meticulous and detailed progress reports prepared by the MEC/PCU for all the 17 key activities comprised in the 7 project sub-components and 4 components including the follow up on agreements reached in previous missions.

On the downside there were significant delays in carrying out the procurement processes for textbooks, construction of the prototypes schools and CDEs and in selecting and contracting the consulting firm to carry out the reengineering to improve planning and management of the MEC. These delays were partly caused by the MEC’s internal review and oversight processes to ensure transparency in all government contracts and also to the fact that some procurement processes had to be declared void due to lack of interested bidders. These delays also resulted in the request by GOP of three extensions to the closing date, which were authorized by the Bank.

Weighing the strengths and weakness manifested by the implementing agency (MEC/PCU), especially with respect to the procurement delays and the lack of impact in improving the efficiency of MEC’s management, but more particularly, the delay of ongoing project activities, the ICR rates the implementing agency performance as moderately unsatisfactory.

(c) Justification of Rating for Overall Borrower Performance

Rating: moderately satisfactory

According to current ICR rating guidelines, the overall Borrower performance is rated moderately satisfactory because the ratings of: (i) the government performance at the preparation stage was moderately satisfactory; (ii) the government performance at the implementation stage was also moderately satisfactory; (iii) the implementation agency performance was moderately unsatisfactory; and (iv) the outcome (section 3.2 of the ICR) was rated moderately satisfactory.

25

6. Lessons Learned

(a) Political support and ownership is a key condition and a good predictor for success. The stability existing within the MEC authorities throughout the entire lending and implementation cycles (up to August 15, 2008) created the required conducive environment by limiting shifts in education policy and implementation strategy and of changes in key implementation players. On the other hand, the revision and appropriation of the ongoing project by the new education authorities resulted in a temporary delay of project activities.

(b) Ensuring implementation readiness conditions prior to negotiations leads to a more effective and timely physical and financial execution cycle. The delays in physical and financial implementation experienced by this project, especially in the period 2004-2006, could have been mitigated, had the Bank team and management ensured appropriate implementation readiness conditions at the time of appraisal, especially in the case of the acquisition of learning goods, construction of schools and CDEs, the alternative secondary education modality and the in-service teacher training, just to mentioned some examples.

(c) Appropriate results framework is needed to inform the Borrower and Bank management about progress towards achieving the expected project developmental objectives. First, the number of outcome and output indicators has to be reduced to only a few ones to facilitate their timely updating, reduce the cost of gathering and processing information and improve the relevancy of the information (good practices recommend between three to four outcome indicators and a maximum of 3 output indicators per project component). Second, there should be connectivity between the stated PDOs, the outcome indicators being selected and the project design. Third, appropriate attributions should be ensured to insulate changes in outcome indicators by actions outside the scope of the project.

(d) The logical chain that links project activities and outputs to the project outcomes should be clear and include all factors that are present in that chain. Even though project outputs – which are under the influence/control of the project - have been achieved the logical chain may have overlooked other factors that are important to produce the desired project outcomes. Even though outputs have been achieved the desired outcome may not achieved due to incomplete logical chain.

(e) Cost-benefit analysis is required during the design of a project to avoid the inclusion of cost- ineffective interventions - like the prototype schools - and have the appropriate technical arguments to debate design options.

(f) Complex project designs are likely to delay the physical and financial implementation progress. The original project design included too many dimensions (17 different key activities packed in 7 sub-components and 4 components), whose effective and timely addressing became an issue throughout the entire implementation cycle. This would have required an early restructuring to simplify the design based on the implementation progress evidence on the ground.

(g) Local stakeholder participation is important to improve quality and efficiency at the school- level. Ownership of the improvement process by stakeholders involved in deciding and implementing their own interventions is a necessary, although not sufficient condition for effective change. Despite that no impact evaluation was undertaken on the effect of the PEIs, PSPs and ACEs, it has been proven elsewhere and documented internationally, that community mobilization in a conducive and decentralized environment greatly contributes to facilitate the implementation of quality and efficiency measures. Furthermore, if school principals that are natural leaders facilitate this local participation process and participating teachers have most of their teaching load assigned to one particular school, the impact in the

26 envisaged changes are even stronger. This conducive environment was not present in a significant number of the participating secondary schools in the project.

(h) The choice of targeting approach is relevant for equity, transparency, program ownership and effectiveness. The scholarship scheme is a good example of: (a) a targeted intervention that was directed to poor secondary education students; (b) an operation that included transparent selection criteria and was accountable to society; (c) a cost-effective investment as proven by the impact evaluation carried out and reported accordingly in the sector work; and (d) a sustainable activity as proven by GOP’s decision to make this a regular program within MEC’s annual budget.

(i) Piloting in a small-scale prior to scale up contributes in the identification and addressing of implementation obstacles when introducing new paradigms, especially in entrenched systems, like in the case of the scholarship scheme.

(j) Impact evaluation included at the onset of project implementation and monitoring are needed as a means to correct course if it becomes necessary. They also provide appropriate information to test alternative mechanisms, prior to scaling-up. The scholarship program is a good example. Unfortunately, for reasons already explained, the leadership program, the in-service teacher training and the school-community based strategies were not assessed with respect to their possible contribution to the attainment of the project’s PDOs.

7. Comments on Issues Raised by Borrower/Implementing Agencies/Partners

1. Borrower/implementing agencies

(i) What the project proposed to accomplish

The project design aimed to achieve improvements in the education system efficiency, in the quality of education and in increased access, providing financial and technical resources for a set of strategic initiatives. The objectives were defined in line with the following: (a) improve education system management; (b) improve secondary education quality; and (c) expand access to secondary education.

(ii) What was achieved and what was not achieved - main causes

In terms of coverage, the enrolment rate increased significantly from 44 percent to 53 percent in the 2001- 2006 time period. However, this increase is still insufficient. The persistence of internal efficiency problems in the education system, the worsening of socio-economic conditions of the students, and the insufficient supply of education services are some of the main barriers to expand access to secondary education.

With respect to learning quality, SNEPE results showed limited improvements in education achievement for secondary students. Even though the project design included actions to improve learning quality, the delayed implementation and the lack of coordination of these actions does not allow estimating the impact of the interventions yet.

Some factors affected project implementation, including among other the following: delayed in implementation, problems with the original design, exchange rate variations, expansion of textbooks provision to the three grades in secondary education (10, 11, and 12), financial management, use of the loan’s availability balance, under-estimation of the costs of the civil works, change of government, etc.

27

(iii) Lessons learned

Although the goals for components and subcomponents have been achieved and, in some cases, the results have been greater than expected at appraisal, some lessons have been learned from this experience that can be useful for future projects:

1. The construction of a clear, articulated and concrete logical framework for the project, with objectives and results and impact indicators. The logical framework should be constantly supported by the monitoring and evaluation system to sustain continuous decision making and the introduction of adjustments during implementation; 2. The importance of having technically qualified and stable staff in the MEC line units involved with the implementation of the project. The staff should be committed to achieve the objectives for each activity in the project and should be accountable for its results. This way project sustainability would be technically guaranteed through the human resources capacity available at MEC. Staff should have access to updated information about the project budget to improve its efficiency, rationalization in the usage and optimization of financial resources; and 3. The intervention in the third cycle of the basic education (grades 7 – 9) is crucial to increase coverage in secondary education, one of the project’s goals.

(iv) Relationship MEC/PCU (Program Coordination Unit) with the Bank

1. The World Bank’s Team and technical staff working for the project were open to continuously evaluate the implementation progress; 2. Even though the relationship MEC/PCU with the Bank was in general satisfactory, the implementation of project activities and their effectiveness to achieve the envisioned goals were affected by local rules both at the MEC, the central level and the Minister of Finance; and 3. Finally, the lack of staff stability in the key areas of the PCU created delays in the implementation of the planned project activities.

(b) Cofinanciers

Not applicable

(c) Other partners and stakeholders (e.g. NGOs/private sector/civil society

Not applicable

28

Annex 1. Project Costs and Financing

(a) Project Cost by Component (in USD Million equivalent)

Appraisal Estimate Actual/Latest Estimate Percentage of Components (USD M) (USD M) Appraisal (US$ million) 1. Improved tools for planning and managing of the education system 8.86 8.08 91.20

2. Secondary school improvements 8.40 7.68 91.31

3. More equitable access to secondary education 7.80 8.83 113.21

4. Project management 1.20 2.28 190.00

Total financing required 26.26 26.87 102.32

Front-end-fee 0.24 0.24 100.0

Total 26.50 27.11 102.30

(b) Financing

Appraisal Estimate (USD Actual/Latest Estimate (USD Source of Type of Percentage of M) M) Funds Cofinancing Appraisal (US$ million) (US$ million) IBRD 24.00 22.80 95.00 GOP Counterpart 2.50 4.31 172.40 Total 26.50 27.11 102.30 The front-end-fee of US$240,000 is being tallied in the GOP row.

(c) Disbursement Profile

29

Annex 2. Outputs by Component

This section describes the outputs of each of the four components, seven sub-components and 17 key activities included in the original project design as well as the sources of financing. Two cost tables are presented to substantiate the story provided in the ICR: (a) the documented expenditure by component and by source of financing at appraisal and at the time of the closing of the loan; and (b) the documented expenditure by category and by source of financing at appraisal and at the time of the closing of the loan.

Component A – Improved tools for planning and management of the education system. According to Table A2.1 below, this component estimated to cost US$8.86 million (US$8.52 million from the loan and US$0.34 million from GOP), ended up documenting expenditures amounting to US$8.08 million (US$6.81 million from the loan and US$1.27 million from GOP), 91.2 percent of the total estimated at appraisal. This slight financial shortfall is mainly explained by the significant delays in implementing a more limited version of the management re-engineering of the MEC as well as the pre and in-service teacher training models developed by the project and the unsatisfactory attainment of the social communication activities. The following is the assessment of the 7 key activities included under this Component A.

Enhancement of macro and micro-planning. The implementation of this activity under Component A is rated satisfactory. The project was able to contribute technically, although not financially, to the implementation of a competency-based screening and selection process of new teachers that has been institutionalized in MEC since 2004. This scheme has provided transparency and accountability into the hiring of new teachers for the public school system.

In addition, the statistical management information system, as well as the micro-planning (school mapping) units in the central MEC and all the UDEs in the country were strengthened, including the provision of the required training for the staff. All the educational statistical data was validated. As a consequence, MEC is able to produce on an annual basis a timely and reliable educational statistics of the entire country disaggregated by different concepts. The official statistics for the school year 2008 is available, and a preliminary version for the school year 2009 is ready. This information is key for the decision-making processes. In addition, updated school-mapping information, to balance the supply of available school capacity at all levels of education with the projected demand for services, is available for the entire country.

Management re-engineering. The management re-engineering process financed by the project was envisaged as the new tool to enable a more effective management of the education sector in the entire country linking in an articulated and timely fashion the personnel and financial resources and student certification management information systems. The target set at appraisal for this process was to have this system designed, validated by the MEC’s key stakeholders (Dirección General de Talento Humano and the Dirección General de Administración y Finanzas) and fully operational in the central MEC and all its supervisory coordination units of the city of Asuncion at the closing of the loan. This target was partially accomplished. By late 2009 the system was designed, validated and set in motion in only two areas of central MEC (the Dirección General de Administración y Finanzas and the Dirección General de Talento Humano), but not in the supervisory coordination units of Asuncion. It is expected to have the system fully operational in the central MEC and all the supervisory coordinations of the city of Asuncion during 2010.

The ICR identified three main reasons for this shortfall - considerable delay in selecting and contracting a qualified consulting firm, limitation on the availability of project-related financial resources and the

30 arrival of new authorities that reviewed the procurement process for the acquisition of a server and required computers for almost 7 months after taking charge of the MEC ending canceling this intended purchase.

It took almost three years after Board approval to complete the selection process for a qualifying consulting firm in 2006 through an international competitive bidding process. Unfortunately, the selected firm backed out at the last moment due to a refusal by the MEC to a request of the firm to renegotiate the price after the selection process was completed. It seems that the consulting firm underbid its pricing to win the competition and then tried to renegotiate the price once it was selected. A decision was then made by the Borrower, and agreed with the Bank, to offer the contract to the firm that came second in the selection process. This new consulting firm signed the contract with MEC in March 30, 2007. By December 31, 2009, the development of the system had been completed and validated in two areas of the central MEC; (ii) a provisional Data Center had been established for the validation of the system; (iii) the migration of the old data base to the new system had been completed; and (iv) all the procedural and organizational Manuals were completed, revised and approved by MEC.

In addition, the reallocation of loan proceeds to cover significant increases in the civil works category of expenditure, the appreciation of the Guaraní on a US dollar-based loan with expenditures in local currency and a higher than expected cost for this key project activity as compared to the projected costs at appraisal, limited the availability of resources to acquire the required hardware to implement the re- engineering process and pilot it in the central MEC and the city of Asuncion. An international procurement process launched by the outgoing administration in late July 2008 to acquire the server and computers to validate and the implement the system was declared void in November 2008 by the incoming administration (mostly for political reasons but arguing that none of the bidders satisfied the eligibility criteria). MEC ended up supporting the validation process with limited existing hardware (server and computers), far short of the amount required by the project design.

The ICR has therefore no factual evidence concerning how this envisaged re-engineering scheme is going to be used to achieve the stated objectives of increased managerial efficiency, effectiveness, opportunity and transparency through a more effective articulation of the various management information systems. Accordingly, and balancing the achievements with the shortfalls, especially the purposeful stalling of an ongoing procurement process, the ICR ranks the accomplishment of this line of activity as moderately unsatisfactory.

Construction of CDEs. As explained in section 1.6 of the ICR, although it was not made explicit in Schedule 2 of the LA, during the implementation cycle, the Bank agreed to support the construction of 5 CDEs out of the 8 considered by the MEC in the Departments of Caacupé, Caazapá, Concepción, Encarnación and Filadelfia. All the CDEs, with the exception of the one in Caazapá, have been satisfactorily completed and received by MEC, although with significant delay with respect to the contracted timetable. By December 31, 2009, the construction firm in charge of Caazapá had not been able to comply with the observations made by the fiscalizadora (on-site supervision) preventing the MEC to consider this work as completed. For this reason and the delays incurred in the completion of the other CDEs, the ICR ranks this activity as moderately satisfactory.

Design and approval by MEC of an accreditation system for pre-service and in-service teacher-training institutes and a certification system for MEC professionals, administrative staff and teachers. The ICR rates this activity under Component A as satisfactory as it timely achieved and surpassed its expected targets set at appraisal. This activity comprised two models. The first model to accredit and certify teacher-training institutes was completed and approved by MEC under a formal Ministerial Resolution (number 24,237) dated November 14, 2007. An Operational Manual was completed and approved by MEC. This model is currently being applied in 20 teacher-training institutes. The National Agency for

31

Evaluation and Accreditation of Higher Education (Agencia Nacional de Evaluación y Acreditación de la Educación Superior) has been charged by MEC as the official accreditation institution.

The second model to assess and certify professional educators was also completed and approved by MEC. An Operational Manual has also been completed, approved and disseminated by MEC. These were the targets set for this activity at appraisal and as such they have been attained. Furthermore, and despite the political complexities, at the time of the writing of the ICR, this model is being applied to teachers of basic and secondary education, albeit that the process is moving much faster with the basic education teachers than with the secondary education teachers.

Both activities have been institutionalized under a recently created General Directorate for Teacher Professional Development (Dirección General de Evaluación de la Carrera Magisterial recently renamed Dirección General de Desarrollo Profesional del Educador).

The pre and in-service teacher training activity financed by the project surpassed its expected target set at appraisal. The pre-service teacher training design (in the areas of basic and social sciences, Spanish, Guaraní and mathematics) was finalized and approved by MEC and introduced in 13 training institutes, way above the target included in Annex 1 of the PAD (30 percent of the official training institutes meaning 5). This design aims at modifying the traditional subject-base pre-service training for secondary education teachers to one based on areas/competencies to conform with the education reform that modified the curriculum to one of learning areas. A total of 260 students (20 per institution) started their pre-service training in the second semester of 2008 and are expected to graduate in the second semester of 2010.

Despite the achievement of the target, the ICR detected several weaknesses with the above-mentioned model that need to be addressed urgently by MEC. By and large, the teacher trainers that are supposed to deliver this model in the participating teacher-training institutes have very little or no experience at all teaching the new secondary education curriculum (based on learning areas), because some of them have never taught at secondary schools or have taught under the dated subject-based curriculum. The likelihood that this new model might be somehow disconnected to the realities of secondary schools is not minor. For this reason, these teacher trainers were provided additional training beyond the 240 hours already delivered by the model (360 hours for reinforcement and 600 in the curricular area of interest). The model urgently also requires a reality check through on-site follow up to ascertain if trained teachers are applying the training received in their classrooms and assess if the model really addresses teaching demands in secondary schools.

This activity also included in-service teacher training (locally called reconversión or plan transitorio) to update secondary teachers that were prepared under the dated subject-based curriculum into the current area/competency-based curriculum. There are different training models developed under this project activity. One, covering 240 hours, was initially provided to about 178 teacher trainers working in the new pre-service teacher training described in the paragraph above, followed by two more reinforcing training of 360 and 600 hours each. Other training models combining a face-to-face classroom modality with distance education included: (i) one comprising 1,300 hours benefiting teachers familiar with the area of their teaching but lacking teaching pedagogical skills (about 12 percent of the total universe of teachers); (ii) another, comprising 1,800 hours for teachers not familiar with the area assigned to them to teach in secondary schools, conforming about 85 percent of the total universe of teachers; and (iii) one comprising 2,300 hours for specialized educational technicians working in secondary schools (about 3 percent of the total universe). At the time of writing this ICR, about 1,730 teachers are undergoing the reconversión scheme in any one of the three modalities described above. In addition, this activity also provided training to 240 teachers and technical staff in the use of the secondary education textbooks acquired and

32 distributed under the project. A total of 34 teacher-training institutes are participating, of which 20 benefited with CRAs acquired under the project.

The ICR notes several shortcomings with this in-service teacher-training model. First, the dropout rate is high in the order of 17 percent due to the following reasons (just in the last semester of 2009 about 10 percent of the participating teachers dropout from the training). Teachers do not receive any financial aid for transportation to and from the in-service teacher-training sites, some of them located far away from the participants’ home and/or school of work. The long duration of the training models which oscillate between 1,300 and 2,300 hours distributed in weekend classes. Training is provided during the weekends disadvantageously competing with the teachers’ free time. There is no field verification if trained teachers are in fact changing teaching practices. Out of the 36 modules required by the in-service teacher training only 12 have been completed at the time of writing this ICR.

Despite that these activities achieved or over-achieved their stated targets, the ICR rates this activity as moderately satisfactory mainly because there has not been a field verification to ascertain if the in- service teacher training model financed by the project has really achieved its objectives of changing teaching practices and improving the teaching quality of secondary education teachers attuned to the new area/competency-based curriculum.

Expansion of the student performance assessment system (SNEPE) for secondary education in the areas of language, mathematics and critical thinking. The ICR considers the implementation of this activity under Component A as highly satisfactory. In 2002, SNEPE established a learning assessment baseline data to gauge future quality improvements based on the findings of the 2001 measurement. However this baseline measuring learning on a content-based curriculum was not going to be comparable with subsequent measurements measuring learning on a competency-based curriculum. Accordingly, a decision was taken, and agreed with the Bank, to replace this baseline with the findings of a measurement undertaken in 2006 on a sample basis of about 13,000 second grade secondary students of 378 secondary schools (public and private, urban and rural) including the 100 project schools with PEIs. This measurement will be comparable with the one to be carried out at the end of 2010 and then again in 2014.

An analysis on the determinant factors of learning was carried out with the 2006 baseline. Under SNEPE, the MEC has installed some initial, although still insufficient capacity to continue writing and validating test items, administering the tests, processing and analyzing the field data and synthesizing the findings to support decision making-process in MEC. Finally, SNEPE has been institutionalized by the incoming education authorities under the Dirección de Evaluación de la Calidad Educativa (DECE) depending of the Directorate General of Planning within the MEC. Finally, at the time of the writing of the ICR, MEC had just completed the validation of a testing instrument to be administered to all the graduates of grade 12 at the end of 2010.

Leadership program. This activity achieved its stated target set at appraisal ensuring that 98 professionals of MEC out of 100 that started completed their leadership program in two qualified universities selected for this purpose following Bank procedures – the University of San Andres in Buenos Aires, Argentina and the Alberto Hurtado of Santiago, Chile. The participants were grouped into 5 thematic areas: education policy (23 participants), education management (24 participants), learning and learning resources (24 participants), learning assessment (18 participants) and curriculum (11 participants). Even though the project assumed that this activity was going to be evaluated before the closing of the loan to ascertain working changes and possible short-term impact of these graduates in improving the effectiveness of the MEC, the envisaged evaluation was never carried out. Accordingly, the ICR does not have any evidence to measure the effectiveness and impact of this activity, beyond the fact that 98 out of 100 professionals completed their program and that some of this graduates are currently holding high-level positions within the current administration in MEC.

33

Field visits carried out during some of the last supervision missions highlighted that, by and large, the hindering working context of these graduates has remained largely unchanged once they were reinserted to their original jobs after completion of the leadership program. Under this situation is very likely that the hindering constraints prevent these graduates from influencing changes in the education sector. The ICR is of the view that, unless proven differently, the impact of the leadership activity has not been very significant and rates it as moderately satisfactory.

Implementation of a national education communication plan to reach a diversity of stakeholders. The ICR rates this activity as unsatisfactory. Despite having implemented a series of uncoupled tasks, disseminated materials and carried out workshops and training for about 450 communication facilitators (dinamizadores comunicacionales) under the project, the ICR could not find any solid evidence that a social communication system was established to raise public awareness and garner greater support for the education reform. Opposite to these original targets set at appraisal, the ICR is aware of some press and TV releases during the implementation cycle criticizing some of the investments financed under the project, in particular the prototype secondary schools.

Rating component A. Weighing the achievement as well as the shortfalls fully described above, the ICR rates the overall implementation of this Component A as moderately satisfactory.

Component B – Secondary School Improvements. According to Table A2.1 below, this component estimated to cost US$8.40 million (US$7.57 million from the loan and US$0.83 million from GOP), ended up documenting expenditures amounting to US$7.68 million (US$6.75 million from the loan and US$0.93 million from GOP), 91.3 percent of the total estimated at appraisal. This financial shortfall is explained mainly by the transferring of financial resources from the category of goods under Component B to civil works under Component C and by the drastic appreciation of the Guaraní preventing the procurement of the science textbooks and other learning equipment. The assessment of the 4 key activities under Component B follows.

Quality-learning inputs for secondary education - textbooks. This activity is rated by the ICR as satisfactory. The project acquired and distributed about 547,000 textbooks of language (Spanish, Guarani and English), mathematics, art and physical education for secondary education students attending the three levels11 of about 1,757 public secondary schools as well as 45,000 teaching guides. Even though the original design only included textbooks for the first grade of secondary education a decision was made during the implementation, supported by the Bank, to extend the acquisition for the second and third grades. This accomplishment becomes an historic landmark in the Paraguayan education history, as it is the first time that secondary education students are provided with free textbooks. The expected life of the textbooks acquired under the project is of 5 years. Alongside this distribution, the project carried out a random sample follow-up of the use of these textbooks and teaching guides in the classrooms. The preliminary findings showed an appropriate utilization of these learning resources by both teachers and students.

The procurement process for textbooks for basic sciences, including 6 titles (two for physics, two for chemistry and two for natural sciences) was declared deserted because the bidders did not satisfied the

11 The first two levels of secondary education and some courses of the third level are common for all the students in secondary education. Starting in the third level, secondary education branches into an academic stream and a technical stream. The former prepares students for their in universities, while the latter prepares for entry into technical colleges and/or technical tertiary institutions. The project only financed textbooks of the first two levels and the common areas of the third level, expecting that under additional financing outside the project, the stream-specific ones for the third level could be also procured under technical and pedagogical specifications set for by the MEC. 34 expected technical specifications. Accordingly, the MEC developed the prototypes for these textbooks under local resources outside the project. Lack of financial resources has prevented the MoPE for printing these science textbooks. However the prototypes are available in the MEC’s web portal. On the downside, the Paraguayan Congress did not approve in the 2008, 2009 and 2010 budgets the MEC’s request for additional textbooks for secondary education.

Quality-learning inputs for secondary education – CRAs. This activity is rated by the ICR as satisfactory. This project activity included the establishment of CRAs in targeted secondary schools, including the prototype ones and in 20 selected teacher training institutes. The project provided the required rehabilitation infrastructure of 74 spaces (14 new spaces constructed and 60 rehabilitated), four more than the original design (the CRA in the third prototype school that was not constructed was canceled). The project also financed the acquisition of a set of learning resources for each CRA, including video-cassettes, TVs, computers, educational videos, books, camera, tape recorder, projector, maps. There were four sets of learning materials depending on the size of the secondary school and/or the training institute, being the largest set the one to be allocated to the two prototype schools. About 220 teachers (facilitators) were trained in the appropriate use of these CRAs. However, the ICR notes that for about 30 percent of the CRAs, MEC has not yet authorized the position of CRA facilitator. In addition, none of the CRAs include a security system to prevent against likely theft of the learning equipment.

Quality-learning inputs for secondary education – science laboratories. This activity is rated by the ICR as satisfactory. The first international procurement process for the acquisition of 25 science laboratories to be allocated to same number of targeted secondary schools was declared deserted. A second procurement process for the acquisition of 24 science laboratories (the one of the third prototype school was canceled) was started mid-2007 and the laboratories were installed in 22-targeted secondary schools and the two prototype schools towards the end of 2009. About 1,000 operational manuals for the use of these laboratories were prepared and distributed to the benefited secondary schools. There was an effort during the latter part of the implementation cycle to articulate the use of textbooks, CRAs and science laboratories in the benefited schools.

Management community-based strategies. The ICR rates this activity under Component B as moderately satisfactory for the following reason. This activity was initially implemented in a disjoint fashion having school stakeholders developed their PEIs and PSPs and ensuring that their ACEs were fully active through separate and independent tracks under one technical coordinating group within the MEC/PCU. Towards 2007, and resulting from persistent Bank recommendations during diverse supervision missions, an effort was made by the MEC/PCU to correct course and integrate these three originally uncoupled tasks into a more systemic and articulated approach through the establishment in each participating secondary school of an Institutional Management Team (Equipo de gestión institucional – EGIs) conformed by the school principal, two representatives of the school teachers, two of the parents and two of the students. Under the EGI scheme, the school’s PEI became the instrument that ordered and guided the educational discussion among school stakeholders. EGIs became active in the 100 targeted secondary schools with an approved PEI. The project financed the development and implementation of 100 PEIs and 100 PSPs and supported 600 ACEs.

The project files reflect that the diverse Bank supervision missions attested to the fact that the effectiveness and relevance of this activity was heavily dependent on two factors - the leadership of the school principals chairing the EGI and the percentage of teachers with a significant time of their daily teaching load allocated to the same school. In effect, participating secondary schools with natural leaders chairing their EGI got more mileage out of their PEIs with respect to their contribution in attaining some of the PDOs, in particular the one concerning increase enrollment, retention and completion rates in the secondary level. Likewise, this synergy was also present in schools having a smaller percentage of the so-called “taxi-teachers.”

35

This activity was thought in the original project design as an appropriate instrument to enhance community participation in the co-management of the school, improve the attitudinal climate among key school stakeholders as well as the communication spaces, and contribute to achievement of the PDOs. The underpinning assumption was that effective PEIs, relevant PSPs and active ACEs would become necessary, although not sufficient conditions, to have participating secondary schools improve their quality and efficiency-related performance. Although not refuting this key, and certainly proven elsewhere assumption, the ICR found no systematic and documented evidence, beyond some anecdotic one, to prove that this was the case in Paraguay under this project. As reported by the MEC/PCU, schools with PEI increased their secondary education enrollment from 28,747 to 28,952 students in the period 2005-2007 implying an average annual growth of 0.4 percent while dropout decreased about 3.6 percent in these schools in the same period. However no information was made available as to changes that might have also occurred with comparable secondary schools without a PEI, thus no inference can be made by the ICR about the impact of the PEIs on enrollment and retention. Another analysis made from the findings of the SNEPE baseline of 2006 show that were no significant differences in learning outcomes between schools with and without a PEI.

The assumed contribution of the PSPs and ACEs to the achievement of some of the project’s PDOs is even less clear to the ICR as no solid documented evidence to that effect was found.

Rating component B. Weighing the strengths and weaknesses described above, the ICR rates the overall implementation of this Component B as satisfactory.

Component C – Mechanisms to Provide More Equitable Access to Secondary Education. According to table A2.1 below, this component originally estimated to cost US$7.80 million (US$ 6.82 million from the loan and US$ 0.98 million from GOP), ended up documenting expenditures amounting to US$8.83 million (US$7.80 million from the loan and US$1.03 million from GOP), 113.21 percent of the total estimated at appraisal. The overspending in civil works mainly explains this difference. The following is the assessment of the three key activities included under this component.

Scholarships. The ICR rates the implementation of this key project activity as highly satisfactory for the following reasons. It achieved its stated goal of providing 4,500 scholarships to poor students in secondary education. It was a targeted and transparent activity ensuring that only poor students in the lower income-decile distribution received the benefit strictly following selection criteria and procedures agreed with the Bank. It is an activity that was evaluated and the findings recorded in the sector work mentioned elsewhere in the ICR. The findings attest to the fact that it was a cost-effective intervention. The project financed the technical capacity build-up for MEC to manage the provision and monitoring of the scholarship scheme. Finally, GOP has continued to provide scholarships to poor secondary education students entering first grade. In addition to the accumulated 4,689 provided by the project in the 2004-06 period, the MEC has provided from its own financial resources 4,518 in 2007, 3,000 in 2008, 10,060 in 2009. This activity has been financially, institutional and technically sustained by the Borrower even before the closing of the loan. The average amount of the scholarship has remained almost the same of about US$261.40 equivalent per year per student.

Alternative Secondary Education (Educación Media Abierta – EMA). The ICR rates the implementation of this activity as unsatisfactory for the following reason. At the time of Board approval, there was no conceptual design of this alternative, and as a consequence, neither and operational description. For almost three years after Board approval, the MEC and the PCU were conceptually struggling to implement an alternative option under the advice of a non-qualified consultant lacking sufficient expertise on this topic. By the end of 2006, the MEC started implementing EMA under an incorrect pedagogical

36 paradigm,12 with no teaching modules and printed material yet developed, nor appropriate staff trained. About 600 out-of-school youngsters between the ages of 15 and 19 years old demanding secondary education were initially enrolled in this option in 20 secondary schools of 4 Departments (Caaguazú, Central, Cordillera and San Pedro) and Asunción during the second semester of 2006. At the time of the writing of the ICR, active enrollment had decreased to 172 out of the initial class of 600 students and the number of original schools to 17. Of this leftover enrollment, 76 students graduated in December 2009 – January 2010 period (more than 3.5 years after they started and longer than the school-based modality requiring only 3 years). The remaining 96 students are likely to graduate during 2010. The improvisation of a technically and operationally poorly-defined option was the main culprit of this significant dropout rate. During the June 2007 supervision mission the Bank team strongly recommended against the extension of this ill-defined model to a sixth Department (Chaco), recommendation that was followed up by the authorities.

A qualified international consultant was retained by MEC to help the local team significantly modify the entire approach to better address the needs of the out-of-school population intended to be served. Some modules were prepared and tested along their evaluation guidelines; staff was selected and trained. The 15 curricular units comprising the 3 modules of this alternative modality (5 units per module) as well as their corresponding evaluation booklets have been developed and printed. There are 60 trainers participating in this modality. MEC has decided to temporarily suspend this option after the remaining students graduate from this modality. The ICR is of the view that this modality has resulted in a cost-ineffective alternative.

Prototype schools. The ICR rates the implementation of this activity as moderately unsatisfactory for the following reasons. The ICR is of the opinion that this activity should not have been included in the project design. As shown in the sector work, other alternatives included in the project design – like the scholarships – resulted to be more cost-effective to increase and retain enrollment to secondary education, especially benefiting poor students. Furthermore, during the several supervision missions, the Bank team attested that smaller secondary schools, holding a maximum of about 500 students and with about one sixth the square meters of the prototype schools were improving access and retention, and their learning scores registered in the SNEPE baseline were above the average. A cost-benefit analysis done during the preparation stage could have provided some clues to justify the inclusion of other alternatives and reject the prototype schools path.

Prior to the launching of the procurement process to select a qualified construction firm, the pedagogical areas of MEC, in particular the General Directorate of Secondary Education, the General Directorate of Educational Development and the Directorate of Curriculum, objected to the prototype’s architectural design included and appraised in the PAD claiming that it was not attuned to the requirements of the education reform in progress. Accordingly, the design was modified from 3,700 to 6,600 square meters (78.4 percent increase in square footage) to serve about 750 students per shift in two shifts (for an accumulated enrollment of 1,500 students per school) and the allocation to the civil works category of

12 The pedagogical model of the alternative secondary option reviewed by the December 2006 Bank supervision mission was in essence an extrapolation of the school-based secondary education curriculum. This option was considered by the mission as inappropriate to address the needs of the out-of-school population and reflected this opinion in the corresponding Aide-Memoire (paragraphs 50(a) to (d)). There was no modular structure, expecting these students to attend classes of long duration, going against their time availability. There was no prior recognition of learning acquired thus everybody had to start from the very beginning. The content was too academically and not based on competencies to be acquired by the students at the end of each learning cycle. 37 expenditure of the loan increased from US$3.8 million to US$4.81 million, with Bank management approval (26.6 percent increase).13

The December 2006 supervision mission alerted about the risk that, even in the event of completing these schools on time, and targeting them to benefit mostly poor students living in their catchment areas, it was going to be very unlikely for the GOP to replicate these prototypes elsewhere in the country, and difficult to appropriately maintain them at the closure of the loan (paragraph 47 of the corresponding Aide- Memoire). On average, it was estimated about 2 percent of the total cost for annual maintenance, once the infrastructure started wearing out, implying for the prototype schools about US$50,000 per school. As a consequence of this significant change, the Bank team objected to the construction of 3 prototype schools, and the one in Fernando de la Mora, presenting more complicated construction challenges, was eliminated from the project along its learning goods requirements.

The procurement process was thus delayed. The construction of the school in Luque started on April 16, 2007 and the one in Caaguazú in April 23, 2007, almost 4 years after Board approval. The issue of construction oversight by a third qualified party (fiscalizadora) for these two large projects, satisfactory to the Bank, was not resolved until the end of 2007. As a consequence, the Bank did not authorize eligible disbursements against this category of expenditure from April to November 2007, thus delaying further the implementation progress. In February 2008, part of the ceiling of the Luque school amounting to 5 percent of the total construction square footage collapsed and the construction was halted for several months until a structural review presented its findings and responsibilities to fix the problem were determined and agreed upon among the key stakeholders (MEC/PCU, the designing consulting firm, the construction firm and the oversight firm).

The May 2008 Bank supervision mission detected serious safety issues in both prototype schools that were not addressed in the original design included in the PAD nor in the modified design that was procured. No protecting railing was considered in the staircases connecting the four floors of each school. There was a significant likelihood of a student or teacher fatality as a consequence of somebody accidentally falling in the building’s staircase shaft. Addressing these safety issues demanded a reallocation of the now limited loan proceeds. In addition, the drastic fluctuation of the local currency against the dollar, the construction price increases, especially of cement and steel and the under- estimation of unit projected costs, had adverse effects on timely decisions on the utilization of available loan resources.

The prototype school in Caaguazú was completed at the end of 2008 and is functioning since February 2009 at minimal capacity (16 percent) with 245 students in two shifts (133 students in first grade and 112 in second grade) with a total of 19 teachers. The CRA, the computers and the science laboratories were made functional at the start of 2010 school year. The civil works of the prototype school in Luque were completed by December 2009. This school is operating at minimal capacity with 146 students in first grade (9.7 percent) with 15 teachers. MEC has expressed its intention to complement the utilization of these under-utilized premises with teacher training, education research and community-related activities.

The new education authorities have let the Bank know about their annoyance concerning the construction of these two prototype schools, seen by them as “white-elephant projects” not meriting the financial investments allocated to them. They have voiced their criticism publicly in the press. As mentioned before, the reputation risk for the Bank is high. The challenge now is to have these two prototype schools

13 This increase also required augmenting the counterpart funding from the original US$0.38 million to US$0.55 million (44.7 percent increase). Adding the loan proceeds with the counterpart funding, the total allocation for the civil works category of expenditure jumped from US$4.18 million to US$5.36 million (28.2 percent). 38 functioning at maximum capacity as soon as possible and to target their utilization to better serve poor secondary students and the community in the area of influence.

The existing potential to fully exploit these two schools in the near-future to best address the PDOs makes the ICR rate this activity as moderately unsatisfactory instead of unsatisfactory.

Rating component C. Weighing the highly satisfactory rating for the scholarship scheme, with the unsatisfactory ratings for the alternative secondary option and the moderately unsatisfactory rating for the prototype schools, the ICR rates the overall implementation of this Component C as moderately unsatisfactory.

Component D – PCU. According to table A2.1 below, this component originally estimated to cost US$1.2 million (US$ 1.09 million from the loan and US$ 0.11 million from GOP), ended up documenting expenditures amounting to US$2.28 million (US$1.44 million from the loan and US$0.84 million from GOP), 190 percent of the total estimated at appraisal. The increase was partially caused by the operating costs that went from 310,000 estimated at appraisal to 530,000 at the time of writing the ICR (71 percent increase) to cover PCU staff, office rent, office materials and supplies, transportation and communication expenses for 30 months longer than expected at appraisal.14

The ICR highlights the effective working articulation existing between the PCU in charge of the project- related fiduciary aspects (procurement and financial management) and the selection and hiring of required technical assistance financed by the loan and the MEC, through the Vice-minister, in charge of the implementation of all project-related substantive aspects. The only area that escaped the effective and timely oversight of the Vice-minister and the PCU was the civil works sub-component, whose first coordinator (selected following Bank procedures) ended up acting as a free agent directly accountable to the Minister of Education. This was also another, among many reasons, that resulted in a moderate unsatisfactory implementation of the civil works financed under the project. This coordinator resigned before the change of government.

This component included two additional activities – monitoring and evaluation and project management. The implementation of the monitoring and evaluation is a mixed story. On the upside, there was an evaluation of the scholarship scheme whose findings are explained elsewhere in the ICR and detailed in the sector work referred to in Annex 9 of the ICR. The PCU timely provided updated physical and financial implementation progress information and prepared extensive and detailed progress reports prior to the start of every supervision mission.

On the downside, the project was not able to contract impact evaluation studies of other key activities of the project, like inter-alia, the leadership program, the in-service teacher training and the community- based strategies. Despite the findings and recommendations made by a qualified consultant on the design of a more comprehensive, crosscutting, relevant, reliable and timely monitoring system to follow up on inputs, processes, outputs and outcomes beyond the ones described in the paragraph above and inform with respect to the achievements of stated final and intermediate goals, this system was not fully developed, less implemented.

Another management shortcoming was the lack of proactivity of the PCU Coordinator and the financial specialist, letting in many occasions, especially mid-2007 onwards, to very shortsighted reactive decisions

14 According to the PCU records provided to the ICR, the number of full-time and part-time consultants paid under this component where: (i) 30 in 2004; (ii) 49 in 2005; (iii) 109 in 2006; (iv) 117 in 2007; (v) 64 in the first semester of 2008; (vi) 45 in the second semester of 2008; (vii) 24 in the first semester of 2009; and (vi) about 10 from July 2009 onwards fully paid by counterpart resources. 39 hindering a timely implementation of key project activities, especially concerning procurement. Ironically, this lack of strategic vision by the PCU coordinator will generate a serious disconnect between the need to timely acquire learning goods and services and ensuring the availability of authorized loan and counterpart resources to proceed. For example, the 2009 project budget requested by the PCU coordinator of US$1.5 million was about half the required financial needs. Accordingly, about US$1.2 million equivalent of loan proceeds will not be disbursed at the closing of the loan.

Rating component D. Given the strengths reflected by the achievement of several of the project activities fully described elsewhere in the ICR for which the PCU was a significant contributor and the weaknesses described above, especially lack of strategic managerial approach from mid-2007 to the closing of the loan, the ICR rates the performance of the PCU as moderately satisfactory.

Table A2.1: Documented expenditures by component and by source at appraisal and closing of the loan (in US$ millions) Component At the time of appraisal At the time of loan closing Percentages IBRD GOP Total IBRD GOP Total (1) (2) (3) (4) (5) (6) (4)/(1) (5)/(2) (6)/(3) Component A. 8.52 0.34 8.86 6.81 1.27 8.08 74.30 273.53 91.20 Component B. 7.57 0.83 8.40 6.75 0.93 7.68 89.04 112.05 91.31 Component C. 6.82 0.98 7.80 7.80 1.03 8.83 114.37 105.10 113.21 Component D 1.09 0.11 1.20 1.44 0.84 2.28 132.11 663.64 190.00 Total project 24.00 2.26 26.26 22.80 4.07 26.87 95.00 180.09 102.32 costs Front-end fee 0.00 0.24 0.24 0.00 0.24 0.24 NA 100.00 100.00 Total 24.00 2.50 26.50 22.80 4.31 27.11 95.00 172.40 102.30

According to table A2.1, the total project cost estimated at appraisal was US$26.5 million equivalent, corresponding US$24 million equivalent from the Bank (91 percent of the total) and US$2.50 million equivalent from the GOP (9 percent of the total). At the time of the closing of the loan, the total documented expenditure amounted to US$27.11 million equivalent (102.3 percent of the appraised total), corresponding US$22.80 million equivalent from the Bank (95 percent of the appraised total) and US$4.31 million equivalent from the GOP (172.4 percent of the appraised total).

The under spending in Component A against the appraised estimate (91.2 percent) is explained by: (a) the delay in having the MEC’s reengineering completed and the shortfall in the amount of hardware, in particular the required server and computers, that the project could not acquire for having the MEC cancel the procurement process at the end of 2008; (b) the reallocation of loan proceeds from this component to the construction of the two prototype schools; and (c) the limited budget authorized for 2009 which was not attuned to the project requirement for this particular year. The slight under spending in Component B against the appraised estimate (91.3 percent) is explained by the acquisition of one less science laboratory and CRA than the total included in the original design and the no acquisition of science books. Despite that one prototype school was dropped, the increase in the allocation of resources to the civil works resulted in a real spending of Component C 13.21 percent above the appraised estimate. The significant overspending in the management of this project against the appraised estimate (190 percent) is an indication that the management of this project was far from efficient.

40

Table A2.2: Documented expenditures by category and by source at appraisal and closing of the Loan (in US$ millions) Category of At the time of appraisal At the time of loan closing Percentages Expenditure

IBRD GOP Total IBRD GOP Total (1) (2) (3) (4) (5) (6) (4)/(1) (5)/(2) (6)/(3) Infrastructure 3.80 0.38 4.18 5.89 0.66 6.55 155.00 173.68 156.70 Goods 10.7 1.07 11.77 7.50 1.11 8.61 70.09 103.74 73.15 Consultants 2.95 0.29 3.24 4.14 1.42 5.56 140.34 389.66 171.60 Training 2.80 0.28 3.08 2.06 0.37 2.43 73.57 132.14 78.90 PEIs 0.23 0.03 0.26 0.37 0.06 0.43 160.87 166.67 165.38 PSPs 0.23 0.03 0.26 0.36 0.05 0.41 156.52 166.67 157.69 Scholarships 2.16 0.16 2.32 2.09 0.26 2.35 96.76 162.50 101.29 Operating costs 0.29 0.02 0.31 0.39 0.14 0.53 134.48 600.00 170.97 Unallocated 0.84 0.00 0.84 0.00 0.00 0.00 NA NA NA Front-end fee 0.00 0.24 0.24 0.00 0.24 0.24 NA 100.00 100.00 Special Account 0.00 0.00 0.00 0.00 0.00 Total 24.00 2.50 26.50 22.80 4.31 27.11 95.00 172.40 102.30

The table A2.2 above indicates relevant overspending against the appraised estimates in civil works (156.7 percent), consultants (171.6 percent) and operating costs (170.97 percent) and under spending in goods (73.15 percent) and training (78.9 percent). Spending in scholarships mirrors the appraised estimate. Due to the little amount allocated to the socio community activities their overspending does not really alter the big picture neither the explanation that follows.

The ICR has fully explained elsewhere the reasons of the overspending in civil works, consultants and operating costs as well as the under spending in goods. The significant delays in having started the implementation of the pre-service and in-service teacher training partially explains the under spending in the training category of expenditure.

41

Annex 3. Economic and Financial Analysis (including assumptions in the analysis)

The Net Present Value (NPV) was US$3.8 million and the estimated Economic Rate of Return (ERR) was 15 percent, considering a 10-year time horizon since the beginning of the project. The estimated NPV and ERR for a 20-year time horizon since the beginning of the project were US$ 25.4 million and 33 percent, respectively. The Cost-Benefit Analysis was re-estimated to calculate the NPV and the ERR using end of the program actual values.

The Cost-Benefit analysis in the PAD considered the following costs and benefits:

1. Benefits

1.1 Benefits due to higher enrollment: The higher number of secondary graduates expected from the scholarships, the new schools built, and the alternative secondary program would increase the number of graduates and, therefore, increase their future earnings streams. 1.2 Benefits due to quality improvements: The program was expected to increase quality and efficiency at the secondary level (reducing both repetition and dropout rates). This increase in efficiency would generate savings in recurrent costs at the secondary level. 1.3 Potential savings due to re-allocation of students: The strengthening of the Ministry of Education’s Planning Unit and the strengthening of the leadership and managerial skills of the educational system leaders were expected to make a more efficient allocation of students across schools in the education system. This re-allocation would increase the number of students per section, which in turn would reduce the Teachers’ Salaries Budget (no replacement of the teachers who retire).

2. Costs

2.1 Capital Expenditure: Included in the PAD were investments and expenses financed with the IBRD loan (US$ 23.27 million). 2.2 Recurrent Expenditure: In the PAD this included the government’s financial contribution to the program (US$ 10.13 million). 2.3 Forgone Income: The income that the new secondary education graduates would have received if they would have worked instead of attending school. 2.4 Incremental Expenditure due to higher enrollment: The decrease in repetition and dropout rates in secondary education would generate savings (as mentioned in benefit 1.2 above) but would also increase current expenditure, for example, the Teacher’s Salaries Budget.

These benefits and costs for the 10 year scenario are summarized in Table 1 below.

42

Table 1: Estimated Costs and Benefits include in PAD – Time Scenario: 10 years

Net Benefit Expected Cost (In Thousands of US$) Benefits (In Thousands of US$) (In New Recurrent Capital Forgone Incremental Benefit Benefit due Potential Thousands Graduates Years Expenditure Expenditure Income Expenditure due to to Higher Savings due to of US$) (Number Due to quality enrollment re-allocation of higher of students Graduates) enrollment 1 0 214 3946 110 374 -3896 2 0 1005 8061 221 747 1206 -7334 3 1780 2011 11264 4936 331 1121 7150 2411 -7860 4 1600 2471 6486 4437 441 1494 6427 3617 -2297 5 3199 2471 6486 8871 551 1868 12851 4823 1163 6 9738 2471 6486 27005 662 2241 39119 6029 10765 7 2400 6655 1491 2241 9640 6029 9764 8 2933 8133 1491 2241 11781 6029 10427 9 3466 9610 1491 2241 13921 6029 11090 10 4000 11092 1491 2241 16068 6029 11755 Information for the whole program, unless specified PV (In US$ Million) Benefits: Increased efficiency due to reallocation of students 24.7 Savings derived from quality-enhancing measures 10.2 Increased access 68.6 Costs: Investment Costs 33.9 Recurrent Costs 7.9 Forgone Income 47.4 Incremental Expenditure due to higher enrollment 4.6 NPV: 9.7 IRR: 19%

43 The Cost Benefit Analysis was re-estimated to calculate the NPV and the ERR using end of the program actual values. The model remained the same as in the appraisal; however the following changes were introduced due to the end of program actual values:15

1. The number of graduates was adjusted to reflect the lower number of graduates students due to the construction of two prototype schools as opposed to the three included in the original CB analysis and the lower coverage of the alternative secondary program. As mentioned he number of scholarships met expectations (9,000 students). This change decreases the benefits due to higher enrollment in the model, and therefore decreases the NPV. 2. The reduction in Teachers’ Salaries Budget was not as successful as expected. Based on budgetary information it was estimated that the reduction in Teachers’ Salaries Budget was approximately 20 percent of one originally included in the PAD’s estimates. This change decreases the NPV. 3. The capital expenditure was reduced from US$ 23.7 million originally included in the PAD to US$ 22.8 million which was the actual financial information at the end of the program. This change will increase the NPV. 4. The recurrent expenditure (government’s financial contribution) was reduced from US$ 10.13 million originally included in the PAD to US$ 4.0 million which was the actual financial information at the end of the program. This change will increase the NPV. 5. The opportunity cost (foregone income) was lower than the one originally included in the PAD because of the lower number of new graduates. This change will increase the NPV.

Table 2 presents the detail of the calculation using a 10 year time period. The estimates indicate a negative NPV of US$ 1.4 million and a revised ERR of 8 percent. Under this scenario (10 years) the project could not be considered a good social investment. However it is important to highlight two aspects. First, the Cost Benefit Analysis model used at project preparation did not quantify the benefits associated with a more educated labor force. These benefits may have two types of effects: (i) productivity and growth effects above the direct income increases that beneficiaries are receiving, and (ii) lower risks of unemployment that more educated people face. Additionally, intangible benefits were not included (e.g. aesthetic appreciation, better health, and better voting behavior and social behavior that research has linked to increases in education). If these benefits would have been quantified and included in the analysis at appraisal the expected direction of the effects would be an increased NPV and an increased ERR.16 Second, a longer time horizon would change this result. For example, using a 15 year time period since the beginning of implementation would make the project to break even using end of program values and considering a 20 year time period the estimations indicate a revised NPV of US$600,000 and a revised ERR of 11 percent.17

15 The average promotion rate for the last three years of secondary education was 83.8 percent in the year 2000. An increase of 10 points in the average score of a standardized test was estimated to increase the promotion rate by 5 percentage points, generating savings in recurrent costs. According to the Government information, the average score of the standardized test increased in 10 points from 2001 to 2008 (from 44 points to 54 points) and the promotion rate for secondary education also increased as expected (from 83.8 percent to 88.8 percent, 5 percentage points). Therefore, the benefits due to quality improvements did not change in the revised analysis. 16 The analysis conducted in this report did not make an attempt to value these benefits. 17 The selection of the time horizon is an option of the analysis but it is expected that the benefits of these intervention are closer to 15 years since the beginning of the implementation of the project rather than the 10 year considered in the PAD. 44

Table 2: Revised Costs and Benefits using end of program information – Time Scenario: 10 Years

Cost (In Thousands of US$) Benefits (In Thousands of US$) Net Benefit Recurrent Capital Forgone Incremental Benefit Benefit due Potential (In Thousands Years Expenditure Expenditure Income Expenditure due to to Higher Savings due Due to quality enrollment to re- of US$) higher allocation of enrollment students 1 449 2560 110 374 -2745.1 2 746 4250 221 747 241.2 -4228.4 3 1756 10010 2468 331 1121 3575 482.2 -9386.9 4 904 5150 2219 441 1494 3214 -3282.1 5 146 830 4436 551 1868 6426 964.6 3296.0 6 0 13503 662 2241 19560 1205.8 8841.8 7 3328 1491 2241 4820 1205.8 3448.3 8 4067 1491 2241 5891 1205.8 3779.8 9 4805 1491 2241 6961 1205.8 4111.3 10 5546 1491 2241 8034 1205.8 4443.8 Benefits: PV (In US$ Million) Increased efficiency due to reallocation of students 4.9 Savings derived from quality-enhancing measures 10.2 Increased access 37.7 Costs: Investment Costs 19.1 Recurrent Costs 3.4 Forgone Income 26.0 Incremental Expenditure due to higher enrollment 4.6 NPV: -1.4 IRR: 8%

45

Annex 4. Bank Lending and Implementation Support/Supervision Processes

(a) Task Team members

Names Title Unit Responsibility/Specialty At the preparation stage

Cynthia Hobbs Senior Social Sector Specialist LCSHE Task team leader Rui Aguiar Administration Specialist Consultant Decentralization Juan Agulla Administration Specialist Consultant Administration Mario Alvarez Communication Specialist Consultant Communication Antonio Blasco Financial Management Specialist LCC7C Financial Management Vicente Brunetti Legal Counsel Consultant Lawyer Carli Castellano Focus Group Specialist Consultant Beneficiary survey Susana Cirigliano Financial Management Specialist LCC7C Financial management Cristian Cox Education Reform Specialist Consultant Policy dialogue Madalena Dos Santos Sector Leader LCC5C Peer reviewer Celia Farias Education Specialist Consultant Alternative education Ariel Fiszbein Sector Leader LCC7C Policy dialogue Sebastián Galiani Economist Consultant Demand-driven activities Maria Lucy Giraldo Senior Procurement Specialist LCOPR Procurement Antonio Gomes Pereira Education Specialist Consultant Teacher training Vilma Guimaraes Education Specialist Consultant Alternative education Bruce Jones Senior Economist AFTH1 Peer reviewer Federico Mejer Planning and Administration Specialist Consultant Education management Luz Meza-Bartrina Legal Counsel LEGLA Lawyer Jose Molinas Economist Consultant Economic analysis Suhas D. Parandekar Education Economist LCSHE Economic analysis Luis Secco Arquitect Consultant Civil works Andrea Staeheli Focus Group Specialist Consultant Beneficiary survey Raúl Talán Technical Education Specialist Consultant Technical education Gordon Temple Costab Specialist Consultant Project design and costing Dense Vaillant Teacher Training Specialist Consultant Teacher training Morag Van Praag Disbursement Officer LOAG7 Disbursements Marina Velasco Parra Focus Group Specialist Consultant Beneficiary survey Aracelly Woodall Team Assistant LCSHE Mission logistics Universidad Católica de Nuestra Señora de Asunción Consulting firm Curriculum reform ALECOP/Mondragón Consulting firm Teacher training ProAgro Consulting firm School mapping

At the implementation stage Mario Cristian Aedo Education Specialist LCSHE Task team leader Cynthia Hobbs Senior Social Sector Specialist LCSHE Task team leader Alexandria Valerio Señor Education Specialist LCSHE Task team leader Daniel Arguindegui Architect Consultant Civil works Alexander Arrobio Señor Financial Management Specialist LCC7C Financial management Francisco Ayala Social Safety Net Specialist AFTH1 Community participation Natalia Bavio Financial Management Specialist LCC7C Financial management Antonio Blasco Financial Management Specialist LCC7C Financial management

46

Pedro Cerdán Infantes Junior Prodessional Associate LCSHE Sector study Rene Cisternas Civil Engineer Consultant Structural analysis Zulema de Fuentes Community Participation Specialist Consultant Community participation Maria Lucy Giraldo Senior Procurement Specialist LCOPR Procurement Antonio Gomes Pereira Education Specialist Consultant Teacher training Efraim Jiménez Lead Procurement Specialist LCC7C Procurement Jesko Hentschel Sector Leader LCC7C Supervision guidance Andrés Mac Gaul Señor Procurement Specialist LCC7C Procurement Federico Mejer Planning and Administration Specialist Consultant Education management Luz Maria Meyer Financial Management Specialist LCC7C Financial Management Natalia Moncada Program Assistant LCSHE Support and logistics Thomas Otter Economist Consultant Cost benefit analysis Juan Prawda Education Specialist Consultant Project assessment Monica Tambucho Financial Management Specialist Consultant Financial Management Graciela Sánchez Social Development Specialist LCC7C Sector Study Joseph Shapiro Junior Professional Associate LCSHE Impact evaluation Luis Secco Arquitect Consultant Civil works Eduardo Velez Bustillo Sector Manager LCSHE Supervision guidance

(b) Staff Time and Cost

Staff Time and Cost (Bank Budget Only) Stage of Project Cycle USD Thousands (including No. of staff weeks travel and consultant costs) Lending FY01 14 45.51 FY02 13 154.22 FY03 22 149.15 FY04 1 3.96 FY05 55.10 FY06 0.00 FY07 0.00 FY08 0.00 Total: 50 407.94 Supervision/ICR FY01 0.00 FY02 0.00 FY03 0.00 FY04 13 86.42 FY05 15 99.23 FY06 17 117.65 FY07 16 90.98 FY08 17 88.99 FY09 18 0.00 Total: 96 483.27

47

Annex 5. Beneficiary Survey Results

Not applicable

48

Annex 6. Stakeholder Workshop Report and Results

Not applicable

49

Annex 7. Summary of Borrower’s ICR and/or Comments on Draft ICR

MEC’s Contribution to the ICR - Summary

1. What the project proposed to accomplish

The project design aimed to achieve improvements in the education system efficiency, in the quality of education and in increased access, providing financial and technical resources for a set of strategic initiatives. The objectives were defined in line with the following: (a) improve education system management; (b) improve secondary education quality; and (c) expand access to secondary education.

2. What was achieved and what was not achieved - main causes

In terms of coverage, the enrolment rate increased significantly from 44 percent to 53 percent in the 2001-2006 time period. However, this increase is still insufficient. The persistence of internal efficiency problems in the education system, the worsening of socio-economic conditions of the students, and the insufficient supply of education services are some of the main barriers to expand access to secondary education.

With respect to learning quality, SNEPE results showed limited improvements in education achievement for secondary students. Even though the project design included actions to improve learning quality, the delayed implementation and the lack of coordination of these actions does not allow estimating the impact of the interventions yet.

Some factors affected project implementation, including among other the following: delayed in implementation, problems with the original design, exchange rate variations, expansion of textbooks provision to the three grades in secondary education (10, 11, and 12), financial management, use of the loan’s availability balance, under-estimation of the costs of the civil works, change of government, etc.

3. Lessons learned

Although the goals for components and subcomponents have been achieved and, in some cases, the results have been greater than expected at appraisal, some lessons have been learned from this experience that can be useful for future projects:

(a) The construction of a clear, articulated and concrete logical framework for the project, with objectives and results and impact indicators. The logical framework should be constantly supported by the monitoring and evaluation system to sustain continuous decision making and the introduction of adjustments during implementation; (b) The importance of having technically qualified and stable staff in the MEC line units involved with the implementation of the project. The staff should be committed to achieve the objectives for each activity in the project and should be accountable for its results. This way project sustainability would be technically guaranteed through the human resources capacity available at MEC. Staff should have access to updated information about the project budget to improve its efficiency, rationalization in the usage and optimization of financial resources; and (c) The intervention in the third cycle of the basic education (grades 7 – 9) is crucial to increase coverage in secondary education, one of the project’s goals.

50

4. Relationship MEC/PCU (Program Coordination Unit) with the Bank

(a) The World Bank’s Team and technical staff working for the project were open to continuously evaluate the implementation progress; (b) Even though the relationship MEC/PCU with the Bank was in general satisfactory, the implementation of project activities and their effectiveness to achieve the envisioned goals were affected by local rules both at the MEC, the central level and the Minister of Finance; and (c) Finally, the lack of staff stability in the key areas of the PCU created delays in the implementation of the planned project activities.

51

52

53

54

55

56

57

58

59

60

61

62

Annex 8. Comments of Cofinanciers and Other Partners/Stakeholders

Not applicable

63

Annex 9. List of Supporting Documents

Bank preparation documents Paraguay Country Assistance Strategy (CAS) for the period 1997-2003, discussed by the Board on March 27, 1997. Paraguay Country Partnership Strategy for the Period 2004-2007, discussed with the Board on December 2006 (Report 27341-PA). Paraguay’s Poverty Reduction Strategy (Estrategia Nacional de Reducción de la Pobreza y la Desigualdad, November 2002) Statement of mission objectives, Aide-Memoires and Back-to-Office Report of the pre- identification mission carried out March 5-9, 2001. Statement of mission objectives, Aide-Memoires and Back-to-Office Report of the identification mission carried out June 11-20, 2001. Statement of mission objectives, Aide-Memoires and Back-to-Office Report of the preparation missions carried out on October 16-26, 2001, December 10-14, 2001, March 11-15, 2002, Statement of mission objectives, Aide-Memoires and Back-to-Office Report of the pre-appraisal missions carried out July 17-26, 2002, November 6-14, 2002 and February 17-26, 2003. Statement of mission objectives, Aide-Memoires and Back-to-Office Report of the appraisal mission carried out April 2, 2003. Project Concept Document (PCN) and Minutes of the PCN Review Meeting, July 16, 2002. Project Appraisal Document (PAD), Report No. 25630. The World Bank, June 25, 2003. Minutes of the PAD Review Meeting, April 7, 2003. Agreed Minutes of Negotiations. June 12-13, 2003.

Bank project implementation documents Loan Agreement 7190-PA. The World Bank. Conformed copy September 10, 2003 PHRD Grant Agreement TF026846 for the preparation of the Paraguay Secondary Education (Bank-executed). October 3, 2001, countersigned by the GOP on October 19, 2001. Statement of Mission Objectives, Aide-Memoires, Back-to-Office Reports, Management Letters, and Implementation Status and Results (ISRs) of all the Supervision Missions recorded in the Data Sheet of this ICR and in the project’s electronic files. The World Bank. From June 25, 2003 to December 4, 2009. Quarterly Financial Management Reports for the period 2004 to 2009. Audit Reports for years 2004 up to 2008.

Bank and Borrower other project implementation-related documents

CARE. Reforma Educativa Compromiso de Todos. 1992.

64

CARE. El desafió educativo: Una propuesta para el dialogo sobre las oportunidades educativas del Paraguay. 1996.

CARE. Paraguay 2020: Enfrentemos juntos el desafio educativo. 1996.

GOP’s National Development Plan 2003-2008

Harvard Institute of International Development and Centro Paraguayo de Estudios Sociológicos. Situación del sistema educativo en el Paraguay: Sugerencias de política y estrategia de acción. 1993.

Plan Ñanduti (Plan Nacional de Educación para Todos 2003-2015).

The World Bank. Educación Media en el Paraguay. Logros, opciones y desafios. Report 25630. June 25, 2007.

Universidad Católica de Asunción. “Estudio sobre Educación Media y Desarrollo en el Paraguay”. MEC/ISE/MECES, 1999.

65

IBRD 33464R 62W 60W 58W 56W 54W

18S 18S

BOLIVIA

PARAGUAY

Capitán PabloPablo Lagerenza To Lagerenza Santa Fé 20S Puerto 20S Cerro León ALTO Bahía Negra (1,000 m)

General EugenioEugenio PARAGUAY To A. Garay A. Garay Boyuibe Fortín Madrejon Fuerte Olimpo

Fortín Carlos BRAZILB R A Z I L A. Lopez Puerto Tres Palmas To Fortín BOQUERON Campo Infante Rivarola Kilómetro Grande 160 o 22S MariscalMariscal c EstigarribiaEstigarribia a Puerto To Doctor h La Victoria Dourados Pedro P.P. PeñaPeña C Filadelfia n Puerto Pinasco AMAMBAY P i Pedro Juan lc a o m r Caballero a y o quidabá G A n Yby Yaú Verd Fortín e Leonida Escobar CONCEPCION Pozo Fortín Ávalos ConcectiónConcepción Capitán Sánchez Colorado Bado Fortín Monte Lindo General PRESIDENTE Díaz Fortín Teniente San Lima Rojas Silva P To a Pedro r 24S Las Lomitas HAYESH A Y E S a Salto del 24S g u SANS A N Guairá a CANENDIYÚ y

P PEDROP E D R O ilco Curuguaty ma Rosario yo

San A Villa N Hayes Estanislao Itaquayry A CORDILLERA CAAGUAZU R To Coronel ARGENTINAA R G E N T I N A Caacupé A Cascavel ASUNCIÓN Oviedo P Ita Ciudad del CENTRAL Paraguari Cerro Pero Este (842 m) O This map was produced by the Map Design Unit of The World Bank. To Villarrica T Formosa IRA L Santa The boundaries, colors, denominations and any other information UA A Rita Villa PARA- G Abaí 26S shown on this map do not imply, on the part of The World Bank 26S Group, any judgment on the legal status of any territory, or any Oliva Caazapá N A endorsement or acceptance of such boundaries. GUARI A P E Z A E A á M C n ra M San Juan a B I Bautista P U S ITAPUAI TA P U A I PARAGUAY Pilar C O San Ygnacio San Pedro Ú N del Paraná

Desmochado ES SELECTED CITIES AND TOWNS Coronel Encarnación Paraná Bogado DEPARTMENT CAPITALS To NATIONAL CAPITAL Santo Tomé RIVERS 28S PAN-AMERICAN HIGHWAY MAIN ROADS RAILROADS 0 50 100 150 Kilometers DEPARTMENT BOUNDARIES 0 50 100 Miles INTERNATIONAL BOUNDARIES 60W 58W 56W 54W

JULY 2006