Khyber Pakhtunkhwa Reconstruction Program: Mid-Term Performance Evaluation Report

Home , Asharay, Bannu District, Banr, Bara Bandai, Biha, Buner District

October 20, 2014 This publication was produced at the request of the United States Agency for International Development by Tariq Husain, Aftab Ismail Khan, David Garner, and Ahmed Ali Khattak. It was prepared independently by Management Systems International (MSI) under the Monitoring and Evaluation Program (MEP).

ACKNOWLEDGMENTS

The authors would like to express their thanks to all those who facilitated the work of the team and enabled it to complete this evaluation. These include, but are not limited to, the following:

 Jamshed ul Hasan, Peshawar office Director of the Monitoring and Evaluation Program, who participated in evaluation team meetings, provided information and insight on institutional and infrastructure issues, and facilitated secondary data collection;

 Maqsood Jan, Shehla Said, and Hina Tabassum, who worked diligently under challenging conditions to collect qualitative data for the evaluation through individual interviews and focus group discussions;

 Officials of the Provincial Reconstruction, Rehabilitation and Settlement Authority (PaRRSA), Government of Khyber Pakhtunkhwa, who shared their valuable time and insights with the evaluation team, provided a wealth of information through discussion and relevant documents and arranged successful field visits in three districts of Malakand Division;

 Officials of the Elementary and Secondary Education Department, who provided school-level data from the Education Management Information System;

 Officials of the Flood Damages Restoration Directorate of the Department of Irrigation and the Pakhtunkhwa Highways Authority, who provided information on the headworks and bridges included in KPRP;

 Representatives of the Al-Kasib Group of Engineering Services and Associates in Development, the monitoring and evaluation consultants for KPRP, who responded to several queries and explained implementation modalities through interviews and electronic means; and,

 USAID officials responsible for KPRP, who facilitated the team’s work and shared their insights and information through interviews.

KHYBER PAKHTUNKHWA RECONSTRUCTION PROGRAM MID-TERM PERFORMANCE EVALUATION REPORT

Contracted under: Order No. AID-391-C-13-00005

Monitoring and Evaluation Program (MEP)

DISCLAIMER The authors’ views expressed in this publication do not necessarily reflect the views of the United States Agency for International Development or the United States Government.

CONTENTS

Project Summary ...... iii Executive Summary...... 1 Program Background ...... 5 Evaluation Purpose and Evaluation Questions ...... 10 Evaluation Methods and Limitations ...... 11 Findings and Conclusions ...... 14 Recommendations ...... 41 Bibliography ...... 42 Annexes ...... 44 Annex I: Evaluation Statement of Work ...... 45 Annex II: Sources of Information ...... 81 Annex III: Data Collection Instruments ...... 84 Annex IV: Bios of Team Members ...... 108 Annex V: Additional Information for Evaluation Question 1 ...... 109 Annex VI: Additional Information for Evaluation Question 2 ...... 119 Annex VII: Additional Information for Evaluation Question 3 ...... 130 Annex VIII: Disclosure of Any Conflicts of Interest ...... 135

Tables and Figures

Table 1: Program Summary ...... iii Table 2: KPRP Components and Their Status, December 2013 ...... 7 Table 3: KPRP Implementation and Monitoring Arrangements ...... 8 Table 4: Schools Included in Focus Groups of Parents and Head Teachers of Children in KPRP-assisted Schools ...... 81 Table 5: Total Number of Focus Group Discussions and Participants by District ...... 82 Table 6: Interviews in Islamabad, April 1, 2014 to April 5, 2014 ...... 83 Table 7: Interviews in Peshawar, April 7, 2014 to April 12, 2014 ...... 83 Table 8: Interviews in Malakand, April 15, 2014 to April 23, 2014 ...... 83 Table 9: Number of Affected Schools in Malakand Division, 2009 ...... 109 Table 10: Number of Affected Health Facilities in Malakand Division, 2009...... 110 Table 11: Number of Affected Water Supply and Sanitation Schemes in Malakand Division, 2009 ...... 111 Table 12: Cost and Duration of the Project Unit in the C&W Department ...... 113 Table 13: Eight Educational Indicators for Government Schools in Buner, Lower Dir, and Swat, 2007-08 to 2012-13 ...... 120 Table 14: Eight Educational Indicators for Government Primary Schools in Buner, Lower Dir, and Swat, 2007-08 to 2012-13 ...... 121

Table 15: Enrollment (in ’000) in Government Schools in Buner, Lower Dir, and Swat, 2007-08 to 2012- 13 ...... 121 Table 16: Percentage of Government Schools in Buner, Lower Dir, and Swat with Basic Facilities, 2007- 08 to 2012-13...... 122 Table 17: Comparison of Gross Enrollment Rates1 for Primary, Middle, and Matric Levels (All Types of Schools) 2, 2006-2011 ...... 123 Table 18: Number of Observations by District, School Type, and Year ...... 125 Table 19: Distribution of Percentage Change in Enrollment ...... 126 Table 20: Composition of Sample ...... 126 Table 21: Regression Results ...... 128 Table 22: List of Government Schools in Malakand Where the KPRP Modular School Building Design Has Been Replicated ...... 131 Table 23: Design and Cost Comparisons between the PaRRSA-KPRP and Government-adopted Primary School Modules ...... 133

Figure 1: Map of KPRP Focus Districts in Malakand, KP...... iv Figure 2: KPRP Theory of Intervention ...... 6 Figure 3: Progress of the Schools Component, April 2011-December 2013 ...... 16 Figure 4: Enrollment in Government Schools in Buner, Lower Dir, and Swat, 2007-08 to 2012-13 ...... 26 Figure 5: Number of Government Schools and Classrooms in Buner, Lower Dir, and Swat, 2007-08 to 2012-13 ...... 27 Figure 6: Enrollment per Government School in Buner, Lower Dir, and Swat, 2007-08 to 2012-2013 ...... 27 Figure 7: Percentage of Government Schools in Buner, Lower Dir, and Swat with Basic Facilities, 2007- 08 to 2012-13...... 29 Figure 8: Gross Enrollment Rates (GER) for All Age Groups in Buner District (All Types of Schools), 2006-2011 ...... 123 Figure 9: Gross Enrollment Rates (GER) for All Age Groups in Lower Dir District (All Types of Schools), 2006-2011 ...... 124 Figure 10: Gross Enrollment Rates (GER) for All Age Groups in Swat District (All Types of Schools), 2006-2011 ...... 124 Figure 11: Changes in Facilities and Enrollment by School Type (2005-06 to 2012-13) ...... 127

PROJECT SUMMARY

The Khyber Pakhtunkhwa Reconstruction Program (KPRP), previously called the Malakand Reconstruction and Recovery Program (MRRP), supports efforts by the Government of Pakistan to rebuild public infrastructure destroyed by the conflict during Taliban insurgency (2007-2009)1 and the 2010 floods in the Malakand Division of Khyber Pakhtunkhwa (KP) Province. The program works under USAID Assistance Agreement2 for the implementation of Emergency Supplemental Funding. The evaluation covers the first 45 months of the program, from March 26, 2010 to December 31, 2014. Table 1 summarizes key facts about KPRP.

TABLE 1: PROGRAM SUMMARY

Program Name/Title Khyber Pakhtunkhwa Reconstruction Program (KPRP)

Implementation Letter MLK-01 Agreement Number Activity Agreement No. 391-014 Project Manager Bahrobar Project Start Date March 26, 2010 Project Completion Date September 30, 2016 Six districts of Malakand Division, Khyber Pakhtunkhwa Province, namely Swat, Malakand, Upper Dir, Lower Dir, Buner and Shangla, and Project Location Charsadda District of Khyber Pakhtunkhwa Province outside the Malakand Division. Development Objective (DO3): Increased stability in focus locations. The most directly relevant Intermediate Result (IR) of DO3 is IR 3.3: USAID Objective Addressed Essential services that respond to priority community grievances delivered. Provincial Reconstruction, Rehabilitation and Settlement Authority Name of Implementing Partner (IP) (PaRRSA), Government of Khyber Pakhtunkhwa (GOKP)  Communication and Works (C&W) Department, GOKP  Flood Damages Restoration Directorate (FDRD), Irrigation Names of Executing Agencies Department, GOKP  Pakhtunkhwa Highways Authority (PKHA), GOKP  Public Health Engineering Department (PHED), GOKP Budget $85.19 million3

1 The Taliban entered the region in 2007 and controlled large parts of it until 2009. The Pakistan Army evicted the Taliban after a forceful counter-insurgency operation in early 2009. During the insurgency the Taliban destroyed public facilities, which were also damaged by the military action during the counter-insurgency. 2 The Islamic Republic of Pakistan, acting through the Economic Affairs Division of the Ministry of Economic Affairs and Statistics, and USAID entered into the Pakistan Enhanced Partnership Agreement (PEPA), dated September 30, 2010, as amended, the Assistance Agreement No. 391-AAG-011 dated September 30, 2009, as amended, and Assistance Agreement No. 391-010 dated September 30, 2007, as amended to provide assistance under the Malakand Reconstruction and Recovery Program (Source: Activity Agreement No. 391-014 for MRRP, by and between PaRRSA and USAID, dated June 10, 2011). 3 This is the ceiling amount for KPRP according to the KPRP Project Summary Document provided by USAID.

iii

FIGURE 1: MAP OF KPRP FOCUS DISTRICTS IN MALAKAND, KP

ACRONYMS

AGES Al-Kasib Group of Engineering Services AiD Associates in Development BHU Basic Health Unit C&W Communication and Works Department CSR Composite Schedule of Rates CRISP Community Rehabilitation Infrastructure Support Program DNA Damage and Needs Assessment DRRC District Reconstruction and Rehabilitation Committee DWSS Drinking Water Supply System EMIS Education Management Information System FARA Fixed Amount Reimbursement Agreement FDRD Flood Damages Restoration Directorate of the Irrigation Department G2G Government-to-Government GER Gross Enrollment Rate GOKP Government of Khyber Pakhtunkhwa LG&RDD Local Government and Rural Development Department KP Khyber Pakhtunkhwa KPRP Khyber Pakhtunkhwa Reconstruction Program M&E Monitoring and Evaluation MEP Monitoring and Evaluation Program MSI Management Systems International PaRRSA Provincial Reconstruction Rehabilitation and Settlement Authority PC-1 Planning Commission Form Number 1 PKHA Pakhtunkhwa Highways Authority PHED Public Health Engineering Department PIL Project Implementation Letter SOW Statement of Work TMA Tehsil Municipal Administration USA United States of America USAID United States Agency for International Development

EXECUTIVE SUMMARY

The Khyber Pakhtunkhwa Reconstruction Program (KPRP) is a government-to-government (G2G) program initiated in December 2009 through which USAID/Pakistan has agreed to provide $85.19 million to the Government of Khyber Pakhtunkhwa (GOKP) for the reconstruction and rehabilitation of infrastructure destroyed or damaged by conflict (in 2009) and floods (2010) in the Malakand Division of the Khyber Pakhtunkhwa (KP) Province. School reconstruction accounts for 59 percent of this amount, 18 percent is for two headworks that feed large canal systems in KP, 5 percent for two main bridges in the Swat District of Malakand, 7 percent for enhancing government capacity, and 10 percent for health and drinking water supply. The project is implemented and coordinated by the GOKP’s Provincial Reconstruction Rehabilitation and Settlement Authority (PaRRSA) and its activities executed through relevant GOKP line departments.

The evaluation assessed activities completed and almost-completed by December 2013 (the evaluation cut-off date) including school reconstruction; the provision of furniture, equipment, and medical kits to health facilities; and reconstruction and rehabilitation of the headworks and bridges. Reconstructed schools and assisted health facilities are located in the Buner, Lower Dir and Swat Districts of Malakand, where the evaluation field work was focused. Evaluation findings and conclusions are based on an extensive review of USAID/Pakistan and GOKP documents, published GOKP data on schools, and secondary data on the headworks and bridges. In addition, the evaluation team conducted personal interviews with 42 individuals representing USAID/Pakistan, GOKP, PaRRSA, and the GOKP line departments (including first-level health-care workers); the monitoring and evaluation (M&E) consultants (engineering firm) engaged by USAID/Pakistan; and school construction contractors. It also conducted focus group discussions with 25 contractors (in two groups), 25 head teachers (12 female and 13 male in two separate groups) and 59 parents (30 mothers and 29 fathers, separately, in six groups) of children attending KPRP- assisted reconstructed schools. In all, the evaluation team engaged 109 people in focus group discussions.

The purpose of this evaluation was to examine KPRP performance in order to determine whether mid-term adjustments in implementation are warranted and also to contribute to designing future projects of a similar nature, particularly G2G projects. USAID/Pakistan identified three main evaluation questions designed to assess: 1) what worked well and what needs improvement in the G2G modality; 2) the extent to which KPRP has influenced access to services through infrastructure and assistance to health facilities; and 3) the extent to which capacity building of PaRRSA and the line departments has occurred. The evaluation addressed each of these and a large number of sub-questions in detail, based on aforementioned sources of information for triangulation, and derived the main conclusions on the basis of which recommendations are proposed.

The G2G modality used to implement KPRP made substantial progress toward the reconstruction activities at which it was aimed and showed resilience during project implementation. Through this modality, KPRP can point to a number of institutional achievements, including those that are due to the choice of PaRRSA as an implementing partner and the authority given to it by GOKP for implementing KPRP. This authority allowed PaRRSA to establish inter-departmental committees at the district level for approving local sub-projects and selecting construction contractors for these projects, and similar committees at higher levels for approving larger projects. It also enabled PaRRSA to set technical standards and act as an inter-departmental coordinator during the planning and implementation processes. Thus, PaRRSA played a pivotal role in the processes, structure, and resources that characterize the G2G modality for KPRP.

At the start of the program (December 2009 - June 2010) there was efficient and effective cooperation among USAID/Pakistan, GOKP, and PaRRSA, including the highest levels of USAID/Pakistan, the United States Embassy, and the GOKP elected and administrative leadership. At the operational level, PaRRSA streamlined planning and approvals and displayed coordination and problem-solving capacity at both the field and provincial levels. Working with each other during the planning process, PaRRSA and USAID/Pakistan developed approaches for reconstructing schools, health facilities, bridges, and headworks with improved design, durability, and functionality

(this is referred to in the evaluation as the build-back-better approach). During implementation, technical oversight by USAID/Pakistan monitoring and evaluation consultants ensured that construction practices were followed and quality maintained. USAID/Pakistan’s M&E consultant upheld standards and provided regular informative reports. The flow of funds, including advances and reimbursements, worked well between USAID/Pakistan and PaRRSA and, usually, between PaRRSA and the GOKP line departments.

Despite these achievements, implementation has been characterized by delayed delivery of outputs. No schools were reconstructed during the first two years (2010-2011) of the program, which many reported was due to the floods which occurred in 2010 distracting attention of USAID staff and their government partners. By December 2012 (the original project closing date), only 31 schools (out of a target of 122) had been completed. By December 2013, 75 schools had been reconstructed (and 52 handed over to the GOKP Elementary and Secondary Education Department); no other construction activity was completed. PaRRSA and USAID/Pakistan originally estimated that a school would be completed, on average, in 1.5 years, but it took 2.5 years. The rehabilitation and reconstruction of the two headworks and two bridges is considered to be almost completed; however, the GOKP had completed most of the work on these projects before seeking USAID/Pakistan assistance.

Analysis of findings suggests that the following six factors affected school reconstruction as it was planned:

1. Original estimates of the time required to complete the school reconstruction did not adequately take into account new designs (including larger school size, compared with the destroyed schools), construction, and materials standards.

2. KPRP consistently estimated costs below market rates, which deterred an unknown number of contractors from offering their services and required C&W to negotiate with bidders to reduce their quoted rates. It also prompted some contractors, who were not acquainted with KPRP quality standards, to try and cut corners until USAID/Pakistan’s M&E consultants enforced rectification and compliance.

3. KPRP did not allow line departmentsto hire their own supervisory consultants. 4. C&W did not have continuous presence of the Project Director of its KPRP-dedicated Project Unit and timely payment to contractors.

5. Local contractors sub-let (sold) their contracts for a fee to smaller contractors, whose capacity constraints affected the pace of construction.

6. Construction activities were delayed by the effects of floods in 2010. Weaknesses in the planning, review and coordination processes also affected progress. KPRP, reportedly, does not prepare and review annual work plans for its construction and non-construction activities, which would help manage such a complex, inter-departmental project. Thus, KPRP management and the Project Steering Committee lack an important tool for proactive coordination of an overall planning and review process. Moreover, PaRRSA’s M&E function, which is another instrument for supporting high-level coordination, produced field-based reports only for a short period (January-May 2013) and quarterly progress reports of uneven quality, and was seldom used for decision making. In addition, as highlighted in the next paragraph, inter-departmental coordination was slow in obtaining water supply and electricity connections for reconstructed schools during the evaluation period.

In terms of outcomes, students, their parents and teachers found KPRP-assisted schools attractive and these schools accommodated increased enrollment; however, there were also deficiencies in implementation. Specifically, KPRP made positive contributions to parental choices and school enrollment through a build-back- better approach, adherence to quality, and larger schools, which is an important step toward accommodating the (unexpected) large increases in enrollment between 2009-10, the first post-conflict school year, and 2012-13. However, a large proportion of the reconstructed schools (47 to 59 percent during 2013) lacked electricity or water supply connections, which requires coordination with agencies responsible for these services. The percentage was reduced to approximately 15 percent of the schools toward the end of 2013, indicating greater attention to coordination. Overcrowding, which is common in government schools and reflects unexpectedly large

increases in enrollment, was also observed in KPRP-assisted schools. Though not widespread, lack of purdah4 for women teachers and students was reported by three sources. The evaluation team did not find any evidence regarding whether or not the overcrowding and purdah issues have been addressed, or if there is a plan to address them with KPRP or other resources.

In the health sector, the combination of equipment, supplies, and furniture for health facilities has led to improved healthcare and reduced patients’ travel requirements. Health facilities were able to provide continuous and improved healthcare services for local populations, particularly women and children, because of KPRP assistance. The intervention also led to large increases in outpatient visits, especially among women and children, and more patients now are reported by health workers to be receiving free quality healthcare in their communities, instead of having to travel to distant hospitals.

KPRP assistance for the Amandara and Munda Headworks has improved infrastructure that provides regulated flow of irrigation water and electricity as well as mobility. Specifically, the infrastructure regulates flow of water to 470,000 acres of land, a population of 315,000, and three power stations (producing 112 megawatts of electricity). While irrigated areas, the beneficiary population, and electricity production have not increased as a result of KPRP, both structures incorporate build-back-better features that enhance their safety and durability. In addition, the bridge on the Munda Headworks provides improved mobility to the neighboring population.

The Khwazakhela and Mingora Bridges are being reconstructed with enhanced safety and strength requirements and will enhance mobility for a large population. The build-back-better features include improved design and materials that have enhanced the safety and durability of the bridges, and the addition of sidewalks and greater width of the new bridges will facilitate pedestrians’ movement, reduce traffic congestion, and lead to improved traffic flow. Available traffic volume data do not allow inferences to be drawn about any increases in traffic due to the new bridges. Data on traffic flows at the temporary arrangements suggest, however, that at least 550-650 vehicles will use the Khwazakhela Bridge on a daily basis, and at least 3,400 to 3,700 vehicles per day will use the Mingora Bridge.

Some capacity building of PaRRSA that supports USAID/Pakistan activities has occurred; however the agreed-upon 2012 capacity building plan has not yet been fully implemented. Specifically, KPRP assistance is the largest funding source sustaining PaRRSA. However, PaRRSA has not yet fully implemented specific activities in the capacity building plan agreed with USAID/Pakistan in 2012. M&E performance remained weak (as noted above) and steps taken to improve gender balance in staffing did not lead to notable results. Moreover, part of the plan was to reduce PaRRSA’s contractual staff funded by USAID/Pakistan, but this had not yet been realized at the time of this evaluation. Various people expressed that this was due to a change in priorities in view of the demands of ongoing KPRP activities during 2012-2013. A more realistic process, not yet formalized between PaRRSA and USAID/Pakistan, is in place that provides for reducing the number of contractual staff during 2014.

This evaluation found no evidence regarding whether or not KPRP has any intervention to directly support capacity building of GOKP line departments. The evaluation asked whether any line department, using GOKP’s own funds, had adopted any new design or construction practice introduced through KPRP, which would suggest indirect capacity building. There was no evidence up to December 2013 that line departments had adopted any KPRP design or construction practices using GOKP’s own funds. However, in April 2014, GOKP adopted a KPRP modular design for replication in 100 primary schools throughout KP, for implementation during 2014-15 with its own funds which may indicate improved capacity.

The conclusions summarized above lead to two main recommendations. The first of these recognizes the pivotal role of PaRRSA in the G2G modality for KPRP and the need for strengthening its key functions. The second recommendation aims to address problems associated with the modality for school reconstruction.

4 Purdah (literally “curtain” in Urdu) refers to women covering their bodies as well as women’s physical segregation in public places with the use of curtains, screens and walls.

The first main recommendation is for USAID/Pakistan to work with KPRP management to improve regular preparation and review of annual work plans with the approval of the Project Steering Committee that include all KPRP-assisted construction and non-construction activities. Plans for non-construction activities should include: (a) a revised capacity building plan with annual milestones, reflecting the funds available, the projected KPRP workload and staffing requirements, and measures to improve gender balance in staffing; (b) plans for regular field-based M&E and improved and regular progress reporting that incorporates findings and recommendations from field-based M&E; and (c) a systematic approach for identifying lack of water supply and electricity connections, over-crowding in reconstructed schools, and teacher, parent, and student perceptions regarding purdah, and for addressing these limitations with funds from GOKP (which already has a program for adding classrooms and missing facilities5), USAID/Pakistan, other donors, and possibly the communities.

The second main recommendation is for USAID/Pakistan to continue to engage actively with PaRRSA to improve the modality for implementing the school reconstruction component managed during the evaluation period by the Communication and Works Department. Measures that need to be pursued in this process include: (a) the use of market rates rather than standard government rates for costing and tendering sub- projects; (b) the executing agency hiring its own supervisory consultants, retaining a project director for its project unit at all times and, when the position is vacant, delegating the powers of this officer to another officer, so that work does not suffer; making timely payments to contractors, and selecting contractors, including reputable large contractors from within and outside the province who have the technical and financial capacity required for doing the work; and (c) facilitating timely payments to the contractors and overcoming delays in this regard. One suggestion toward this end is for KPRP to establish a grievance redress mechanism in Malakand with the help of the divisional or district administration to resolve payment problems between the executing agency and its contractors as they arise.

5 According to a recent newspaper report, GOKP is already supporting a program in which “schools [are] selected for provision of facilities such as construction of additional classrooms, boundary walls, computer labs, washrooms, water supply schemes, etc.” and “philanthropists, well-to-do families and donor agencies take responsibility for providing missing facilities in government schools of their choice” (“Tameer-i- School [School Construction] Program Extended to Entire KP,” by Mohammad Ashfaq, daily Dawn, July 12, 2014).

PROGRAM BACKGROUND

The Khyber Pakhtunkhwa Reconstruction Program (KPRP), previously called the Malakand Reconstruction and Recovery Program (MRRP), supports efforts by the Government of Pakistan (GOP) to rebuild public infrastructure destroyed during the conflict6 in 2007-2009 in the Malakand Division of Khyber Pakhtunkhwa (KP) Province and by floods in 2010 that destroyed infrastructure in Malakand Division and the downstream district of Charsadda in KP. The program focuses on rebuilding one major and one minor bridge at Khwazakhela and Mingora, respectively; two irrigation headworks at Munda and Amandara; several facilities for education, health, and drinking water; and building capacity of government departments that facilitate and coordinate rehabilitation7 and reconstruction8 work in the focus areas.

THE PROBLEM STATEMENT

Malakand Division (one of the seven administrative divisions of KP province) consists of the districts of Swat, Upper Dir, Lower Dir, Buner, Shangla, Chitral, and Malakand. The Taliban occupation of a large part of this area and the resulting damage and challenge posed to the writ of the state led to a counter-insurgency operation from May to July 2009. The fighting between the military and the militants resulted in loss of human lives and large scale population displacement9 as well as destruction of infrastructure including health and education facilities, and water supply systems.10 According to the Damage and Needs Assessment (DNA)11 report dated November 2009, estimated damages resulting from the conflict amounted to Rs. 86.918 billion ($1.09 billion) in the Malakand Division and the Federally Administered Tribal Areas of Pakistan. Details of damage caused to schools, health facilities, and water supply systems are provided in the KPRP evaluation Statement of Work (Annex I).

The DNA 2009 report estimated that 411 (8 percent) of the 5,250 schools, 63 (29 percent) of the 217 health facilities, and 451 (30 percent) of the 1,508 water supply systems in Malakand had been either destroyed or damaged.12 In addition to this damage, devastating floods in 2010 also damaged schools, roads, bridges, and irrigation headworks in Malakand Division.

THE THEORY OF THE INTERVENTION

KPRP is managed by the USAID/Pakistan team that is responsible for activities in Khyber Pakhtunkhwa and the Federally Administered Tribal Areas. The program contributes to the USAID/Pakistan Mission Strategic Framework Development Objective13 3: “Increased stability in focus areas.” According to the KPRP Results Framework, the project goal is, “to accelerate reconstruction and recovery efforts in Malakand Division.” Project objectives, according to the KPRP quarterly progress reports are:

6 Taliban insurgents entered the Malakand region in 2007 and controlled large parts of it until 2009. In early 2009, the Pakistan Army carried out a counter-insurgency operation and expelled the Taliban. 7 Rehabilitation means restoration to a previous condition; repair is similar and generally indicates a lesser degree of work than rehabilitation, e.g., painting, replacing broken glass, etc. 8 Reconstruction means building a new similar structure at the site of a previous structure that has been demolished for whatever reason. 9 Almost three million people left their homes and took refuge in safer districts (World Bank, Emergency Project Paper on a Proposed Credit to the Islamic Republic of Pakistan for a Khyber Pakhtunkhwa and FATA Emergency Recovery Project. Islamabad: World Bank, 2010.) 10 Preliminary Damage and Needs Assessment (DNA) report, November 2009; prepared by Asian Development Bank and World Bank for Government of Pakistan; http://www.pdma.gov.pk/PaRRSA/documents/DNA.pdf 11 The World Bank and the Asian Development Bank in collaboration with the Provincial Disaster Management Authority/PaRRSA carried out the DNA. 12 PaRRSA and other government staff subsequently carried out field visits to affected facilities and concluded that the damages were less than the estimates provided in DNA 2009. This point is elaborated in Annex V. 13 USAID/Pakistan Mission Strategic Framework approved on July 30, 2013.

 To support the rehabilitation and reconstruction of social services infrastructure including health, education, water and sanitation, governance, irrigation and communication; and

 To build the capacity of the Provincial Reconstruction Rehabilitation and Settlement Authority (PaRRSA). According to the KPRP Results Framework, the project has the following intermediate results:

 Intermediate Result 1: Increased enrollment at focus public schools;

 Intermediate Result 2: Improved public access to water and sanitation services in focus communities;

 Intermediate Result 3: Improved health services at focus health facilities;

 Intermediate Result 4: Enhanced government capacity to provide public services. Figure 2 below illustrates the KPRP theory of intervention, depicting how project interventions are expected to lead to KPRP’s intended results and the higher-level USAID Development Objective 3:

FIGURE 2: KPRP THEORY OF INTERVENTION

Increased stability

Improvements in state legitimacy, governance, economic opportunities and people's attitudes and behavior

Enhanced access to essential services

Key infrastructure facilities reconstructed, rehabilitated and strengthened

THE DESIGN AND CURRENT STATUS OF THE PROGRAM

KPRP is undertaking reconstruction and rehabilitation of infrastructure destroyed or damaged by conflict and floods with a build-back-better approach. In general terms, build-back-better is an approach in which a reconstructed facility represents improved features of design, durability, and functionality. In the KPRP context, this means schools with more rooms and basic facilities (such as latrines, boundary walls, water supply, and electricity), schools, health facilities, and office buildings constructed with better quality material and greater resistance to earthquakes, and stronger bridges and headworks that can better withstand floods. This is not an exhaustive definition and additional information is provided for each component in response to evaluation question 2 (page 10).

Program components and their current status are summarized in Table 2 below, which also reports the specific amounts and purpose of assistance formally agreed between GOKP and USAID/Pakistan, as reported in PaRRSA’s eleventh Quarterly Progress Report (October-December 2013). Additional information on the evaluated components is included in response to evaluation questions 2 and 3 (page 10).

TABLE 2: KPRP COMPONENTS AND THEIR STATUS, DECEMBER 2013

FARA14/ FARA/ Line as FARA/ PIL15 PIL % of Progress Purpose PIL End Signing Amount Grand (As of December 31, 2013) Date Date $ ’000 Total Militancy-Damaged Infrastructure Reconstruct 5 schools 23-Jul-10 31-Dec-12 1,070 2% Target: 117 Reconstruct 74 schools 25-Jun-10 31-Dec-12 13,614 22% Constructed: 75 Reconstruct 36 schools 10-Jun-11 30-Jun-13 8,972 14% Under Construction: 42 Equipped & Handed over: 52 Reconstruct 7 schools 14-May-12 31-Dec-13 1,281 2% Sub-Total for 122 Militancy Damaged Schools 24,937 40% Reconstruct 5 BHUs16 22-Aug-11 30-Jun-13 1,188 2% Completed: None Rehabilitate 11 BHUs, 2 CDs17 10-Aug-12 31-Dec-14 1,254 2% Under Construction: 18 Health recovery and supplies (6 Equipped Health Facilities: 47 8-Aug-12 31-Mar-13 2,863 5% districts of Malakand Division)18 Sub-Total for 18 Health Facilities 5,305 9% Construct 13 DWSSs19 in Buner 10-Jun-12 30-Jun-14 1,093 2% Target: 21 DWSSs & 138 hand Construct 10 DWSSs in Swat 16-Jul-13 31-Dec-14 1,396 2% pumps. Completed: 2 hand pumps Sub-Total for 23 DWSSs (including 2 for hand-pumps) 2,489 4% Building PaRRSA capacity 1-Aug-11 31-Dec-14 3,600 6%

Construct Tehsil Complex 28-Oct-13 31-Dec-14 2,339 4%

Sub-Total for Enhancing Government Capacity 5,939 10%

Total for Militancy-damaged Infrastructure 38,670 62%

Flood-Damaged Infrastructure Reconstruct 24 schools 29-Oct-13 31-Dec-14 4,326 7% Work not started Sub-total 24 Flood Damaged Schools 4,326 7%

Rehabilitate Amandara Headworks 8-May-12 31-Dec-13 6,193 10% 96 percent completed Rehabilitate Munda Headworks 8-May-12 31-Dec-13 8,777 14% 97 percent completed Sub-total for 2 Headworks 14,970 24%

Reconstruct Khwazakhela Bridge 7-Jul-12 30-Sep-14 3,671 6% 77 percent completed Reconstruct Mingora Bridge 26-Aug-13 31-Dec-14 536 1% 90 percent completed Sub-total for 2 Bridges 4,207 7%

Total for Flood-damaged Infrastructure 23,503 38%

Grand Total 62,173 100%

14 Fixed Amount Reimbursement Agreement (FARA), is a document signed by USAID/Pakistan and PaRRSA specifying outputs for which USAID/ Pakistan agrees to reimburse the cost upon satisfactory completion of stipulated milestones. 15 Project Implementation Letter (PIL), is a document signed by USAID/Pakistan and PaRRSA agreeing to an activity and its budget or arrangements for implementation. 16 Basic Health Units. 17 Civil Dispensaries. 18 This does not include the amount of $440,000 for health equipment and supplies to Basic Health Units that were procured directly by USAID/Pakistan (and not through the G2G modality) pursuant to Implementation Letter Number MLK-04, dated June 21, 2010. 19 Drinking Water Supply and Sanitation Systems.

PROGRAM IMPLEMENTATION

KPRP is the first USAID/Pakistan program that is being implemented under the government-to-government (G2G) modality. It is being implemented through agencies of the provincial Government of Khyber Pakhtunkhwa (GOKP) with coordination and oversight provided by the GOKP Provincial Reconstruction Rehabilitation and Settlement Authority (PaRRSA), which also has the primary responsibility for dialogue with USAID/Pakistan on all program activities.20 The GOKP agencies responsible for the actual rehabilitation and construction work are the Communication and Works (C&W) Department, the Flood Damages Restoration Directorate (FDRD) of the Irrigation Department, the Public Health Engineering Department (PHED), Tehsil Municipal Administration (TMA) of Swat, and the Pakhtunkhwa Highways Authority (PKHA). The Elementary and Secondary Education Department and the Department of Health are the owners of the schools and health facilities rehabilitated or reconstructed under KPRP.

USAID/Pakistan has contracted two full-time engineering and monitoring firms for KPRP: Al-Kasib Group of Engineering Services (AGES) and Associates in Development (AiD). AGES provide monitoring support21 for all infrastructure activities except Amandara Headworks, for which is AiD is providing monitoring support. These firms supervise construction22 activities of the local contractors and provide monthly reports to USAID/Pakistan. Table 3 below summarizes the implementation and monitoring support for all KPRP components.

TABLE 3: KPRP IMPLEMENTATION AND MONITORING ARRANGEMENTS

Monitoring Component Executing Agency GOKP Host Department23 Firms Reconstruction of Schools C&W Department Elementary and Secondary AGES Education Department Reconstruction and rehabilitation C&W Department Department of Health AGES of BHUs Rehabilitation of Drinking Water PHED (for Buner District) and PHED and TMA of Swat AGES Supply Systems TMA of Swat District (for Swat) District Rehabilitation of Munda Headworks FDRD Irrigation Department AGES Rehabilitation of Amandara FDRD Irrigation Department AiD Headworks Reconstruction of Khwazakhela and PKHA PKHA AGES Mingora Bridges Construction of Integrated Tehsil Details not available Details not available AGES Complex

20 KPRP Activity Approval Agreement (AAD), dated December 17, 2009. 21 Monitoring support is focused on an engineering role and is distinct from conventional project M&E. 22 AGES and AiD supervisors are present at the construction sites during construction activities. 23 The “host department” is the department designated to take over completed facilities and provide services.

FOCUS AREAS AND GROUPS

The initial focus area of the project comprises the five districts of Swat, Buner, Shangla, Lower Dir, and Upper Dir in the Malakand Division of KP Province. Thus, KPRP’s focus groups initially comprised the population affected by the Taliban insurgency and the consequent destruction and loss of essential services (health, education and drinking water supply systems) in the five districts of Malakand Division mentioned above. Following the floods in 2010, USAID/Pakistan expanded the focus groups to include the populations affected by flood damage to two irrigation headworks (Amandara and Munda) and two bridges (Khwazakhela and Mingora). The focus groups for these interventions include large populations of farming communities and commuters both inside and outside of Malakand Division.

EVALUATION PURPOSE AND EVALUATION QUESTIONS

The overall goal of this mid-term performance evaluation is to assess progress towards achievement of the program’s objectives, to determine successes and shortcomings and the underlying causes, and to provide recommendations for improving program design and implementation strategy. Evaluation findings may also contribute to designing future projects of a similar nature, particularly G2G projects. To address the objectives mentioned above, the evaluation is expected to answer the following three key questions:

1. In what ways and to what extent have the administrative processes, site identification, coordination, and monitoring of sub-contractors, particularly through the G2G modality, influenced project implementation to date?

2. In what ways and to what extent have the project’s schools, supplies to basic health units (BHUs), bridges and irrigation components24 contributed to the achievement of intended results to date through completed and almost completed activities?

3. In what ways and to what extent has KPRP built the capacity of PaRRSA and other government line departments to date to provide public services? Explanations for these questions are summarized below under the findings and conclusions for each evaluation question. The evaluation Statement of Work is reproduced in Annex I.

24 Capacity building is excluded from this question because it is addressed separately in question 3.

EVALUATION METHODS AND LIMITATIONS

The evaluation team applied a mixed-methods approach, using both quantitative and qualitative data from multiple sources to ensure multiple levels of triangulation. To answer evaluation questions 1 and 3, the team conducted individual interviews with USAID/Pakistan, PaRRSA, and GOKP line department officials, representatives of the two monitoring and evaluation firms (AiD and AGES), and KPRP construction subcontractors. Two focus group discussions with 25 KPRP construction contractors and subcontractors also provided useful information for questions 1 and 3. In view of the expected difficulty in accessing contractors and subcontractors, and as agreed with USAID/Pakistan, the team relied on Malakand-based PaRRSA officials to select the focus group participants. The team also reviewed a large number of project documents, including all 11 progress reports25 prepared by PaRRSA between March 2011 and December 2013, all four of the PaRRSA field-based monitoring and evaluation reports (all of them prepared in 2013), all 40 monthly reports submitted by AGES to USAID/Pakistan between July 2010 and December 2013, all the agreements signed between PaRRSA and USAID/Pakistan, and the capacity building plans submitted by PaRRSA to USAID/Pakistan, minutes of all six Project Steering Committee meetings and seven tender approval committee meetings in Buner, Lower Dir and Swat Districts, and other relevant documents provided by PaRRSA (including its website).

Although evaluation question 1 asks for an assessment, not a description, of the G2G modality, the evaluation team considered it important to prepare a description of relevant processes, as no single document provides a description of the operational policies and procedures that span the processes the evaluation team was asked to assess. The description and the sources and process used to construct the description are included in Annex V. Similarly, in relation to evaluation question 3, no single document described the KPRP approach to PaRRSA’s capacity building, which evolved during project implementation. The evaluation team included a brief description of capacity building in its findings in order to answer question 3.

The main limitation encountered in answering question 1 is that the extent of problems identified in sub-project planning and implementation through document review, individual interviews and focus groups cannot be quantified precisely or at regular time intervals. The evaluation presents numerical data wherever available but these data are illustrative rather than precise, and illustrate snapshots rather than trends in time. However, the evaluation team used multiple sources of information to verify findings that are used in support of the conclusions offered in the report. Similarly, there was insufficient written information to explain issues associated with PaRRSA’s capacity building in response to question 3. For a better understanding of these and other issues, the evaluation team held a detailed meeting with USAID/Pakistan and five senior PaRRSA officials responsible for KPRP on June 5, 2014, recorded the information provided at the meeting, and invited USAID/Pakistan and PaRRSA to review and correct the written record as appropriate.

The team used a greater variety of sources for addressing evaluation question 2. To assess the education-related outcome of KPRP, it reviewed media reports on education in Malakand in the immediate post-conflict period, conducted individual interviews and focus groups discussions, and analyzed three types of secondary data available from government sources (described below) 26. For individual interviews, it met with the four school administrators (three men and a woman) in Buner, Lower Dir, and Swat Districts who represent the GOKP’s Elementary and Secondary Education Department in these districts. It also conducted six focus group discussions with the parents (three each with the mothers and fathers, involving 30 mothers and 29 fathers) of children going

25 Three of these reports (for April-June 2012, October-December 2012 and January-March 2013) were available only in draft form. The evaluation team queried PaRRSA for information that appeared to be incorrect or missing from these and other quarterly progress reports. 26 The methodology outlined in the evaluation Statement of Work was modified in agreement with USAID/Pakistan to exclude a sample survey aimed at relating enrollment in KPRP-assisted schools to the build-back-features of these schools.

to six KPRP-assisted schools (two each in Buner, Lower Dir, and Swat Districts) and two with 12 female and 13 male head teachers (in separate groups) from 23 reconstructed schools in Buner, Lower Dir, and Swat27.

The team selected the schools from the list of completed schools found in PaRRSA’s eleventh Quarterly Progress Report (October-December 2013) through a combination of purposive and convenience-based sampling. The total number of schools selected for focus groups (six) was decided in view of the time available to the team for field work. The team decided to select two schools (one boys’ and one girls’ school) from each district and ensured that there would be three primary and three middle schools among the six. It consulted PaRRSA’s 2013 monitoring and evaluation reports to identify at least some schools that suffered from one or more reported deficiencies (such as over-crowded classrooms or the absence of boundary walls, latrines, and water and electricity connections). The team then consulted a map with school locations to identify those primary and middle schools in each district, with and without reported deficiencies that were close to the district headquarters and those that were relatively distant on the assumption that schools closer to the district headquarters would be better served than more remote schools. For girls’ schools, where female team members had to conduct mothers’ focus groups, the team kept in view the travel time to KPRP-assisted BHUs, where these team members had to interview healthcare workers on the same day that they conducted their focus groups. As multiple schools fulfilled the above-mentioned criteria, the team randomly picked one school each in different areas within each district from the map. Two team members, one facilitating the discussion and the other taking notes, conducted focus group discussions in each selected school.

From the list of completed schools mentioned above, the team decided to select 7-10 schools from each district from which 10-12 male and female head teachers were invited to each of the two focus groups in Saidu Sharif. The team identified some primary, middle and secondary schools (for both boys and girls) that suffered from one or more reported deficiencies and some that did not, some that had high enrollment levels, several for each year (2010 and 2011) in which most of the work orders were issued, and two of the three for which work orders were issued in 2012. The team then looked for schools that fulfilled these criteria and were located in different union councils of a district. Multiple schools met the purposive sampling criteria in many of the union councils, in which case the team randomly picked one school in a union council from the map. Two female team members conducted the focus group discussions with female head teachers, and two male team members lead the focus group discussions with male head teachers.

For part of the answer to evaluation question 2, the evaluation team analyzed enrollment data for government schools in Buner, Lower Dir, and Swat for 2007-08 to 2012-13, available from the GOKP Elementary and Secondary Education Department’s Education Management Information System (EMIS) website. It analyzed time trends and trends across districts, sex, age groups, and levels of schooling. The team conducted similar analyses with data on the Gross Enrollment Rate (GER) from the Pakistan Social and Living Standards Measurement Survey, which is a large sample survey conducted every two years by the Pakistan Bureau of Statistics, a federal government agency.

In attempting to relate enrollment to KPRP’s build-back-better approach to school reconstruction, the team used two years of selected school-level data made available to MEP by the Elementary and Secondary Education Department from its EMIS database. The statistical analysis used regression methods to relate changes in enrollment in 1,230 schools (682 boys’ schools and 523 girls’ schools) between the 2005-06 (pre-damage) and 2012-13 (post-reconstruction) academic years to reported changes in electricity, water, latrines, and boundary walls in each school. Because the evaluation team was not able to obtain data on the number of classrooms, another key feature of build-back-better, the estimated impact of individual facilities may not be accurate. Furthermore, because the evaluation team did not have information on pre-damage facilities in KPRP-assisted schools, it was not possible to estimate the increased enrollment associated with KPRP. However, predictive analysis provides strong evidence that improving facilities increases enrollment. Annex VI describes the statistical analysis in detail.

For assessing the KPRP’s health-related outcome, the evaluation team relied on individual interviews with healthcare service providers (the person in charge, in each case) responsible for seven BHUs in three districts

27 Two girls’ schools sent senior teachers in addition to the head teachers.

(Buner, Lower Dir, and Swat) where KPRP had provided medical supplies, equipment and furniture. This led to 6 interviews with medical officers and one with a medical technician where a medical officer was not available. The team also reviewed a 2013 PaRRSA M&E report and examined health facility records shared by the interviewed service providers. The team selected the seven BHUs through a combination of purposive and convenience-based sampling from among the 47 that had received KPRP assistance. It decided to visit three BHUs in Swat and two each in Buner and Lower Dir. The team consulted a map with BHU locations to identify those that were close to the district headquarters and those that were relatively distant on the assumption that BHUs closer to the district headquarters would be better equipped and staffed than more remote BHUs. The team kept in mind the travel time to KPRP-assisted girls’ schools, where female team members had to conduct focus groups on the same day that they had to interview healthcare workers. Where multiple BHUs fulfilled the above-mentioned criteria, the evaluation team randomly selected one in a remote area and one relatively closer to the district headquarters. The main limitation of the health-related assessment is that it is based on interviews with and records from 7 BHUs and the findings may not be generalizable. However, the evaluation team has offered its conclusions only on the basis of findings common to at least five of the seven BHUs that are supported by interviews and official records.

The team used secondary data, obtained from PaRRSA, other GOKP sources, the two monitoring and evaluation firms, and GOKP construction consultants to assess the access-related outcomes associated with the headworks and bridges. Secondary data available for the two headworks pertained to the reconstruction and rehabilitation of these structures with a build-back-better approach, as these sub-projects did not aim to increase access in terms of command area or beneficiaries. The traffic volume data available for the two bridges represented vehicles counted by the government’s consulting engineers at two points in time during implementation, and at the temporary bridges rather than on the bridges destroyed and rebuilt. Thus, these data could not be used for estimating an increase in access. The evaluation team used available data to present conservative estimates of bridge usage.

In all, the evaluation team interviewed 42 individuals and conducted 10 focus group discussions, which engaged 109 people (59 parents, including 30 mothers; 25 head teachers, including 12 women; and 25 contractors and subcontractors). Annex II provides additional information on these. Annex III reproduces the instruments used for interviews and focus group discussions.

Through individual interviews and focus group discussions, qualitative methods provided valuable insight into project performance and helped capture the diversity of perceptions about various KPRP project components. However, purposive sampling used in qualitative methods has the shortcoming of selection bias and is, therefore, not representative or generalizable to the population at large. To address this shortcoming, the evaluation triangulated results against secondary qualitative and quantitative data from the relevant government departments. Responses given by a particular individual or organization that could not be substantiated through triangulation were excluded from consideration in the conclusions offered in the evaluation.

FINDINGS AND CONCLUSIONS

FINDINGS AND CONCLUSIONS FOR QUESTION 1

Question 1: In what ways and to what extent have the administrative processes, site identification, coordination, and monitoring of subcontractors, particularly through the G2G modality, influenced project implementation to date?

This question assesses how well the current G2G modality has worked in terms of its processes, the efficiency of processes, the delivery of outputs, and relationships among key stakeholders. The relevance of these terms in the context of this question is clarified below28.

“Outputs” mean the completed physical outputs of the program, such as completed schools, and the almost- completed headworks and bridges. Outputs on which work is ongoing, such as health facilities, water supply schemes, and the integrated tehsil complex, are outside the scope of this evaluation. In responding to evaluation question 1, however, the evaluation addresses process issues for some of these outputs, too, wherever relevant information was available and it was useful for illustrating how the G2G modality is working.

The processes addressed in this question include: needs assessment (identification of damaged infrastructure in the region as well as specific sub-projects for KPRP-assisted reconstruction); site selection for sub-projects, including all steps after the approval of a sub-project and up to and including preparation of tender documents; the sub- project planning process; and the entire sub-project implementation process and its financial management and reporting aspects. Implementation includes tendering for sub-project construction and the award of contracts for sub-projects; mobilization of selected contractors; construction of infrastructure through contractors and subcontractors (including the practice of contractors sub-letting their work to unapproved sub-contractors); technical oversight and supervision; monitoring and evaluation mechanisms; reporting; and the process of funds transfers among GOKP agencies/departments as well as to contractors/sub-contractors29.

Efficiency includes timeliness and also the extent to which a process itself is efficient and whether it can be streamlined. Relationships and coordination refer to coordination and communication among various stakeholders, USAID/Pakistan’s relationship with the GOKP and other stakeholders, and the relationship of GOKP agencies/departments and contractors/subcontractors to the outside monitoring and evaluation firm.

The evaluation also assessed whether modifications in G2G processes are warranted to facilitate overall project implementation during the remainder of the project.

FINDINGS

Chronology of Significant Milestones

In December 2009, GOKP, PaRRSA, and USAID/Pakistan signed an agreement for assistance for KPRP through December 2012. The agreement was signed by the Chief Minister of KP, the United States Ambassador to Pakistan, and the USAID/Pakistan Mission Director. The remainder of this chronology sub-section gives an overview of project implementation between December 2009 and December 2013 as part of the essential context for answering evaluation question 1.

28 The explanations provided here draw on the evaluation SOW and clarifications agreed between MEP and USAID/Pakistan at their meeting of February 19, 2014, after SOW approval. 29 The word “implementation” is used in this report in two ways, one referring to the sub-project implementation process and the other to overall project implementation. The difference will be either self-evident or clarified in the context whenever this word appears.

Within six months of signing the initial agreement, PaRRSA and USAID/Pakistan completed the conditions precedent for USAID/Pakistan to give PaRRSA an advance to start the project.30 Working on a parallel track during the same time, USAID/Pakistan and PaRRSA designed, checked the design and re-designed school buildings for reconstruction, introduced modular designs, and signed the first Fixed Amount Reimbursement Agreement (FARA-I)31 on April 9, 2010 for the reconstruction of 48 schools. USAID/Pakistan contracted the monitoring and evaluation (M&E) consultant, the Al-Kasib Group of Engineering Services (AGES), on April 10, 2010, a day after signing FARA-I, and took steps to put the ongoing USAID/Pakistan-assisted Community Rehabilitation Infrastructure Support Program (CRISP) in place as a supervisory consultant for the GOKP Communications and Works (C&W) Department by August 2010. PaRRSA notified and established the KPRP Project Steering Committee in January 201032 and it had held two meetings by June 2010.

In June 2010, USAID/Pakistan and PaRRSA signed an Implementation Letter (Number MLK-05, dated June 22, 2010) specifying eligible expenditures for the PaRRSA capacity building sub-component. PaRRSA and the executing agencies (GOKP line departments) held pre-construction meetings in September 2010 to finalize the roles and responsibilities of all stakeholders.33 In June 2010, a government team found that 43 of the 48 school sites could not accommodate the L-shaped school design.34 PaRRSA developed modular designs and submitted the architectural and structural drawings and cost estimates to USAID/Pakistan for technical clearance. By June 2011, within 18 months of project start-up (Figure 3), the District Reconstruction and Rehabilitation Committees (DRRCs) notified35 and established by PaRRSA had approved 115 school construction sub-projects; construction work for 102 schools had been tendered, work orders for 81 had been issued, and 21 school sub-projects were in the tender evaluation and contract award process.36 Work on the Amandara and Munda Headworks and the Khwazakhela and Mingora Bridges had already started with GOKP resources before KPRP funding for these sub- projects was approved. The status of all infrastructure components is reported above in the chapter on program background.

30 The conditions precedent are specified in Implementation Letter No. MLK-02, dated March 26, 2010, and their fulfillment confirmed in Implementation Letter Number MLK-03, dated May 11, 2010. Information reported in this paragraph is based on the Implementation Letters and similar documents signed by USAID/ Pakistan and PaRRSA between December 2010 and May 2011, the contract signed by USAID/ Pakistan with AGES and the Activity Plan agreed between USAID/ Pakistan and CRISP. 31 Number 391-MLK-FARA-001-00, dated April 9, 2010. 32 A notification is an official written announcement, in this case of the establishment of a committee. The KPRP Project Steering Committee is chaired by the Director General of the KP Provincial Disaster Management Authority, who is also, ex officio, the Director General of PaRRSA, and includes representatives of USAID/Pakistan and the GOKP executing agencies involved in KPRP. Additional details are provided in Annex VI. 33 Minutes of the pre-construction meeting held on September 22, 23 and 24, 2010, in Buner, Lower Dir and Swat, issued by AGES, October 1, 2010. 34 This was reported in the minutes of the second PSC meeting (p. 1), held on June 4, 2010, Provincial Disaster Management Authority/ Provincial Reconstruction Rehabilitation & Settlement Authority (PaRRSA), Government of Khyber Pakhtunkhwa, Peshawar. 35 A notification is an official written announcement, in this case of the establishment of a committee. 36 These numbers are taken from the first KPRP Quarterly Progress Report (April-June 2011).

FIGURE 3: PROGRESS OF THE SCHOOLS COMPONENT, APRIL 2011-DECEMBER 2013

140 120 100 80 Target 60 Sub-projects approved by DRRC 40 20 Sub-projects awarded/ tendered Number of SchoolsofNumber 0 Schools reconstructed Schools furnished

Quarter

Based on information provided in the KPRP Quarterly Progress Reports, no schools were reconstructed during the first two years (2010-2011) of the program. PaRRSA and the senior-most GOKP and USAID/Pakistan officials expressed serious concerns about delays during January-May 2012, which led to the establishment of a dedicated KPRP Project Unit in C&W in June 2012.37 By December 2012, the original closing date of the project, only 31 schools had been completed and USAID/Pakistan and PaRRSA agreed to extend the completion date to September 2013. By December 2013, the end of the timeframe for this evaluation, 75 schools, but no other construction activity had been formally completed, although the project’s bridges and headworks were almost complete.

Needs Assessment and Identification of Sub-projects for KPRP-assisted Reconstruction Most of the KPRP sub-projects consist of those facilities that were destroyed or damaged during the conflict in 2008 and 2009 and those that were affected by floods in 2010 and 2011. The latter include the two headworks and two bridges that are funded through KPRP, on all of which GOKP started reconstruction work before receiving USAID/Pakistan assistance; there was no needs assessment leading to their inclusion in KPRP. A damage and needs assessment report prepared by the Asian Development Bank and the World Bank in 200938 at the request of the Government of Pakistan, and with the cooperation of several key stakeholders, served as the basis for needs assessment for the schools, health facilities, and water supply schemes included in KPRP. The findings of this report (referred to as DNA 2009) are reflected in PaRRSA’s 2012 annual report, called “PaRRSA at a Glance,”39 with clear relevance to KPRP-assisted reconstruction and rehabilitation of schools, health facilities, and water supply systems. The report identified destroyed and damaged schools, health facilities, and water supply schemes and estimated the cost of rebuilding them. PaRRSA staff visited the facilities identified by the DNA as damaged to verify the reported damage, and concluded, in all three sectors, that the number of facilities actually affected was less than the number reported in DNA 2009. It then consulted with relevant departments to select specific sub- projects for KPRP based on the availability of funds from USAID/Pakistan and other sources and other factors. There is no evidence in the evaluation team’s interviews and project reports that community consultation played a part in the selection of these sub-projects.40

37 These concerns and the decision to establish the Project Unit are reflected in documents attached to “Second Revised PC-I/ Detailed Cost Estimate of ‘Establishment of Project Unit for PaRRSA/USAID Projects in C&W Department,’ ” prepared by the Project Director, PaRRSA/USAID Projects Unit, C&W Department, Government of Khyber Pakhtunkhwa, Saidu Sharif, February 3, 2014, from which relevant quotes are reproduced in Annex V. 38 “Pakistan North West Frontier Province and Federally Administered Tribal Areas Preliminary Damage and Needs Assessment, Immediate Restoration and Medium Term Reconstruction in Crisis Affected Areas,” prepared by Asian Development Bank and World Bank for Government of Pakistan, Islamabad, November 2009. 39 “PaRRSA at a Glance,” PaRRSA, Peshawar, n.d. 40 Additional information on the needs assessment process and findings is provided in Annex V.

Data given in DNA 2009 show that 411 (8 percent) of the 5,250 schools in Malakand were “affected,” that is, either destroyed or damaged, of which 210 (51 percent) were girls’ schools and 236 (57 percent) were primary schools. The affected schools included 24 percent of the girls’ middle schools in Malakand, 35 percent of the girls’ high and higher secondary schools, and 28 percent of the boys’ high and higher secondary schools. KPRP assistance focused on reconstructing destroyed schools, rather than repairing damaged schools, because reconstruction provides much greater opportunity for applying the build-back-better approach41. For KPRP, the GOKP Elementary and Secondary Education Department, in consultation with PaRRSA, identified 122 destroyed schools for reconstruction42, generally excluding schools with low enrollment and those located in far-flung areas where the cost of transporting materials to the building site was very high.43 The total number of schools selected for KPRP assistance was based on the funding made available by USAID/Pakistan. “PaRRSA at a Glance” (p. 22) reports that, in 2012, 116 of the 180 destroyed schools in Malakand Division were being reconstructed with the assistance of USAID/Pakistan through KPRP. At the time of writing of this report, USAID/Pakistan had authorized funds for the reconstruction of 122 schools.44

DNA 2009 reports that 63 (29 percent) of the 217 health facilities in Malakand Division were destroyed or damaged; 19 destroyed and 44 damaged. Senior PaRRSA officials informed the evaluation team that PaRRSA staff carried out physical verification and found that the actual number of health facilities damaged and destroyed was less than the number reported in DNA 2009.45 Some of the damaged facilities had suffered minor damage, which the Department of Health repaired with its own available resources. The department identified 18 facilities (five destroyed and 13 damaged) for KPRP assistance in view of its priorities and the availability of funds from KPRP and other sources, and these are the 18 included in KPRP.

The DNA report also provides information on the water supply schemes damaged in Malakand Division. These schemes are owned by two GOKP departments: the Local Government and Rural Development Department (LG&RDD) and the Public Health Engineering Department (PHED). The LG&RDD schemes are mainly hand pumps, open wells and protected springs, and the DNA counts each one of them as a scheme. All the PHED schemes are tubewell-based with distribution networks and household and/or street connections. DNA 2009 reported that 451 (30 percent) of the 1,508 schemes in Malakand Division were destroyed or damaged, including 111 LG&RDD schemes that were destroyed (no PHED schemes were destroyed).46 PaRRSA’s physical verification showed that a large majority of the schemes reported damaged or destroyed in DNA 2009 were not actually affected by conflict but just old schemes that had fallen into disrepair.47 The departments concerned and PaRRSA, with input from the Al-Kasib Group of Engineering Services (AGES), the KPRP monitoring and evaluation consultants contracted by USAID/Pakistan, prioritized 23 schemes for KPRP, keeping in view the availability of funds from KPRP and other sources and the population expected to benefit. All of these sub-projects are for rehabilitation and were not necessarily affected by the conflict.

Sub-project Site Selection

All the schools, health facilities, water supply systems, headworks, and bridges are being reconstructed or rehabilitated on their original sites. Issues (described in Annex V) came up in relation to 16 school sites when reconstruction was not found to be feasible due to space limitations on the site, access to the site proved too

41 As explained below in response to evaluation questions 1 and 2, KPRP has followed a consistent build-back-better approach. 42 “Information Provided by PaRRSA and USAID/Pakistan to the Evaluation Team at a Meeting Held at PaRRSA, Peshawar, on June 5, 2014,” MEP evaluation team note. 43 Information provided to the evaluation team by the Chief Planning Officer of PaRRSA in an email dated May 21, 2014. 44 Other donors and international non-governmental organizations have funded the reconstruction of the remaining destroyed schools (except one, which is financed by the GOKP). 45 “Information Provided by PaRRSA and USAID/Pakistan to the Evaluation Team At a Meeting Held at PaRRSA, Peshawar, on June 5, 2014,” MEP evaluation team note. 46 The DNA report (p. 84) explains that many more LG&RDD schemes were destroyed compared with PHED schemes because the former were “mostly single-component schemes and of small size.” 47 This and the remaining sentences of this paragraph are based on “Information Provided by PaRRSA and USAID/Pakistan to the Evaluation Team at a Meeting Held at PaRRSA, Peshawar, on June 5, 2014,” MEP evaluation team note.

difficult and costly, or contractors were not willing to work at the rates offered by the Construction and Works (C&W) Department. Sites that were not feasible were replaced with other schools.48

A site-related issue also came up early in the program when only five of the 48 school sites selected under the Fixed Amount Reimbursement Agreement Number I (FARA-I) were found to be feasible for the initial design, which had been approved by all stakeholders.49 The reason for this problem was that the original L-shaped design included more classrooms and other rooms than those in the destroyed schools and did not fit the sites where reconstruction was to take place.50 This problem was overcome by the modular designs that KPRP introduced and USAID/Pakistan approved (including designs and costs) to fit larger school buildings on the sites of the smaller, destroyed schools. Composite Schedule of Rates Sub-project Planning The Composite Schedule of Rates The Planning Commission Form I (PC-I) is the main output of the (CSR) is a standard government planning process.51 The core of it consists of sub-project design document that provides the rates with and costing, which has mainly been done on the basis of the which government departments cost Composite Schedule of Rates (CSR) and is discussed below. All construction projects. Items of work designs are cleared by USAID/Pakistan. As indicated above, the are defined as composite items (e.g. initial L-shaped school building design was not found to be feasible plain cement concrete, which is made and KPRP replaced it with modular designs for larger schools with of cement, aggregate, sand and water, an increase in the number of rooms from the destroyed schools, and mixed together by manual or along with other build-back-features.52 Relevant line departments mechanical means at the site), each of prepared the PC-Is and the tender documents and bid evaluation which has a unit rate for a specified criteria for selection of contractors according to GOKP quantity (e.g. cubic feet). The Bill of procedures.53 Quantities for a project is also

prepared on the basis of composite Sub-project costing in KPRP is based on prevailing government items. Contractors participating in a bid rules and regulations, as amended from time to time. Most of the are given the Bill of Quantities and school sub-projects were costed on the basis of the most recent required to state whether the rates Composite Schedule of Rates (CSR), either that of 2009 or 2012, they are offering are at par with the rather than market rates. More recently, Basic Health Units and CSR rates or a specific percentage some of the schools have been costed under the Market Rate above or below. The GOKP C&W System 2013. Many contractors bidding on KPRP sub-projects have Department produced its first CSR in not found this costing feasible in view of their own cost estimates. 1999, which was revised in 2008 and For example, the minutes of the second PSC meeting held in June again in 2009 and 2012. In 2013, the 201054 note that “Tenders for 38 out of 108 schools had been GOKP Finance Department changed opened and the evaluation of bids completed. The item rates the name of the CSR to the Market quoted by contractors were 20 to 30 percent higher than the Rate System (MRS) and announced the Engineer’s estimate. The Executive Engineer of C&W negotiated intention to update it bi-annually, so with successful bidders, who agreed to reduce their rates and that it is reflective of prevailing market accept at par rates.” rates. The December 13, 2011 minutes of the meeting of the Tender

48 The points mentioned in this paragraph, elaborated in Annex V, are taken from the sixth KPRP Quarterly Progress Report (July-September 2012), the minutes of the sixth Project Steering Committee meeting held in October 2013, and “Information Provided by PaRRSA and USAID/Pakistan to the Evaluation Team At a Meeting Held at PaRRSA, Peshawar, on June 5, 2014,” MEP evaluation team note. 49 This was reported in the minutes of the second Project Steering Committee (p. 1), held on June 4, 2010. 50 This and the next sentence are based on “Information Provided by PaRRSA and USAID/Pakistan to the Evaluation Team at a meeting held at PaRRSA, Peshawar, on June 5, 2014,” MEP evaluation team note. 51 The Planning Commission Form I (PC-1) is the prescribed template in which government departments prepare project proposals for approval by higher government authorities. 52 Additional information on school design is included in the answer to question 2. 53 Descriptive information on the sub-project planning and implementation is taken from diverse sources and elaborated in Annex V, where the sources are also identified. 54 Minutes of the second Project Steering Committee meeting (p. 3), held on June 4, 2010.

Approval Committee of Lower Dir District show that all 18 contractor bids for the reconstruction of six schools were 20-35 percent higher than the CSR 2009 rates. The minutes of the February 16, 2012 meeting of the Tender Approval Committee of Lower Dir show that the 12 bids for four school sub-projects were 20-65 percent above the CSR 2009 rates. The minutes of the Swat District Tender Approval Committee meeting of March 28, 2012 show nine bids for the reconstruction of two schools and rehabilitation of seven facilities that are 20-50 percent higher than the CSR 2009 rates.

According to the USAID/Pakistan KPRP Program Manager, USAID/Pakistan at the start of the project had suggested that C&W should use market rates for costing sub-projects. C&W at that time had objected to this suggestion by saying that contractors would then prefer to work on KPRP-assisted projects, and other C&W projects in the region would be adversely affected. Therefore, C&W decided to use CSR 2009 for costing the reconstruction and rehabilitation of schools and health facilities.

PC-Is prepared by executing agencies are approved within GOKP by committees at various levels, depending on the cost of the project proposed in the PC-I. For KPRP, GOKP streamlined the PC-I approval process by delegating powers in a number of ways, including the notification of four district-level reconstruction and rehabilitation committees (DRRCs) and a Divisional Approval Committee for the Malakand Division, described in Annex VI. PaRRSA’s ability to establish decentralized inter-departmental committees for tender and project approval of local sub-projects (and a provincial-level committee for larger sub-projects) flows from the special authority given to it by GOKP to fast-track reconstruction and rehabilitation initiatives throughout KP. This and the GOKP’s rationale for creating PaRRSA and giving it special powers for setting standards and leading inter- departmental coordination is explained on PaRRSA’s website (http://pdma.gov.pk/parrsa.php).

Another institutional arrangement that played a role in planning (as well as implementation) was introduced in June 2012 in response to serious concerns expressed by USAID/Pakistan, GOKP, and PaRRSA about delays in KPRP implementation. At that time, GOKP established a Project Unit for KPRP in the C&W Department, based in Swat, with “full administrative, technical and financial powers for procurement of works, bidding, approvals, technical sanction, time extension, enhancement, etc.” in order to minimize the time needed to complete these processes.55 At least five senior managers and implementers involved with KPRP indicated that these arrangements had fast- tracked the approval process.

Sub-project Implementation Process The minutes of the first Project Steering Committee meeting (February 3, 2010, p. 1) record the Chief Engineer of the C&W Department saying that a minimum of 9-12 months will be required for the reconstruction of a school. In the minutes of the second Project Steering Committee meeting (June 4, 2010, p. 3), the Director General of PaRRSA is reported to have observed that the reconstruction of schools would take 18 months. According to senior PaRRSA officials, the GOKP often plans to complete a school in 12 months, which explains the comments by the Chief Engineer at C&W, but KPRP reconstruction was affected by the 2010 floods, which diverted contractors. According to the USAID/Pakistan Program Manager, USAID and PaRRSA initially agreed on 18 months but the average duration for completion has been 30 months.56 Annex V provides a step-by-step description of the implementation process, whereas key issues encountered during implementation are discussed below.

According to GOKP rules and regulations57, the tendering process, which is the first step in implementation, cannot start until the funds required are available with the department concerned. C&W can issue an invitation for bids only after PaRRSA has advanced it funds for the sub-projects it will undertake. Tenders were initially called from C&W pre-qualified contractors, and then through open bidding in view of the lack of response from pre-

55 Annex V includes evidence of concern expressed by GOKP and USAID/ Pakistan on implementation delays, the kind of discussion that led to the creation of the C&W Project Unit and its budget and powers. 56 The last two sentences of this paragraph are based on “Information Provided by PaRRSA and USAID/Pakistan to the Evaluation Team at a meeting held at PaRRSA, Peshawar, on June 5, 2014,” MEP evaluation team note. 57 This is a reference to paragraph 178(iv) of the General Financial Rules of GOKP, which reads, inter alia, “Except in cases covered by any special rules or orders of Government, no work should be commenced or liability incurred in connection with in until … funds to cover the charge during the year have been provided by competent authority.”

qualified contractors. GOKP established district-level Tender Approval Committees to streamline the tendering process.

Contractors and subcontractors reported in both their focus group discussions with the evaluation team that few contractors applied and only a few of them were qualified. In addition, there were many reports of contractors sub-letting (selling) their contracts or withdrawing, and also that some used other contractors’ licenses and authority letters (in exchange for a certain percentage of contract value) to obtain the contracts. AGES reports, and the evaluation team’s interviews and focus group discussions with contractors indicate, that an estimated 60- 80 percent58 of the school sub-project contracts have been sub-let to subcontractors without approval.

AGES reports, and the views of contractors in focus groups, highlight delays in payments by C&W to the contractors. When the selected contractors got their checks, they took their time paying the unapproved subcontractors, which delayed project implementation, as indicated in AGES Progress Report Number 9 for May 2011 (p. vi). Also, when there were problems connecting a school with a water supply source or with electricity, the school could not be declared completed and contractors’ payments were held up.

All relevant interviews indicated that technical oversight by AGES had been strict. AGES Progress Reports for February (p. 4) and March (p. 5) 2012 speak of the reluctance of contractors to undertake work at the low rates approved by KPRP when, at the same time, they were being held accountable by consultants engaged by USAID/Pakistan to enforce standards and quality with which they were not acquainted. Although GOKP systems exist, there is little in available reports to suggest that C&W played an equally active role in technical oversight and quality control. At the same time, all the interviewed USAID/Pakistan and PaRRSA officials confirmed that the fixed amount reimbursement method was strictly followed and the M&E consultants (AiD and AGES) certified the quality of work completed by contractors before USAID/Pakistan approved reimbursement. These interviews and focus group discussions with contractors indicate that work not approved by the M&E firms had to be re-done to satisfactory standards. There is no indication from these sources that there was any noticeable failure to exercise quality control.

While FDRD and PKHA hired the consultants they required for the design and supervision of the headworks and bridges, respectively, USAID/Pakistan hired the supervision consultant for C&W. The USAID/Pakistan Program Manager explained to the evaluation team that FDRD and PKHA had already hired their design and supervision consultants and started construction activity before GOKP approached USAID/Pakistan for assistance for these components.59 KPRP reimbursed the cost of the headworks and bridges as well as the consultants. For the components executed through C&W, USAID/Pakistan at the start of the project asked C&W to hire its own consultant. C&W at that time felt that it would be problematic for C&W to do so and that USAID/Pakistan should engage a consultant for C&W.

USAID/Pakistan arranged for consulting services to be provided through the USAID-assisted CRISP. CRISP reported to USAID/Pakistan and was given the responsibility for supervising C&W sub-projects, whereas PaRRSA, using its own consultants, was responsible for design. Issues came up between CRISP and C&W as well as between CRISP and AGES.60 USAID/Pakistan agreed with GOKP to end CRISP involvement and reassign its role to AGES starting January 2012, calling it “close monitoring.” AGES has its own oversight mechanism through engineers present at its offices in Peshawar and Malakand. AGES reports to USAID/Pakistan on a monthly basis and its reports are comprehensive and specific. They include occasional references to missing water supply and electricity connections and the need for coordination with various agencies to secure these services. They also indicate AGES’s adversarial relationship with C&W, due, in part, to differences between AGES and C&W on quality control (as noted in AGES monthly reports).

58 These two focus groups discussions reported that 80 percent of the school reconstruction contracts had been sub-let. According to AGES Progress Report Number 13 for September 2011 (p. v), 39 (58 percent) of the 67 school contracts awarded in Swat were sub-let. 59 This and the next paragraph are based on “Information Provided by PaRRSA and USAID/Pakistan to the Evaluation Team at a meeting held at PaRRSA, Peshawar, on June 5, 2014,” MEP evaluation team note. 60 AGES monthly reports highlight the lack of CRISP staff numbers and capacity for supervision as well as an adversarial relationship between AGES and C&W on several matters in which each party considered itself to be correct and the other at fault.

AGES reports also highlight the issue of frequent transfers of the Project Directors of the C&W Project Unit, which has adversely affected the pace of implementation (AGES Progress Report for October 2013, p. ix).61 AGES recommended that “The C&W Department should have arranged at the earliest to depute someone for stopgap arrangements to facilitate payments to the contractors and to run day to day affairs of the project” (Progress Report for October 2013, p. ix). It added in the same report (p. iii) that “[Payment] to the contractors since Eid holidays could not be made. In retaliation most of the contractors have stopped or slowed down activities. The C&W Department must designate someone for the stopgap arrangements for payments and running day to day affairs of the PU/C&W till posting of a formal PD.”

The evidence is that funds flow smoothly from USAID/Pakistan to PaRRSA, and usually from PaRRSA to the Irrigation Department and Pakhtunkhwa Highways Authority (PKHA). However, three C&W officials, one contractor and one M&E firm’s official reported problems with the flow of funds from PaRRSA to C&W, but discussion with PaRRSA and review of KPRP compliance requirements suggest that the compliance process took time. Project Steering Committee meetings convey the impression that the availability of funds was never a problem for the construction activities of KPRP.62 PaRRSA releases portions of the USAID/Pakistan advance incrementally to the relevant line departments, which make periodic demands for additional funds as work is completed and payments made to contractors. Reimbursement for all construction activities is based on the milestones for each activity. The relevant agreements between USAID/Pakistan and PaRRSA spell out the number of milestones and the fixed amounts reimbursable for the completion of each of these.63 AGES reports64, however, mention delays in payment by the executing agencies to the contractors, leading to delays in implementation.

There is ample evidence in reports that construction proceeded more slowly than planned65. Project Steering Committee minutes, project reports, and interviews with senior managers and implementers identify multiple reasons for the longer-than-expected construction period. These reasons include: a longer time period required for constructing KPRP-assisted schools because of the effects of the 2010 floods, which reduced the number of contractors bidding on KPRP activities; the short seven-month building season available for construction in Malakand because of the severe winters; KPRP’s demanding design; construction and quality control practices with which C&W and the contractors were not acquainted; C&W’s inability to ensure the continuous presence of a Project Director in the Project Unit, which led to delays in payments to contractors; adversarial relationships between C&W and CRISP and between C&W and AGES; lack of technical and financial capacity of contractors and subcontractors; and costing based on significantly less than local market rates, which deterred an unknown number of contractors from bidding, required C&W to negotiate with bidders to reduce their quoted rates, and induced some contractors66 to use practices that were considered sub-standard by the M&E firms and needed rectification.

PaRRSA’s M&E function has been visible in two ways so far: it has submitted 11 Quarterly Progress Reports to USAID/Pakistan between April 2011 and December 2013 and produced four field-based M&E reports during January-May 2013.67 There are no progress reports for 2010 and the first quarter of 2011, even though (as noted above) important developments such as a needs assessment, verification of damaged and destroyed schools and health facilities, school building design and re-design, sub-project approval and tendering, capacity building initiatives, construction activities, and disbursement for construction were taking place during this period. Of the 11 progress reports submitted to USAID/Pakistan since June 2011, three remain in draft form and present data

61 This problem is also mentioned in the minutes of the sixth Project Steering Committee meeting (p. 5) held on October 11, 2013. 62 In the minutes of the third Project Steering Committee meeting (p. 6) held on March 12, 2011, the Director General of PaRRSA is reported to have observed that funds were available, so the line departments were requested to expedite reconstruction for efficient utilization of available funds. 63 Annex V provides additional information on the flow of funds. 64 These reports include the AGES monthly reports for July and October 2011, March 2012, and November and December 2013 and are supported by both focus groups for contractors conducted by the evaluation team. 65 Refer to Annex V for additional information. 66 This statement is supported by a senior C&W official and 3 of the 4 subcontractors interviewed by the evaluation team. The number of contractors whose work needed to be rectified during sub-project implementation cannot be found in available documents. 67 PaRRSA has also produced annual reports for 2012 and 2013, called “PaRRSA at a Glance,” both of which contain chapters on KPRP. The KPRP chapter in the 2012 report consists largely of background information on post-conflict damage and needs assessment and has no M&E content. In the 2013 report, the KPRP chapter includes a narrative of approximately half a page, evidently reproduced from the eleventh Quarterly Progress Report, followed by 12 pages of photographs.

that is either internally inconsistent or at variance with another report. At least one provides incorrect financial data on PaRRSA’s capacity building, reporting zero expenditure during a quarter, and the cover page of the second progress report refers to it as the first progress report.

Of the four field-based M&E reports68, one focused on the distribution of KPRP-funded health equipment and supplies in 18 Basic Health Units in Lower Dir and Swat and three on 48 completed and ongoing school reconstruction sub-projects in Buner, Lower Dir, and Swat, 32 of which had been completed and handed over to the Department of Elementary and Secondary Education. All four reports provided information on deficiencies in program implementation, including problems with the distribution of health equipment and supplies and the quality of school construction, inadequate supply of furniture (one school69), overcrowding (one school), lack of functioning water supply connections (19 schools), prevalence of unsanitary conditions (two schools), absence of electricity connections (three schools), electrical wires left dangerously open (two schools) and issues of purdah70 for women (two schools). The evaluation team did not find any evidence of these findings being reflected in the 11 PaRRSA progress reports, discussed in meetings of the Project Steering Committee and decisions being taken to rectify deficiencies. Indeed, three AGES monthly reports (May, June and August 2013) point out that observations made by the PaRRSA M&E team had not been addressed.

Coordination and Relationships At the higher level, USAID/Pakistan and PaRRSA have, by all accounts, maintained a cordial working relationship71 and responded to each other’s concerns in terms of financing, implementation and reporting. Evidence on coordination among GOKP agencies, however, is mixed: problems between PaRRSA and C&W have been reported (particularly over flow of funds), but not between PaRRSA and the Flood Damages Restoration Directorate or the Pakhtunkhwa Highways Authority.

The Project Steering Committee is the highest-level coordination body for KPRP. It has held six meetings in 4.5 years, one-third of the number mentioned in its notification. There was recognition at the fourth Project Steering Committee meeting, held on May 14, 2012, that it should have been meeting more frequently.

The minutes of Project Steering Committee meetings show that the committee has been discussing and deciding financial and operational issues pertaining to almost all project components, with the exception of PaRRSA’s capacity building.72 While the Steering Committee’s problem-solving role is evident in its minutes, it has never discussed or asked for a KPRP annual work plan and its review. Unlike most other USAID/Pakistan projects, KPRP does not have annual work plans. Moreover, there is no evidence in its minutes that the Project Steering Committee has used PaRRSA’s M&E function or its Quarterly Progress Reports in its decision-making.

At the field level, as mentioned above, PaRRSA established inter-departmental committees at the divisional and district levels for project approval and, at the district level, for the tendering process that leads to the selection of contractors. In addition, it organized pre-construction meetings on September 22, 23, and 24, 2010, in Buner, Lower Dir, and Swat to discuss and finalize the roles and responsibilities of all stakeholders and provided the minutes of this meeting to all concerned.

68 The four reports are: “Follow up Visit to District Buner, Swat and Dir (lower) Against USAID Health Equipment Distributed,” January 2013, “Monitoring and Evaluation Report (27 Completed Schools in Swat),” March 2013, “Monitoring and Evaluation Report (08 Completed Schools in Dir Lower),” April 2013, and “Monitoring & Evaluation Report - Schools Reconstructed in District Buner under USAID Supported KPRP,” May 2013. 69 The number of schools mentioned in this sentence is the number from among schools completed and handed over. 70 Purdah (literally “curtain” in Urdu) refers to women covering their bodies as well as women’s physical segregation in public places with the use of curtains, screens and walls. In the KPRP context, low boundary walls and the absence of a suitable cover on the grills of parapet walls have, in some cases, raised concerns about the lack of purdah they provide to female teachers and students. 71 The evidence for this includes the minutes of Project Steering Committee meetings, at which USAID/ Pakistan responded positively to opportunities for assisting PaRRSA, and an email from PaRRSA to the evaluation team, dated May 21, 2014, describing the relationship with USAID/ Pakistan as “more than cordial.” 72 A USAID/Pakistan manager explained that issues that concern only PaRRSA are discussed only with PaRRSA and not brought up in the Project Steering Committee, which includes organizations other than PaRRSA.

For securing external electricity connections for reconstructed schools, KPRP also requires coordination with the Peshawar Electric Supply Company, which is not part of any of the committees mentioned above. This is acknowledged, for example, in the AGES monthly report for October 2013, which also notes that electricity connections had not been provided by that time to 18 schools in Swat, six in Lower Dir, and nine in Buner, a total of 33 schools out of the 70 reported to have been reconstructed in PaRRSA’s progress report for July-September 2013. By the time of its December 2013 report, AGES was reporting that 11 schools (out of the 75 reported by PaRRSA in its quarterly report for October-December 2013 to have been reconstructed) did not have electricity connections. AGES reports (for example, for March and November 2013) also suggest the need for better coordination (particularly through the district administration) in securing water supply sources for schools.

CONCLUSIONS

The G2G modality is a hybrid system, one that combines GOKP rules and regulations with those agreed with USAID/Pakistan. In terms of financial management, the progress of project components, and high-level decision- making, the system is transparent to relevant decision makers in USAID/Pakistan and PaRRSA, who have noticed and drawn attention when necessary to slow progress to results. The system is also characterized by dual accountability to GOKP and USAID/Pakistan. This did not create any noticeable problems for the headworks and bridge components. It did, however, result in delays when enforcement of satisfactory quality standards affected timely completion of outputs in the school reconstruction component, an issue that is discussed below.

The authority given to PaRRSA by GOKP enabled it to play a pivotal role in the processes, structure and resources that characterize the G2G modality. This authority allowed PaRRSA to establish inter-departmental committees at the district level for approving local sub-projects and selecting construction contractors for these projects, and similar committees at higher levels for approving larger projects. It also enabled PaRRSA to set technical standards and act as an inter-departmental coordinator during the planning and implementation processes.

In addition, with the involvement of GOKP and United States Embassy officials at the highest level, the first steps taken in the program, particularly between December 2009 and June 2010, led to efficient and effective G2G cooperation and a fast start to the program. PaRRSA and USAID/Pakistan cooperated in the planning process to develop build-back-better approaches for reconstructing destroyed infrastructure. The streamlined approval mechanisms (committees) established by PaRRSA also delivered results as intended. During implementation, technical oversight by USAID/Pakistan monitoring and evaluation consultants ensured that construction practices were followed and quality maintained. At the G2G and inter-departmental levels, the flow of funds mechanisms, including advances and reimbursement, performed according to the relevant rules and procedures. KPRP demonstrated a number of institutional achievements for the G2G modality, including those that are due to the choice of PaRRSA as an implementing partner and the authority given to it by GOKP for implementing KPRP.

Sub-project identification proceeded smoothly based on a standard damage and needs assessment approach used by GOKP, combined with on-the-ground checks and inter-departmental consultation coordinated by PaRRSA. Sub- project planning and implementation, however, suffered from the six factors listed below that affected school reconstruction.

 Original estimates of the time required to complete the school reconstruction did not adequately take into account new designs (including larger school size, compared with the destroyed schools), construction and materials standards..

 KPRP adopted the GOKP practice of costing below market rates, which deterred an unknown number of contractors from offering their services and required C&W to negotiate with bidders to reduce their quoted rates. It also enticed some contractors, who were not acquainted with KPRP quality standards, to try and cut corners until USAID/Pakistan’s M&E consultants enforced rectification and compliance.

 KPRP did not follow the standard GOKP practice for line departments, in this case the C&W Department, to hire their own supervisory consultants.

 C&W was unable to ensure the continuous presence of the Project Director of its KPRP-dedicated Project Unit and timely payment to contractors.

 Local contractors sub-let (sold) their contracts for a fee to smaller contractors, whose capacity constraints affected the pace of construction.

 Construction activities were delayed by the effects of floods in 2010. PaRRSA’s coordination and problem-solving role has been evident ever since the project started. PaRRSA has played this role through the Project Steering Committee, as well as inter-departmental coordination meetings and also by establishing inter-departmental bodies at the field level. There has been much less emphasis, however, on pro-active coordination of an overall planning and review process, for example, through project-level annual work plans and regular reviews by the Project Steering Committee. Coordination for electricity and water supply connections, however, has not been as systematic as coordination through the established committees. PaRRSA’s field-based monitoring and evaluation function has not been consistently visible during the program, except for a short period during January-May 2013, and its Quarterly Progress Reports have, at times, been incomplete and inconsistent. Moreover, there is little evidence of M&E being used for decision making.

FINDINGS AND CONCLUSIONS FOR QUESTION 2

Question 2: In what ways and to what extent have the project’s schools, supplies to basic health units (BHUs), bridges and irrigation components73 contributed to the achievement of intended results to date through completed and almost completed activities?

This evaluation question addresses the outcomes (results) of “improved access to schools,” “access to improved health services,” “access to irrigation water supplied through the Amandara and Munda Headworks,” and “access to surface travel by means of the Khawazakhela Bridge.”74 The education-related outcome has two main dimensions – where children went to school (or did not) before enrolling at the reconstructed schools, and how KPRP has affected enrollment, particularly as a result of the build-back-better features of KPRP-assisted schools.

The health-related outcome has been assessed in relation to the Basic Health Units that have received furniture, equipment, and medical kits with KPRP assistance. The outcomes related to the headworks and bridges have been addressed in terms of the number of beneficiaries, acreage, traffic flows and other available information on access. Gender-specific effects of project activities are also included in the scope of this question. In addition, the evaluation contextualized the reported results with reference to the number and nature of outputs completed by the project.

Improved Access to Schools

Findings

The first aspect under this outcome is where the students currently enrolled in KPRP-assisted schools went to school (or did not) before the schools were reconstructed. The evaluation team collected information on this aspect from six focus group discussions with the parents of enrolled children (three groups each of mothers and fathers) and two with head teachers (one male and one female), and individual interviews with four District Education Officers (three male and one female). The evidence from all these sources is that children’s education was disrupted for about two to four months at the height of the conflict in 2009. Reportedly, most of the children affected continued their education in makeshift arrangements and undamaged (including private and government) schools, but some could not do so because of limited space in makeshift arrangements and their exposure to severe weather. The makeshift arrangements included community volunteered accommodation, mosques, rented places, camps for internally displaced persons, tents, and open spaces in the shade of trees. In order to ease the burden on those whose schools had been destroyed, government schools that had not been damaged operated morning and evening shifts, and both boys and girls attended nearby schools that were originally meant for either boys or girls.

73 Capacity building is excluded from this question because it is addressed separately in question 3. 74 The Mingora Bridge was added to the evaluation scope after approval of the SOW.

Media and independent reports from 2009 onward (portions of which are reproduced in Annex VII) provide similar information as well as additional insight into the hardship suffered by children, parents and teachers affected by conflict. Three reports, one in Dawn75 (a Pakistani daily newspaper), another in the Korea Herald76, and the third by the Associated Press of Pakistan77, mention schools being run in tents when the government announced the reopening of schools on August 1, 2009, after months of closure due to the conflict. Another report78 talks about girls 4-16 years old sharing classrooms with boys without hesitation.

Participants of all the focus groups and interviews mentioned above Malala Yousafzai reported that both boys’ and girls’ enrollment had increased since schools re-opened in August 2009. Both the (male and female) Malala Yousafzai (born July 12, 1997) is District Education Officers of Swat and the mothers of children a Pakistani student and education attending KPRP-assisted schools felt that girls’ enrollment had activist from the town of Mingora in increased significantly. In explaining the increased enrollment, all the Swat District of Malakand. She is focus groups and interviews emphasized that parental attitudes in known for her activism for rights to favor of education had strengthened as a backlash to the Taliban’s education and for women, especially in aversion to education, especially for girls. They also mentioned that Swat, where the Taliban had at the government and other development partners had made times banned girls from attending temporary arrangements and started reconstructing destroyed school. In early 2009, Malala wrote a schools. The District Education Officer of Lower Dir reported that blog under a pseudonym for the British the GOKP Elementary and Secondary Education Department has Broadcasting Corporation describing been carrying out campaigns encouraging parents to enroll their her life under Taliban rule. In October daughters in newly constructed schools, which has increased girls’ 2012, she was shot by a Taliban enrollment. gunman after she boarded her school

79 bus. She survived, was sent to England A 2013 independent report mentions additional reasons for for intensive rehabilitation, became a increased enrollment. It highlights the role of Malala Yousafzai, and global symbol of the right to education, the recognition and awards she has received globally, as a factor and received awards and recognition that has attracted parents and children toward education. It also from several countries. quotes the female Swat District Education Officer saying that “The government has made education free up to the secondary level and (Source: is providing free books to students and stipends to girls. The KP http://en.wikipedia.org/wiki/Malala_Yousafzai). government has given plenty of monetary benefits and incentives to all cadres of teachers. We are motivating and preparing them for the task through refresher courses.”

The GOKP Elementary and Secondary Education Department’s EMIS data on government schools in Buner, Lower Dir and Swat for 2007-08 to 2012-13 and reveals some interesting trends (illustrated in Figure 4, with details provided in Annex VI). Enrollment (initially 620,810) dropped by 17,000 (3 percent) between 2007-08 and 2008-09 (2009 was the height of conflict), and increased to its previous level in 2009-10, the first school year after the conflict. In the three years since then (to 2012-13), total enrollment increased by 25 percent (to 780,141) and girls’ enrollment by 37 percent (to 344,037) in all classes and 49 percent in grades six and higher. In the same period, the reported number of schools decreased by 149 (4 percent) and the number of classrooms increased by 2,791 (24 percent). The average number of classrooms per school rose from 3.2 to 4.1. The average number of students per classroom increased from 51 to 54 in all schools and from 54 to 60 at the primary level. Average enrollment per school increased from 169 to 220 in three years, an increase of 30 percent.

75 Gloria Caleb, “A slow return to normalcy,” Op-ed in Dawn (daily newspaper), August 20, 2009 (http://www.paktalibanisation.com/?p=1616). 76 Shamim Shahid, “Attendance thin as schools reopen in Pakistan’s restive region,” Korea Herald, August 2, 2009 (http://www.koreaherald.com/common_prog/newsprint.php?ud=20090803000002&dt=2). 77 Associated Press of Pakistan, “Govt [sic] schools in Malakand to reopen tomorrow,” posted by AAJ News Archive (of AAJ Television), uploaded July 31, 2009 (http://www.aaj.tv/2009/07/govt-schools-in-malakand-to-reopen-tomorrow/). 78 M. Ibrahim, “Swat school enrolment increases after Taliban’s downfall,” Central Asia Online, November 9, 2011 (http://centralasiaonline.com/en_GB/articles/caii/features/pakistan/main/2011/11/09/feature-01). 79 Tahir Ali, “Swatis love their schools,” January 22, 2013, (http://tahirkatlang.wordpress.com/2013/01/22/education-in-swat-after-militancy-and- malala/).

Data on the Gross Enrollment Rate (GER)80 from the Pakistan Social and Living Standards Measurement Survey81 also reveals relevant time trends. The most significant finding from data is that the GER for girls in all three age groups (corresponding to the primary, middle and matric levels82) declined sharply in Swat between 2006-07 and 2008-09, and then increased to more than pre-conflict levels by 2010-11 for the primary and matric level age groups and close to the pre-conflict level for the middle school age group. For the primary level age group, it came down from 74 percent in 2006-07 to 54 percent in 2008-09 and increased to 85 percent in 2010-11. For the matric level, for the same three school years, it came down from 24 percent to 14 percent and then increased to 26 percent. The most significant finding for boys is that there was a large drop in the GER in the matric-level age group in Swat, from pre-conflict 79 percent to 39 percent in the heightened-conflict period, and then a large increase to 69 percent in the post-conflict year. Annex VI reports the GERs for girls and boys in the three age groups in three districts (Buner, Lower Dir, and Swat) for the three school years mentioned above.

FIGURE 4: ENROLLMENT IN GOVERNMENT SCHOOLS IN BUNER, LOWER DIR, AND SWAT, 2007-08 TO 2012-13

Total Enrollment Girls’ Enrollment

8 7 6 5 4 3 2 Hundred Hundred Thousands 1 0 2007-08 2008-09 2009-10 2010-11 2012-13 Academic Year

80 The Gross Enrollment Rate is the number of children attending a certain school level, divided by the number of children in the relevant age group, and multiplied by 100. The definition varies from one level to another, for which details are provided in Annex VI. 81 Pakistan Social and Living Standards Measurement Survey reports are available at: http://www.pbs.gov.pk/content/pakistan-social-and-living- standards-measurement. 82 The primary level is grades 1 to 5, middle is grades 6-8 and matric (short for matriculation) is grades 9 and 10.

FIGURE 5: NUMBER OF GOVERNMENT SCHOOLS AND CLASSROOMS IN BUNER, LOWER DIR, AND SWAT, 2007-08 TO 2012-13

Number of schools Number of girls’ schools Number of classrooms 16 14 12

10 8 6 Thousands 4 2 0 2007-08 2008-09 2009-10 2010-11 2012-13 Academic Year

FIGURE 6: ENROLLMENT PER GOVERNMENT SCHOOL IN BUNER, LOWER DIR, AND SWAT, 2007-08 TO 2012-2013

Enrollment per school

250 220 190 200 170 160 160 150

100

0 2007-08 2008-09 2009-10 2010-11 2012-13

Number Number of Students per School Academic Year

KPRP’s possible contribution to enrollment is through the number (75) and features (described below) of destroyed schools reconstructed with the program’s assistance. It has also provided furniture for 52 of the 75 reconstructed schools.83 KPRP school designs, developed by PaRRSA, are modular and flexible in nature and can be

83 The numbers of schools reconstructed and furnished are taken from the eleventh Quarterly Progress Report (QPR), covering the period October-December 2013, submitted by PaRRSA to USAID/Pakistan.

adapted to the land available on the sites of the destroyed schools.84 Modules are available for academic blocks, administrative blocks and lavatory blocks.85 Academic block modules are available for three, four and six classrooms each and administrative block modules for two rooms each; some modules combine the academic and administrative blocks. Of the 112 schools designed so far, the primary schools generally consist of four or six classrooms each (although five have 10 or 12 classrooms each), while middle and high schools consist of eight to 10 classrooms each (but three have 12 classrooms each).86

Although documentary evidence is not available on the size of all the destroyed schools that KPRP is replacing, the reported norm is for GOKP schools to have fewer classrooms than those in KPRP-assisted schools. For example, a statement prepared by the GOKP Elementary and Secondary Education Department states that “approximately 50 percent of primary schools [in the province] are functioning in two rooms.” 87 All four District Education Officers interviewed by the evaluation team, and parents and teachers of children attending KPRP-assisted schools in five out of eight focus groups, maintained that, on average, the destroyed primary schools had two classrooms each and the middle schools only three each.88 For 20 boys’ and girls’ primary and middle schools constructed with KPRP assistance, PaRRSA reported in 2013 that there were 65 classrooms in the destroyed schools (an average of 3.3 classrooms per school), which have been replaced by 119 classrooms (an average of six classrooms per school) in the new schools89, an increase of 83 percent.

In addition to a larger number of classrooms per school, compared with GOKP schools, all KPRP-assisted schools are designed to include boundary walls, latrines, water supply and electricity.90 For comparison, EMIS data for 2012-13 show that 79 percent of schools (and more than 90 percent of the girls’ schools) in Buner, Lower Dir, and Swat have boundary walls, 87 percent have latrines, 61 percent have water supply, and 64 percent have electricity, and these percentages have been increasing since 2009-10 (see Figure 5 and the details in Annex VI). KPRP school designs also have built-in seismic resistance, which is not a normal feature of GOKP schools, and geotechnical investigation and the calculation of bearing capacity for the foundation design of each school, which is not employed as a norm in GOKP schools. Moreover, KPRP’s construction practices include the requirement that contractors purchase material only from approved vendors, and testing of construction related materials be on a sample basis to ensure quality, which are not standard GOKP practices.

84 The evaluation team understands that modular designs are independent discrete units designed for the intended requirements. Modules can be added/ subtracted to provide additional space or reduce the space, as required. They can be oriented on north- south, east-west or another alignment, as required for a particular site. 85 Information reported in this and the next sentence is based on Amendment Number 1, Modified Fixed Amount Reimbursement Agreement for Reconstruction of Four- and Six-Room Module Schools in the Malakand Division, FAR Agreement Number 391-MLK-FARA-002-01, dated December 15, 2011. 86 This information is taken from a list of KPRP schools provided to the evaluation team by PaRRSA, April 2014. 87 PC-I entitled “130334 – Establishment of 100 Primary Schools (B&G) on need basis in Khyber Pakhtunkhwa (Phase-III) ADP No. 12, 2013-14 to 2015-16,” Elementary and Secondary Education Department, GOKP, February 2014. 88 They also observed that most of the old schools were poorly constructed, suffered from leaking roofs, lacked water supply, had no offices for staff, no toilets, and no furniture for students and teachers (students used to sit on jute mats). 89 This information is taken from three Monitoring and Evaluation Reports of PaRRSA, Peshawar – “Monitoring and Evaluation Report (27 Completed Schools in Swat),” March 2013, “Monitoring and Evaluation Report (08 Completed Schools in Dir Lower),” April 2013, and “Monitoring & Evaluation Report - Schools Reconstructed in District Buner under USAID Supported KPRP,” May 2013. 90 Except for the percentages calculated from EMIS data, the information included in this paragraph was provided to the evaluation team by the Al-Kasib Group of Engineering Services (AGES), the monitoring and evaluation consultants contracted by USAID/Pakistan for KPRP, on May 21, 2014.

FIGURE 7: PERCENTAGE OF GOVERNMENT SCHOOLS IN BUNER, LOWER DIR, AND SWAT WITH BASIC FACILITIES, 2007-08 TO 2012-13

100

90 80 70 60 Electricity 50 Water 40 30 Latrine 20 Boundary Wall

Percentage Schools of 10 0 2007-08 2008-09 2009-10 2010-11 2012-13 Academic Year

Three of the four District Education Officers interviewed and five of the eight focus groups with male and female parents and teachers of children attending KPRP-assisted schools identified features and incentives that motivated children to attend these schools and their parents to send the children there. These features and incentives include more, better-ventilated and larger classrooms, boundary walls, drinking water, adequate furniture, and the provision of student school bags to students. Both of the Swat District Education Officers interviewed observed that some students from private schools had switched to KPRP-assisted schools because of the quality of these schools, and many had been refused admission due to lack of space.

At the same time, there is evidence from multiple sources that not all the KPRP-assisted schools have all the build- back-features that were planned for these schools. In 2013, three PaRRSA monitoring and evaluation reports91 pointed out certain deficiencies in 48 schools, 32 of which that had been reconstructed and handed over to the Elementary and Secondary Education Department. These deficiencies included lack of functioning water supply connections (in 19, that is, 59 percent, of the 32 reconstructed schools), prevalence of unsanitary conditions (open defecation in two schools), absence of electricity connections (three schools), electrical wires left dangerously open (two schools) and over-crowding (students from two classes sitting in one classroom, in one school). AGES monthly reports mention 33 schools (47 percent of the 70 reported to be reconstructed by that time) that did not have electricity in October 2013 and 11 (15 percent of the 75 reconstructed schools) that did not by December 2013. These reports do not mention the number of reconstructed schools in which water supply connections were not available.92

The evaluation team during its field work heard reports of lack of water supply and electricity in all eight focus group discussions with parents (three each with mothers and fathers) and teachers and interviews with all four of the District Education Officers of Buner, Lower Dir and Swat. A fathers’ focus group in Swat highlighted overcrowding. The evaluation team observed overcrowding in all six of the schools it visited in Buner, Lower Dir and Swat, and recorded this (for three schools) in photographs. In these schools, the team observed that the

91 These are: “Monitoring and Evaluation Report (27 Completed Schools in Swat),” March 2013, “Monitoring and Evaluation Report (08 Completed Schools in Dir Lower),” April 2013, and “Monitoring & Evaluation Report - Schools Reconstructed in District Buner under USAID Supported KPRP,” May 2013. 92 USAID/Pakistan informed MEP by email on September 10, 2014, that 13-15 reconstructed schools had experienced electricity and water supply connections problems, which KPRP addressed in 2014 to a large extent, leaving 3-5 schools in which the connections had not yet been provided. The email does not disclose the number of schools in which either one or the other service was missing at the end of 2013.

number of students in the classrooms was more than 40, which, according to a senior AGES engineer, is the number for which the schools were designed.93

Although not a consideration in the build-back-better approach, the issue of purdah94 for women was also noted during the evaluation. A mothers’ focus group conducted by the evaluation team in Swat reported lack of purdah due to low boundary walls. One of PaRRSA’s 2013 M&E reports95 had also identified a school in Swat where the community reportedly felt that the boundary wall was not high enough for purdah. Another PaRRSA report96 identified a school in Lower Dir where, in the community’s view, the grill on the parapet allowed a view of women students and teachers from outside the school.

PaRRSA, obviously, has access to its M&E and status reports on sub-projects, as does USAID/Pakistan. USAID/Pakistan, in addition, has access to AGES reports as well as more detailed information available with AGES. However, the evaluation team did not find any document in which deficiencies such as those described above had been listed systematically at regular intervals, for example, identifying schools lacking a particular service or facing specific issues in each of the three districts. Moreover, the evaluation team’s interviews and review of relevant documents (including PaRRSA’s quarterly progress reports, Project Steering Committee minutes, project implementation letters and AGES monthly reports) indicate that KPRP did not have a plan in place during the evaluation period for systematically rectifying these deficiencies, either with KPRP resources or through coordination with other donors and GOKP. This observation is consistent with AGES reports (including those for March, October, November and December 2013) calling for attention by and coordination with various departments for securing water supply and electricity connections. All the AGES observations during 2013 suggest lack of systematic attention to and coordination for these services.

The evaluation team did not have the quantitative data necessary to relate enrollment in KPRP-assisted schools to features such as the number of classrooms and the availability of boundary walls, latrines, water and electricity. Instead, it used available EMIS data for 1,230 schools in Buner, Lower Dir and Swat for the years 2005-06 and 2012-13 to estimate the effects of changes in facilities (boundary walls, latrines, water and electricity) on enrollment. The GOKP did not include data on the number of classrooms, another key feature of build-back- better, in the dataset and this omission limits the ability of the analysis to identify the effects on enrollment of individual facilities. To the extent that changes in school size are not related to changes in facilities, it also biases the overall estimate of changes in enrollment related to build-back-better. However, this bias, if it exists, is probably minor. In spite of these limitations, however, the predictive analysis demonstrates that facilities are an important determinant of enrollment, which supports the notion that building-back-better is likely to increase enrollment even if the analysis cannot precisely quantify the effect.

Although the results for individual facilities are questionable, they suggest that in boys’ schools, adding electricity increased enrollment in by 9 percent, adding latrines increased enrollment by 9 percent, and adding boundary by 7 percent relative to what enrollment would have been had the schools been built back to their original, pre-damage configuration. In girls’ schools, adding electricity increased enrollment by an average of 10 percent. The lack of a significant effect of latrines and boundary walls in girls’ schools does not mean they are not important determinants of enrollment. Almost all girls’ schools reported having these facilities in 2005-06 (92 percent reported having latrines and 97 percent boundary walls) so there was little room for improvement and thus few observations on changes. Annex VI describes details of the analysis.

To sum up the findings, students experienced disruption in education and they and their parents suffered hardship during the conflict when schools were destroyed or closed (particularly in 2009) for several months. They utilized inconvenient makeshift arrangements and moved to peaceful areas of the country to continue their schooling. The

93 These observations on overcrowding are consistent with the 2012-13 EMIS data quoted above, which show an average of 51 students per classroom in all schools in Buner, Lower Dir and Swat, and an average of 60 in primary schools. 94 Purdah (literally “curtain” in Urdu) refers to women covering their bodies as well as physical segregation in public places with the use of curtains, screens and walls. In the KPRP context, low boundary walls and the absence of a suitable cover on the grills of parapet walls have, in some cases, raised concerns about the lack of purdah they provide to female teachers and students. 95 “Monitoring and Evaluation Report (27 Completed Schools in Swat),” March 2013. 96 “Monitoring and Evaluation Report (08 Completed Schools in Dir Lower),” April 2013.

gross enrollment rate dropped sharply in Swat for girls in all school-going age groups and for boys at the matric level, before increasing to near or higher than pre-conflict levels in the 2010-11 academic year. There was a surge in enrollment, particularly of girls, between 2009-10 and 2012-13, a period during which GOKP and its partners (including KPRP) built additional classrooms and improved facilities.

Conclusions Many factors could explain the increase in total enrollment and the gross enrollment rate since 2009-10, the first school year after the end of the conflict, such as restoration of peace, construction of larger and better schools, GOKP policies and incentives for education, and the influence of Malala Yousafzai (and possibly others) as a resolved campaigner for education. It is not possible with available data to infer the extent to which KPRP-assisted schools have influenced enrollment or how much enrollment has varied over time in communities where these schools have been constructed.

KPRP has, however, was perceived by many to have made positive contributions to parental choices and enrollment through its approach to school design and construction. In comparison with GOKP school designs and construction practices, KPRP adopted a consistent build-back-better approach, with provisions for larger and safer schools and essential services such as water, latrines, electricity, and boundary walls that are not universally available in GOKP schools. Parents, teachers and GOKP school administrators cited the build-back-better features of schools as contributors to increased enrollment in KPRP-assisted schools. This is consistent with the evaluation team’s predictive modeling, which showed a positive effect of the availability of basic facilities on enrollment.

There were also, however, some problems in implementing the build-back-better approach consistently. During 2013, electricity and water supply connections were reportedly missing in 47 percent and 59 percent of the reconstructed schools, respectively, though the proportion declined to approximately 15 percent by December 2013, suggesting greater attention to coordination with agencies responsible for these services toward the end of the evaluation period. The evaluation team’s observations and data from official sources suggest that overcrowding is common in KPRP-assisted as well as other government schools, reflecting unexpectedly large increases in enrollment. The evaluation team did not find any evidence of a plan to address overcrowding with KPRP or other (GOKP and other donors’) resources. Though not widespread, lack of purdah for women teachers and students was reported by three sources. The evaluation team did not find any evidence that these issues have been or planned to be addressed. Access to Improved Health Services Findings The evaluation team visited seven basic health units (BHUs) located in Buner, Lower Dir and Swat out of 47 BHUs that had received KPRP assistance in the form of equipment and furniture distributed by the Department of Health. The team conducted interviews with one Medical Officer (or Medical Technician, where the Medical Officer was not available) in each of the seven BHUs visited. Five of the seven interviews confirmed receipt of all 52 items listed in the BHU equipment distribution plan list97, while 1 BHU received 48 items and another one 35 items. Five of the seven interviews also suggested that BHUs had returned some furniture items, medical equipment, and supplies to the Department of Health due to insufficient space and excess supply.

According to a January 2013 PaRRSA monitoring and evaluation report98 on 18 BHUs in Swat and Lower Dir, most of the equipment and supplies distributed to these BHUs health facilities were underutilized because of the provision of the same supplies by humanitarian organizations working in Malakand, inadequate storage space for accommodating the equipment and supplies, and a mismatch between what the BHUs needed and what KPRP provided. The report observed that equipment and supplies were not distributed to the BHUs on a needs basis, and the main reason for the underutilization of equipment and supplies was the absence of a needs assessment

97 The list of equipment, furniture and heath supplies distributed to the health facilities is provided in Appendix 6 of the evaluation SOW, which is Annex I of this report. 98 “Follow up Visit to District Buner, Swat and Dir (lower) Against USAID Health Equipment Distributed,” January 2013, PaRRSA, Peshawar. The report does not identify 17 of the 18 BHUs it covered.

prior to distribution. The evaluation team could not confirm these observations independently, since such an exercise was beyond the scope of the evaluation.

In all seven interviews conducted by the evaluation team, it was reported that a majority of the patients frequenting the BHUs comprise of women suffering from gynecological problems and children suffering from respiratory diseases, diarrhea, anemia, and chest infections. All seven interviews suggested that before receiving KPRP assistance the BHUs lacked equipment and supplies to treat many of these patients and referred them to the District Headquarters Hospitals located 35 to 70 kilometers away. Evidence on changes attributed by interviewees to KPRP assistance is summarized below.

Six of the seven interviews suggested that the BHUs had started providing continuous and quality healthcare services in the areas of primary healthcare, maternal care, emergency care, mother and child care, obstetric care, minor surgery (stitches), outpatient services, vaccination (including polio drops) to children and through the field activities of Lady Health Workers after receiving KPRP assistance. One interview suggested that the BHU is not providing adequate healthcare due to the unavailability of a Medical Officer and Lady Health Worker and the reported non-utilization of equipment and supplies.

Six of the seven interviews suggested that the quality of diagnosis and treatment at the BHUs has improved, and that patients are now receiving free quality healthcare in their communities because of the availability of advanced equipment, beds, medicines, advanced kits for handling delivery cases, refrigerators for preserving vaccines, and quality furniture for staff provided through KPRP. Six respondents reported that the number of outpatient visits had increased between 2010 and 2011 due to these factors. Records observed at five BHUs showed increases in outpatient visits from 276 to 1,320 in one case, 15,000 to 19,000 in another, 15,627 to 20,187 in a third, 20,000 to 25,000 in the fourth, and 16,925 to 31,780 in the fifth one. The percentage increases range from 25 percent to 29 percent at three BHUs, 88 percent in a fourth one, and 378 percent in the one case where the base was very small.

Conclusions KPRP assistance to health facilities is associated with access to continuous and improved healthcare services for local populations (particularly for women and children) for primary healthcare, maternal care, emergency care, mother and child care, obstetric care, minor surgery (stitches), vaccination (including polio drops) for children and Lady Health Workers’ services. Moreover, large increases in outpatient visits, especially by women and children, suggest that more patients are now receiving free quality healthcare in their communities, instead of having to travel to distant hospitals. Access to Irrigation Water Supplied through the Amandara and Munda Headworks Findings The Amandara Headworks, constructed in the early twentieth century99, are located in the Malakand District and divert the waters of the River Swat into the Upper Swat Canal, which irrigates parts of the Malakand, Mardan, Nowshera and Swabi Districts of KP. The canal provides water to 300,000 acres of irrigated land, a population of 200,000 living in the command area, and three power stations producing 112 megawatts of electricity.100 The headworks suffered extensive damage during the abnormal floods of July 2010. Data are not available on the effects of the damage on agriculture and farming communities.

Within days of the damage, the GOKP Irrigation Department “partially restored irrigation supplies by restoring the head reach of the canal” 101. The department continued the restoration work102 with GOKP funds before GOKP

99 Project Implementation Letter Number 18, dated May 8, 2012, page 7. 100 PC-I for the restoration of the Amandara Headworks, Flood Damages Restoration Directorate, Irrigation Department, GOKP, Peshawar, November 30, 2012. 101 “Pakistan Floods 2010 Preliminary Damage and Needs Assessment,” prepared by Asian Development Bank and World Bank for Government of Pakistan, Islamabad, November 2010, Annex 10 (pp. 122-123). 102 As mentioned above in a footnote in the first paragraph of the Program Background section, restoration or rehabilitation means bringing a structure back to its previous condition.

received USAID/Pakistan assistance. By the time USAID/Pakistan and GOKP signed an agreement for assistance through KPRP, GOKP had already completed about 50 percent of the restoration with its own resources.103 KPRP covers the entire cost of the restoration, including reimbursement to GOKP for work it had completed with its own funds.

The restoration planned and initiated by GOKP and financed by KPRP is being undertaken without any additions to the main structure of the headworks.104 However, the earthen diversion structures, which were prone to erosion, have been strengthened by adding random rubble (stone pitching) and plantation on the earthen slopes. The worn- out manual gates have been replaced with new motorized gates. As noted in Table 2, the restoration is 96 percent complete.105

The Munda Headworks, commissioned in 1885106, are located in the Charsadda District of KP, outside the Malakand Division, and divert the waters of the River Swat into the Lower Swat Canal, which irrigates parts of the Charsadda District. The canal provides water to 170,000 acres of irrigated land and a population of 115,000 living in the command area; a bridge over the headworks facilitates mobility between the Shabqadar and Tangi Tehsils (sub-districts) of Charsadda District, which have an estimated population of 580,000.107 The headworks suffered extensive damage during the floods of July 2010. Data are not available on the effects of the damage on agriculture and farming communities.

The GOKP Irrigation Department initiated the reconstruction108 of the headworks with GOKP funds before GOKP received USAID/Pakistan assistance. By the time USAID/Pakistan and GOKP signed an agreement for assistance through KPRP, GOKP had already completed about 45 percent of the reconstruction with its own resources.109 KPRP covers the entire cost of the reconstruction, including reimbursement to GOKP for work it had completed with its own funds.

As planned by GOKP and financed by KPRP, the washed-away portion of the headworks, including the bridge, was reconstructed to a raised elevation two meters above the existing level and with pre-stressed concrete embedded into the old masonry structure.110 The old four-meter wide, single lane bridge was reconstructed with pre-stressed concrete as a two-lane bridge, 8.5 meters wide and with a pedestrian walkway. The reconstruction is more than 90 percent complete.111

Conclusions The restoration of the Amandara Headworks improves infrastructure that was providing regulated flow of water to 300,000 acres of irrigated land, a population of 200,000 living in the command area, and three power stations producing 112 megawatts of electricity. The irrigated area, beneficiary population and electricity production has not increased as a result of KPRP. However, by design (as described in the findings) and in the judgment of evaluation team engineers, the use of better material in the diversion structures has improved the durability of the

103 Project Implementation Letter Number 18, dated May 8, 2012. 104 The first three sentences of this paragraph are based on a telephone interview with the Team Leader of Associates in Development, the monitoring and evaluation consultants contracted by USAID/Pakistan for the Amandara Headworks, on May 20, 2014. 105 PaRRSA, eleventh Quarterly Progress Report (October-December 2013). 106 Project Implementation Letter Number 18, dated May 8, 2012, page 9. 107 The population estimate for Shabqadar and Tangi is an extrapolation from the 1998 Population Census, after which a population census has not been undertaken in Pakistan. The extrapolation entailed multiplying the 1998 population by 1.4, a factor consistent with the multipliers of 1.32 (for 2011 projections) and 1.56 (for 2021) used by the official National Institute of Population Studies, Islamabad, in its population projections for the Khyber Pakhtunkhwa Province (“Pakistan Population Datasheet and Estimates of Population by Provinces and Districts of Pakistan, 2001, 2004, 2011 & 2021”). The remaining information is from PC-I for the restoration of the Munda Headworks, Flood Damages Restoration Directorate, Irrigation Department, GOKP, Peshawar, December 30, 2010. 108 As mentioned above in a footnote in the first paragraph of the Program Background section, reconstruction means building a new similar structure at the site of a previous structure that has been destroyed. 109 Project Implementation Letter No. 18, dated May 8, 2012. 110 This information is taken from PC-I for the restoration of the Munda Headworks, Flood Damages Restoration Directorate, Irrigation Department, GOKP, Peshawar, December 30, 2010. 111 PaRRSA, eleventh QPR (October-December 2013).

structures, and the introduction of motorized gates provides for better control over the flow of water compared with the manual gates.

The reconstruction of the Munda Headworks improves infrastructure that was providing regulated flow of water to 170,000 acres of irrigated land and a population of 115,000 living in the command area. The irrigated area and beneficiary population has not increased as a result of KPRP. However, by design (as described in the findings) and in the judgment of evaluation team engineers, the raised height of the headworks and the use of pre-stressed concrete make the structure more durable and also safer in the event of earthquakes and floods. In addition, the new bridge provides improved mobility to the people of the Shabqadar and Tangi Tehsils of Charsadda District.

Access to Surface Travel by Means of the Khawazakhela and Mingora Bridges Findings The Khawazakhela Bridge, located over the River Swat, connects the tehsils of Khwazakhela (estimated population: 300,825) and Matta Kharari (estimated population: 185,291) of the Swat District.112 The bridge was washed away in its entirety during the floods of July 2010. As there is no alternate route, GOKP arranged temporary connections, such as a single-lane temporary connection for light traffic. GOKP started the reconstruction of the destroyed bridge with its own resources, before approaching USAID/Pakistan for assistance.

The old bridge was a 225-meter long and 4.2-meter wide113 single-lane bridge with sidewalks of 0.4 meters each on both sides, constructed with pre-stressed concrete.114 Its replacement, also a pre-stressed concrete bridge, will be 409 meters long, 8.5 meters wide, double-lane, with sidewalks of 0.9 meters each on both sides. The new bridge will be two meters higher than the old one. It is being built with more steel in the pile foundation than the old bridge had; the diameter of each pile is also greater.115 The construction is 77 percent complete.116

One week’s traffic count data collected on the bridge site in the first half of 2013117 show a flow of 548 vehicles per day for a 12-hour day (7 a.m. to 7 p.m.); data on people and freight are not available. One day’s (12-hour) data from March 2014 show that 656 vehicles were crossing the river at the bridge site118, an increase of 20 percent over the 2013 figure.

The Mingora Bridge is located over a tributary of the River Swat.119 It connects the tehsils of Babuzai (estimated population: 162,934) and Kabal (estimated population: 341,798) of the District Swat.120 The bridge was washed away in the floods of July 2010. GOKP started the reconstruction of the bridge with its own resources, before requesting assistance from USAID/Pakistan.

112 The population estimates for Khwazakhela and Matta Tehsils are extrapolations from the 1998 Population Census, after which a population census has not been undertaken in Pakistan. The extrapolation entailed multiplying the 1998 population by 1.4, a factor consistent with the multipliers of 1.32 (for 2011 projections) and 1.56 (for 2021) used by the official National Institute of Population Studies, Islamabad, in its population projections for the Khyber Pakhtunkhwa Province (“Pakistan Population Datasheet and Estimates of Population by Provinces and Districts of Pakistan, 2001, 2004, 2011 & 2021”). 113 This is the width of the roadway, excluding the sidewalks. 114 Unless otherwise stated, information for this and the next paragraph is taken from the PC-I for the construction of the Khwazakhela and Mingora Bridges, Pakhtunkhwa Highways Authority, GOKP, Peshawar, June 2013. 115 This information was provided by the Mingora-based Resident Engineer of Consulting Associates, consultants to the Pakhtunkhwa Highways Authority for the Khwazakhela and Mingora Bridges, on April 17 and May 4, 2013. 116 PaRRSA, Eleventh Quarterly Progress Report (October-December 2013). 117 The relevant PC-I, referenced above, does not mention the dates of data collection and monthly trends in traffic volumes are not known. 118 Email from Consulting Associates, Mingora, to the evaluation team, dated March 17, 2014. The data were collected on March 8, 2014, which was a Saturday. Daily trends in traffic over the week are not known. 119 Information in this paragraph is taken from Project Implementation Letter Number 391-014-PIL-27, dated August 26, 2013, by means of which GOKP and USAID/Pakistan agreed on assistance for the reconstruction of this bridge. 120 The population estimates for Babuzai and Kabal Tehsils are extrapolations from the 1998 Population Census, after which a population census has not been undertaken in Pakistan. The extrapolation entailed multiplying the 1998 population by 1.4, a factor consistent with the multipliers of 1.32 (for 2011 projections) and 1.56 (for 2021) used by the official National Institute of Population Studies, Islamabad, in its population projections for the Khyber Pakhtunkhwa Province (“Pakistan Population Datasheet and Estimates of Population by Provinces and Districts of Pakistan, 2001, 2004, 2011 & 2021”).

The old bridge was an arch-type structure, 47 meters long and five meters wide.121 Its replacement will be a pre- stressed concrete bridge, 48 meters long and 8.5 meters wide, with sidewalks of 1.5 meters each on both sides. The new bridge will be two meters higher than the old one. The construction is 90 percent complete.122

One week’s traffic count data collected on the bridge site in the first half of 2013123 show a flow of 3,406 vehicles per day for a 12-hour day (7am to 7pm); data on people and freight are not available. One day’s (12-hour) data from March 2014 show that 3,734 vehicles were crossing the river at the bridge site124, an increase of 10 percent over the 2013 figure.

Conclusions The new Khawazakhela Bridge will have a larger waterway and elevation, which will make it safer from damage by floods compared with the old bridge, and its pile foundations will make it stronger than the old structure. The new Mingora Bridge will have greater strength and durability than the older bridge, and will also be more resistant to flood damage because of its pre-stressed concrete structure. The addition of sidewalks and the greater width of the new bridges will facilitate pedestrians’ movement and safety, reduce traffic congestion and lead to smoother flow of traffic.

Available traffic volume data do not allow inferences to be drawn about any increases in traffic due to the new bridges because the data were collected from the bridge sites just before and during reconstruction, and do not allow comparison of traffic flows on the old and new bridges, and because it is not possible with these data to account for seasonal variations over the year. Data on traffic flows at the temporary arrangements suggest, however, that at least 550-650 vehicles will use the Khawazakhela Bridge on a daily basis, and at least 3,400 to 3,700 per day will use the Mingora Bridge.

FINDINGS AND CONCLUSIONS FOR QUESTION 3

Question 3: In what ways and to what extent has KPRP built the capacity of PaRRSA and other government line departments to date to provide public services?

The explanation provided for this question in the evaluation SOW highlights three points. The first of these is that the evaluation will assess the contribution of the assistance provided by KPRP to PaRRSA for capacity building. The purpose and nature of this assistance evolved over time and the SOW does not provide any definition of capacity building as far as PaRRSA is concerned. Therefore, the findings section below starts by establishing what PaRRSA’s capacity building entailed. The second point is to recognize that KPRP does not include any direct capacity building activity for GOKP line departments but the evaluation “will identify and analyze the extent to which improved construction practices introduced by KPRP have been officially accepted and notified by the line departments, as reflected in relevant technical documents and standards, and implemented in identifiable projects and districts of KP.” The expectation is that this kind of indirect capacity building might have taken place because of the demonstration effect of KPRP on participating line departments. The third point in the SOW is that the evaluation will endeavor to identify reasons for limited or lack of capacity building through KPRP, if that is what the evidence suggests.

121 Unless otherwise stated, information for this and the next paragraph is taken from the PC-I for the construction of the Khwazakhela and Mingora Bridges, Pakhtunkhwa Highways Authority, GOKP, Peshawar, June 2013. 122 PaRRSA, eleventh Quarterly Progress Report (October-December 2013). 123 The dates of data collection are not mentioned in the PC-I and monthly trends in traffic volumes are not known. 124 Email from Consulting Associates, Mingora, to the evaluation team, dated March 17, 2014. The data were collected on March 13, 2014, which was a Thursday. Daily trends in traffic over the week are not known.

FINDINGS

PaRRSA Capacity Building

In order to assess the contribution of the assistance provided by USAID/Pakistan to PaRRSA through KPRP, it is necessary, first of all, to clarify the purpose, amount and uses of KPRP assistance to PaRRSA. In the first Implementation Letter (Number MLK-01) signed by USAID/Pakistan and the Government of KP on December 17, 2009, an illustrative financial plan for KPRP included $1 million for “Enhancing Government Capacity.” The document also explains that the assistance is intended “to enable the Government of [KP] to contract with a local firm or institution to recruit, retain, and deploy additional staff as needed by Provincial authorities to carry out the program funded under this [Implementation Letter].”125 Implementation Letter Number MLK-05, dated June 22, 2010, added office equipment, an engineering library and equipment, and computer hardware and software to eligible expenditures.

PaRRSA’s first Quarterly Progress Report (April-June 2011, p. 7) for KPRP reports that USAID/Pakistan had approved an interim capacity building note126 and budget of $670,756. Personnel costs (34 percent of the interim budget), office set-up/ administration (20 percent) and communication support (17 percent) are the largest items in the budget. Strengthening monitoring and evaluation (M&E) was presented as a priority in the interim capacity building note (p. 6): “PaRRSA believes that the monitoring and evaluation department should be adequately strengthened in areas of human and infrastructural resources to adequately carry out intended monitoring and evaluation activities. PaRRSA plans to develop a dedicated team for carrying out monitoring and evaluation functions within the cross-section of its geographical and functional outreach.” The budget note included provision for hiring 10 M&E staff, including three M&E Officers (covering all sectors that are relevant to KPRP), three Social Surveyors (for collecting data through surveys, focus groups, etc. on a “daily basis”) and a data manager.

The third Quarterly Progress Report (October-December 2011, p. 5) reported that support from the United Nations Development Program (UNDP) to PaRRSA “was coming to an end and PaRRSA requested USAID to increase its support from $1 million to $3.6 million.” In January 2012, PaRRSA prepared a detailed proposal for USAID assistance through KPRP.127 The proposal (p. 5) observed that “USAID’s contribution to PaRRSA was to ensure continuous operations of PaRRSA, particularly with respect to USAID funded projects in Malakand/KP.” This interpretation of the purpose of KPRP assistance is reinforced in the fourth Quarterly Progress Report (January-March 2012, p. 8) and linked directly to the withdrawal of UNDP support.

USAID/Pakistan accepted PaRRSA’s proposal and the two parties formalized the terms of assistance in Project Implementation Letter (PIL) Number 391-014-PIL-01, dated March 21, 2012. The parties agreed to a budget revision, within the total amount of $3.6 million, in PIL Number 24, dated May 23, 2013. The revised allocations amount to $2.63 million (73 percent of the capacity building budget) for hiring contractual employees, $0.67 million (19 percent of the budget) for operational costs, $0.29 million (eight percent of the budget) for public outreach and communication, and $0.1 million (0.3 percent) for equipment. All other terms and conditions of PIL Number 1 remained in full force and effect.

PIL Number 1 goes beyond the statement of purpose provided in the first Implementation Letter (MLK-01) quoted above. It clarifies that assistance for capacity building is intended to “strengthen PaRRSA’s capacity to effectively implement reconstruction and recovery activities funded by the Government of Pakistan and/or by donors including USAID.” The PIL does not provide either a definition or a broad operational understanding of effectiveness. It does, however, say that “PaRRSA [has] funding available/ committed from [sic] various donors” in

125 This Implementation Letter also introduces the requirement for USAID/ Pakistan to conduct pre-award assessments of PaRRSA’s administrative and financial management systems as a condition precedent to disbursement and strengthen PaRRSA with the help of a USAID- funded contractor (a Certified Public Accounting firm, in this case), which is beyond the scope of this evaluation. However, these steps may be construed as capacity building not through but in support of KPRP. 126 “Capacity Building Budget: Implementation Letter Number MLK-05: Malakand Reconstruction and Recovery Program,” PaRRSA, Peshawar, January 2011. 127 “Capacity Building Budget (Revised) under USAID Grant – KP Reconstruction Program – For the period August 2011 to December 2014,” PaRRSA, Peshawar, January 2012.

the amount of $337.2 million.128 Senior PaRRSA officials informed the evaluation team that funds provided for KPRP, the USAID-assisted housing project and a government project called the GOKP Chief Minister’s Package, amounting to $150.3 million (45 percent of the total), actually flowed through PaRRSA, and that, for other donors, PaRRSA only provided facilitation in the form of No Objection Certificates129 and the approvals required to undertake projects in Malakand.130

The PIL also includes a statement that PaRRSA will create gender balance in its staffing by encouraging women to apply for open positions. In addition, it notes that PaRRSA will ensure that the consultants (contractual employees) it is hiring with KPRP assistance will train government civil servants, who will replace the consultants by December 31, 2014. This is referred to in the PIL as an exit strategy. In this connection, PaRRSA was required under the PIL to provide a plan to USAID/Pakistan by July 2012 showing how it would annually reduce the number of paid consultants. In 2012, according to the proposal it submitted to USAID, PaRRSA had planned to hire 124 contractual employees131 with KPRP assistance (76 percent of the total of 163 on its payroll), covering all its units and operations.

The information summarized above indicates that KPRP assistance for PaRRSA’s capacity building was aimed initially (in December 2009) at a relatively limited concern with project implementation. In January 2011, USAID/ Pakistan and PaRRSA agreed, inter alia, to strengthen PaRRSA’s M&E function “to adequately carry out” M&E activities. By 2012, USAID/ Pakistan and PaRRSA had agreed to finance a substantial part of PaRRSA operations through KPRP, both for KPRP and other donor and government-funded programs, promote gender balance in PaRRSA’s staffing, and replace contractual employees with government civil servants by December 2014. Achievements related to these expectations are reported below.

PaRRSA’s 2012 Annual Report (p. 5)132 presents a positive view of its overall capacity, noting that its credibility and efficiency in formulating, planning and executing reconstruction and rehabilitation in Malakand has attracted donor agencies to work in Malakand through PaRRSA. As indicated above, PaRRSA was able to leverage $3.6 million USAID/Pakistan assistance to manage $150.3 million in project assistance directly and facilitate donors that provided an additional $186.9 million for Malakand.

At the same time, the utilization of capacity building assistance has been much lower than planned: the eleventh Quarterly Progress Report (October-December 2013, Annex 2) reports that PaRRSA had utilized $1.3 million (37 percent, on a pro rata basis) of the capacity building budget, an average of $0.54 million per year since August 2011, which is only 51 percent of the annual average amount (of $1.06 million) it had agreed with USAID/Pakistan.133 PaRRSA’s Finance Wing explained to the evaluation team that not all the budgeted staff had been hired, salaries were projected to be higher than they turned out to be in practice, extensive field work had been anticipated but very little of it took place, amounts for vehicles, transportation and fuel had been over-budgeted, and budgeted activities such as workshops and seminars were not undertaken.134 Senior PaRRSA officials confirmed, however,

128 The PIL identifies China, Germany, Italy, Japan, the Kingdom of Saudi Arabia, the Republic of Korea, the United Arab Emirates and “others.” 129 The No Objection Certificate, obtained through an inter-departmental process within the government, is the permission required by foreign entities and their staff to work in Malakand. 130 “Information Provided by PaRRSA and USAID/Pakistan to the Evaluation Team At a Meeting Held at PaRRSA, Peshawar, on June 5, 2014,” MEP evaluation team note. 131 The sixth Quarterly Progress Report (Annex 3) reports that 43 of the 124 positions are for “officers” and 81 for “staff.” Eight of the officer positions are in Swat, one each in Buner and Lower Dir, and 33 in the head office in Peshawar. 132 “PaRRSA at a Glance,” PaRRSA, Peshawar, n.d. 133 PaRRSA’s January 2012 proposal (p. 18) notes that UNDP support for capacity building amounted to $2.17 million over a three-year period. (The amount mentioned in the document is Rs. 195 million and the exchange rate mentioned in it is Rs. 90 per dollar. The same exchange rate is used here for converting Rs. 195 million to $2.17 million.) This averages $0.72 million per year, compared with the $1.06 million per year sought from USAID/ Pakistan. The latter figure is 47 percent greater than the former, which suggests that the amount PaRRSA requested from USAID/Pakistan significantly exceeded the amount it needed to finance its then-existing structure with UNDP assistance, even though, as the proposal notes, USAID/ Pakistan assistance was for hiring 125 individuals, whereas UNDP assistance paid for 139 individuals. 134 This information was provided by PaRRSA’s Finance Wing and noted by the evaluation team on May 21, 2014.

that USAID/Pakistan (through KPRP) has been paying for all of PaRRSA’s operational expenses, except for the salaries of government civil servants working in PaRRSA.135

In relation to staffing, which accounts for 73 percent of the capacity building budget, data provided in Annex 3 of the QPRs for July-September 2012 to October-December 2013 show that 103 to 106 of the 124 positions have been filled during this period, three of them by women (including one officer out of 36 officers’ positions reported filled). Moreover, PaRRSA has not yet prepared any plan for replacing contractual employees with government civil servants and there is no indication in the QPRs that the number of contractual employees is being reduced annually. The evaluation team did not find any documentary evidence of the reasons for this lack of capacity building or steps taken to address it.

At a meeting attended by USAID/Pakistan, five senior PaRRSA officials involved in managing KPRP and the evaluation team, PaRRSA officials explained that there was lack of information within PaRRSA about the contents of the capacity building proposal it had submitted to and agreed with USAID/Pakistan and as a result PaRRSA did not take timely decisions for implementing the agreed capacity building plan.136 Moreover, they and a USAID/Pakistan program manager explained the rationale behind the idea of the exit strategy and why such a strategy was not developed. According to these sources, PaRRSA and USAID/Pakistan thought in early-2012 that KPRP would progressively come to an end by 2014, so that PaRRSA would be able to gradually reduce the number of contractual staff. With delays, as well as additional activities being programmed, PaRRSA and USAID/Pakistan realized that the program would have to continue until 2018. Consequently, PaRRSA did not prepare an exit strategy and the evaluation team did not find any evidence of USAID/Pakistan’s response to this omission. Now that KPRP has completed a large number of activities, not all the contractual staff is needed. PaRRSA has already reduced the number of contractual staff and expects to reduce the number further as KPRP activities are completed137.

At the same meeting, PaRRSA officials also explained why PaRRSA was unable to hire more women and make progress in terms of achieving a better gender balance in staffing. The officials explained that they had received very few applications from women. They provided the lists of candidates for three positions that showed that there was only one female candidate among a total of 22. They also provided copies of two job advertisements from 2012 that included the statement that “female candidates with equal qualifications will be given preference.”

PaRRSA’s M&E function has been visible in two ways so far: it has submitted 11 Quarterly Progress Reports to USAID/Pakistan between April 2011 and December 2013 and produced four field-based M&E reports during January-May 2013.138 There are no progress reports for 2010 and the first quarter of 2011, even though (as documented in the answer to evaluation question 1) important developments such as needs assessment, verification of damaged and destroyed schools and health facilities, school building design and re-design, sub- project approval and tendering, capacity building initiatives, construction activities and disbursement for construction were taking place during this period. Of the 11 progress reports submitted to USAID/Pakistan since June 2011, three remain in draft form and present data that is either internally inconsistent or at variance with another report. At least one provides incorrect financial data on PaRRSA’s capacity building, reporting zero expenditure during a quarter, and the cover page of the second progress report refers to it as the first progress report.

135 “Information Provided by PaRRSA and USAID/Pakistan to the Evaluation Team at a Meeting Held at PaRRSA, Peshawar, on June 5, 2014,” MEP evaluation team note. 136 The evidence cited in this paragraph is taken from “Information Provided by PaRRSA and USAID/Pakistan to the Evaluation Team At a Meeting Held at PaRRSA, Peshawar, on June 5, 2014,” MEP evaluation team note. 137 In a meeting on September 9, 2014, USAID/Pakistan informed MEP that PaRRSA had reduced its contractual staff by 30 as of March 2014 and expected to reduce it further by approximately 20 by the end of September 2014. 138 PaRRSA has also produced annual reports for 2012 and 2013, called “PaRRSA at a Glance,” both of which contain chapters on KPRP. The KPRP chapter in the 2012 report consists largely of background information on post-conflict damage and needs assessment and has no M&E content. In the 2013 report, the KPRP chapter includes a narrative of approximately half a page, evidently reproduced from the eleventh Quarterly Progress Report, followed by 12 pages of photographs.

Of the four field-based M&E reports,139 one focused on the distribution of KPRP-funded health equipment and supplies in 18 Basic Health Units in Lower Dir and Swat and three on 48 completed and ongoing school reconstruction sub-projects in Buner, Lower Dir, and Swat, 32 of which had been completed and handed over to the Department of Elementary and Secondary Education. All four reports provided information on deficiencies in program implementation, including problems with the distribution of health equipment and supplies, the quality of school construction, inadequate supply of furniture (one school140), overcrowding (one school), lack of functioning water supply connections (19 schools), prevalence of unsanitary conditions (two schools), absence of electricity connections (three schools), electrical wires left dangerously open (two schools) and issues of purdah141 for women (two schools). The evaluation team did not find any evidence of these findings being reflected in the 11 PaRRSA progress reports, discussed in meetings of the Project Steering Committee and decisions being taken to rectify deficiencies. Indeed, three AGES monthly reports (May, June and August 2013) point out that observations made by the PaRRSA M&E team had not been addressed.

Line Departments’ Capacity Building

The evaluation team reviewed all the information available to it on the replication of school building design by the C&W Department using GOKP government funds. This information is recapitulated in detail in Annex VII. The team did not find any evidence that GOKP line departments had replicated any PaRRSA-KPRP school design with government funding by December 2014, the cut-off date for the evaluation. There are indications in the evidence (noted in Annex VII), however, that a design introduced by KPRP has been adopted in five schools funded by donors and international non-government organizations. Independent verification of this information by the evaluation team is beyond the scope of the evaluation and was not attempted.

The team confirmed, however, that on April 17, 2014, the GOKP approved a project for the construction of 100 primary schools in the province based on a PaRRSA-KPRP school design, albeit, with a smaller administrative block, during 2014-2016.142 As noted in Annex II of the approved PC-I,143 approximately 50 percent of the primary schools in the province are functioning with two classrooms each, and a GOKP core-working group on education sector reform recommended that primary schools should consist of six classrooms each, including one for the pre- primary class. This led the GOKP to increase the number of classrooms from two to six in this 100-school project. Annex II of the PC-I also notes that project cost would increase more than three-fold, but substantial savings were expected in the current year budget that allowed the revised project proposal to go forward. Annex VII provides a comparison of the design and cost of the GOKP’s six-classroom schools with the relevant PaRRSA-KPRP design and cost.

Other than the recently-approved 100-school project, the evaluation team found no indication in its interviews with at least 16 government officials and M&E firms’ representatives that GOKP had officially accepted and notified improved designs and construction practices introduced by KPRP, reflected them in relevant technical documents and standards or implemented them in identifiable projects and districts of KP. Contractors and subcontractors in both focus group discussions conducted by the evaluation team and representatives of the C&W Department’s Project Unit in Swat gave several examples of learning about quality issues and quality control.144 These are

139 The four reports are: “Follow up Visit to District Buner, Swat and Dir (lower) Against USAID Health Equipment Distributed,” January 2013, “Monitoring and Evaluation Report (27 Completed Schools in Swat),” March 2013, “Monitoring and Evaluation Report (08 Completed Schools in Dir Lower),” April 2013, and “Monitoring & Evaluation Report - Schools Reconstructed in District Buner under USAID Supported KPRP,” May 2013. 140 The number of schools mentioned in this sentence is the number from among schools completed and handed over. 141 Purdah (literally “curtain” in Urdu) refers to women covering their bodies as well as women’s physical segregation in public places with the use of curtains, screens and walls. In the KPRP context, low boundary walls and the absence of a suitable cover on the grills of parapet walls have, in some cases, raised concerns about the lack of purdah they provide to female teachers and students. 142 This decision was taken by the GOKP Provincial Development Working Party, which is the highest project-approval body for government projects in the province, on April 17, 2014. 143 PC-I entitled “130334 – Establishment of 100 Primary Schools (B&G) on need basis in Khyber Pakhtunkhwa (Phase-III) ADP No. 12, 2013- 14 to 2015-16,” Elementary and Secondary Education Department, GOKP, February 2014. 144 These examples include the KPRP requirement that contractors purchase material from approved vendors, use certain types of paints, steel and mosaic roof treatment, participate in quality testing, and follow the guidelines of the KPRP monitoring and evaluation contractor.

instances of individual learning, but they cannot be construed as capacity building as identified by this evaluation question.

The cost and design data reported above indicate that KPRP-funded schools cost much more than schools constructed with the government’s own funds. The difference depends on the number of classrooms as well as the design and construction practices. The evaluation team did not find any evidence of a systematic analysis of the benefits and costs of improved design and construction practices and a dialogue with the provincial government on the efficacy of replicating good practices in any of the interviews it conducted and the documents it reviewed. Indeed, the impetus for adopting the PaRRSA-KPRP six-classroom design came from an ad hoc government- working group that had no link with KPRP in terms of its membership.

RECOMMENDATIONS

The conclusions detailed in the preceding sections lead to two main recommendations. The first of these recognizes the pivotal role of PaRRSA in the G2G modality for KPRP and the need for strengthening its key functions to improve KPRP’s performance and the achievement of its outputs and outcomes. The second recommendation aims to address problems associated with the modality for the school reconstruction component, which contributes directly to the important KPRP outcome “increased access to schools.”

The first main recommendation is for USAID/Pakistan to work with KPRP management to regularly prepare and review annual work plans with the approval of the Project Steering Committee that include all KPRP-assisted construction and non-construction activities.

Plans for non-construction activities should address the following three considerations:

a) a revised capacity building plan with annual milestones, reflecting the funds available, the projected KPRP workload and staffing requirements, and measures to improve gender balance in staffing;

b) plans for regular field-based M&E and improved and regular progress reporting that incorporates findings and recommendations from field-based M&E; and

c) a systematic approach for identifying lack of water supply and electricity connections and over-crowding in reconstructed schools and teacher, parent and student perceptions regarding purdah, and for addressing these limitations with funds from GOKP (which already has a program for adding classrooms and missing facilities145), USAID/Pakistan, other donors and possibly the communities. The second main recommendation is for USAID/Pakistan to continue to engage actively with PaRRSA to improve the modality for implementing the school reconstruction component managed during the evaluation period by the Communication and Works Department.

Measures that be helpful to this process include:

a) the use of market rates rather than standard government rates for costing and tendering sub-projects; b) the executing agency hiring its own supervisory consultants, retaining a project director for its project unit at all times and, when the position is vacant, delegating the powers of this officer to another officer, so that work does not suffer; making timely payments to contractors, and selecting contractors, including reputable large contractors from within and outside the province who have the technical and financial capacity required for doing the work; and

c) facilitating timely payments to the contractors and overcome delays in this regard. One suggestion toward this end is for KPRP establish a grievance redress mechanism in Malakand with the help of the divisional or district administration to resolve payment problems between the executing agency and its contractors as they arise.

145 According to a recent newspaper report, GOKP is already supporting a program in which “schools [are] selected for provision of facilities such as construction of additional classrooms, boundary wall, computer labs, washrooms, water supply schemes, etc.” and “philanthropists, well-to-do families and donor agencies take responsibility for providing missing facilities in government schools of their choice” (“Tameer-i- School [School Construction] Program Extended to Entire KP,” by Mohammad Ashfaq, daily Dawn, July 12, 2014).

BIBLIOGRAPHY

GOKP Documents and Data (excluding PaRRSA): Education Management Information System (EMIS), Annual Statistical Report of Government Schools, Elementary and Secondary Education Department, Government of Khyber Pakhtunkhwa, April 2013. Available at http://www.kpese.gov.pk. PC-I entitled “130334 – Establishment of 100 Primary Schools (B&G) on need basis in Khyber Pakhtunkhwa (Phase-III) ADP No. 12, 2013-14 to 2015-16,” Elementary and Secondary Education Department, GOKP, February 2014. PC-I for the construction of the Khwazakhela and Mingora Bridges, Pakhtunkhwa Highways Authority, GOKP, Peshawar, June 2013. PC-I for the restoration of the Amandara Headworks, Flood Damages Restoration Directorate, Irrigation Department, GOKP, Peshawar, November 30, 2012. PC-I for the restoration of the Munda Headworks, Flood Damages Restoration Directorate, Irrigation Department, GOKP, Peshawar, December 30, 2010. School-level EMIS data provided to MEP by the Elementary and Secondary Education Department. PaRRSA Documents: “Capacity Building Budget: Implementation Letter Number MLK-05: Malakand Reconstruction and Recovery Program,” PaRRSA, Peshawar, January 2011. “Follow up Visit to District Buner, Swat and Dir (lower) Against USAID Health Equipment Distributed,” January 2013. KPRP Quarterly Progress Reports, 2011 – 2013 (11 reports from April-June 2011 to October-December 2013). Minutes of six meetings of the KPRP Project Steering Committee, 2010-2013. “Monitoring & Evaluation Report - Schools Reconstructed in District Buner under USAID Supported KPRP,” May 2013. “Monitoring and Evaluation Report (08 Completed Schools in Dir Lower),” April 2013. “Monitoring and Evaluation Report (27 Completed Schools in Swat),” March 2013. “PaRRSA at a Glance,” PaRRSA, Peshawar, n.d. (2012 and 2013 annual report of PaRRSA). “PaRRSA Capacity Building Budget (Revised) under USAID Grant KPRP for the period August 2011 to December 2014,” January 2012. Media Reports: Associated Press of Pakistan, “Govt [sic] schools in Malakand to reopen tomorrow,” posted by AAJ News Archive (of AAJ Television), uploaded July 31, 2009 (http://www.aaj.tv/2009/07/govt-schools-in-malakand-to-reopen- tomorrow/). Gloria Caleb, “A slow return to normalcy,” Op-ed in Dawn (daily newspaper), August 20, 2009 (http://www.paktalibanisation.com/?p=1616). M. Ibrahim, “Swat school enrolment increases after Taliban’s downfall,” Central Asia Online, November 9, 2011 (http://centralasiaonline.com/en_GB/articles/caii/features/pakistan/main/2011/11/09/feature-01).

Mohammad Ashfaq, “Tameer-i-School [School Construction] Program Extended to Entire KP,” daily Dawn, July 12, 2014. Shamim Shahid, “Attendance thin as schools reopen in Pakistan’s restive region,” Korea Herald, August 2, 2009 (http://www.koreaherald.com/common_prog/newsprint.php?ud=20090803000002&dt=2). Dawn, August 28, 2010, article titled “Munda Headworks restored in record time.” Tahir Ali, “Swatis love their schools,” January 22, 2013, (http://tahirkatlang.wordpress.com/2013/01/22/education- in-swat-after-militancy-and-malala/). MEP Minutes of Meetings: Information Provided by PaRRSA and USAID/Pakistan to the Evaluation Team At a Meeting Held at PaRRSA, Peshawar, on June 5, 2014, MEP evaluation team note. Meeting between MEP and USAID/Pakistan to clarify the Evaluation Statement of Work, February 19, 2014, USAID/Pakistan Documents: Activity Approval Agreement (AAD), dated December 17, 2009. Amendment Number 1, “Modified Fixed Amount Reimbursement Agreement for Reconstruction of Four- and Six- Room Module Schools in the Malakand Division,” FAR Agreement Number 391-MLK-FARA-002-01, dated December 15, 2011. Fixed Amount Reimbursement Agreement (FARA-I): Number 391-MLK-FARA-001-00, dated April 9, 2010. Implementation Letter Number MLK-03, dated May 11, 2010. Mission Strategic Framework approved on July 30, 2013. Project Implementation Letter Number 18, dated May 8, 2012. Project Implementation Letter Number 391-014-PIL-27, dated August 26, 2013. Documents from other sources: AGES Monthly Progress Reports, 2010-2013. Asian Development Bank and the World Bank, Pakistan Floods 2010 Preliminary Damage and Needs Assessment, prepared by Asian Development Bank and World Bank for the Government of Pakistan, Islamabad, November 2010. Asian Development Bank and the World Bank, Preliminary Damage and Needs Assessment (DNA) Report, prepared by Asian Development Bank and World Bank for the Government of Pakistan, November 2009. Available at http://www.pdma.gov.pk/PaRRSA/documents/DNA.pdf. World Bank (2010). Emergency Project Paper on a Proposed Credit to the Islamic Republic of Pakistan for a Khyber Pakhtunkhwa and FATA Emergency Recovery Project. Islamabad: World Bank.

ANNEXES

ANNEX I: EVALUATION STATEMENT OF WORK

KHYBER PAKHTUNKHWA RECONSTRUCTION PROGRAM (KPRP) MID-TERM PERFORMANCE EVALUATION ANNEX 1: STATEMENT OF WORK

DECEMBER 18, 2013

I. BACKGROUND INFORMATION

A. IDENTIFYING INFORMATION ABOUT THE PROJECT

This Statement of Work (SOW) outlines the requirements for the Monitoring and Evaluation Program (MEP) to conduct a mid-term performance evaluation of the United States Agency for International Development (USAID)’s Khyber Pakhtunkhwa Reconstruction Program (KPRP), previously called the Malakand Reconstruction and Recovery Program (MRRP). KPRP supports efforts by the Government of Pakistan (GOP) to rebuild public infrastructure destroyed by conflict (during 2007-2009)146 and the 2010 floods in the Malakand Division of Khyber Pakhtunkhwa (KP) Province. The project focuses on rebuilding one major and one smaller bridge, two irrigation headworks and facilities for education, health and drinking water, and building the capacity of the government departments that facilitate and coordinate rehabilitation147 and reconstruction148 in these areas.149 Table 1 summarizes key facts about the KPRP.

TABLE 1: PROJECT SUMMARY

Project Name/ Title Khyber Pakhtunkhwa Reconstruction Program (KPRP) 391-011-IL#MLK-02 391-MLK-FARA-001-00 Agreement Number 391-MLK-FARA-002-00 391-011-IL#MLK-05 AID-OAA-TO-10-00066 Project Manager Bahro Bar (Mr.), Project Manager Project Start Date March 26, 2010 Project Completion Date September 30, 2016 Five districts of Malakand Division, namely Swat, Upper Dir, Lower Dir, Buner Project Location and Shangla. Development Objective 3: Increased stability in targeted locations. The most USAID Objective directly relevant Intermediate Result (IR) of DO3 is IR 3.3: Essential services that Addressed respond to priority community grievances delivered. Name of the Provincial Reconstruction, Rehabilitation and Settlement Authority (PaRRSA), Implementing Partner (IP) Government of Khyber Pakhtunkhwa (GOKP) Communication & Works (C&W) Department, GOKP Flood Damage Names of the Executing Restoration Directorate, Irrigation Department, GOKP Pakhtunkhwa Highways Agencies Authority (PKHA), GOKP Public Health Engineering Department, GOKP Budget US$149.9 Million

The original Activity Approval Document (AAD) for KPRP was approved on December 16, 2009 for US$36 million and included assistance for reconstruction and rehabilitation of schools, basic health units, and drinking water supply systems. It also included provision of equipment and supplies for health facilities and enhancing

146 The Taliban entered the region in 2007 and controlled large parts of it until 2009, when they were evicted after a forceful counter- insurgency operation by the Pakistan Army. Public facilities were destroyed by the Taliban during the insurgency and also damaged by military action during the counter-insurgency. 147 Rehabilitation means restoration to a previous condition; repair is similar and generally indicates a lesser degree of work than rehabilitation, e.g., painting, replacing broken glass, etc. 148 Reconstruction means building a new similar structure at the site of a previous structure that has been demolished for whatever reason. 149 Project components and activities are described later in this chapter.

government capacity through provision of operational support such as payment for utilities, transportation and communication.

The AAD has been amended three times. The first amendment, made on March 2, 2011, added a budget for monitoring and evaluation of KPRP components that increased total activity funding to US$43 million. The second amendment (May 19, 2011) increased total activity funding to US$148.3 million to finance reconstruction and rehabilitation of flood damaged infrastructure and provide governance support for construction of an integrated tehsil complex at Kabal, Swat. This amendment also extended the project closing date from December 12, 2012 to September 30, 2016. The third amendment (September 25, 2011) increased total activity funding to US$149.9 million and added assistance for school supplies such as schools bags and furniture and a public media campaign.

The project components and interventions mentioned above are introduced in greater detail in Section I.D. B. DEVELOPMENT CONTEXT

1. Problem or Opportunity Addressed

Malakand Division consists of the districts of Swat, Upper Dir, Lower Dir, Buner, Shangla, Chitral and Malakand. Conflict in the area during 2007-2009 resulted in loss of human lives and large scale population displacement as well as destruction of infrastructure including health and education facilities, and water supply systems.150 The Taliban occupation of a large part of this area and the resulting damage and challenge posed to the writ of the state led to a counter insurgency operation from May to July 2009.

The infrastructure damaged during the conflict in the five districts covered by the project – Swat, Upper Dir, Lower Dir, Buner and Shangla – included the following:151

 237 schools, including 149 girls’ schools, were destroyed152 and another 190 (including 65 girls’ schools) were damaged.153 The estimated cost of reconstruction and rehabilitation for the destroyed and damaged educational facilities, materials, furniture and equipment was Rs. 2,696 million (US$33.7 million).154 Swat was the most seriously affected district with a reported 276 destroyed or damaged schools (including 167 girls’ schools);  19 health facilities were destroyed and 44 were damaged. The total estimated cost to the health sector in Malakand was approximately Rs. 502 million (US$6.3 million). This included offices, staff residences, medical equipment, furniture and vehicles;  About 30 percent (451 out of 1,508) of the water supply systems in the five districts were destroyed or damaged. Out of the 451 affected systems, 111 were destroyed and 340 were damaged. The estimated cost was Rs. 62 million (US$0.8 million); and,  40 bridges were destroyed and 18 were damaged. The total estimated cost was approximately Rs. 628 million (US$7.8 million). Floods in 2010 also damaged schools, roads, bridges, and irrigation headworks.

150 Preliminary Damage and Needs Assessment (DNA),” November 2009, prepared by Asian Development Bank and World Bank for Government of Pakistan Islamabad, Pakistan available at http://www.pdma.gov.pk/PaRRSA/documents/DNA.pdf”. 151 This information is taken from the “Preliminary Damage and Needs Assessment (DNA),” November 2009, prepared by Asian Development Bank and World Bank for Government of Pakistan Islamabad, Pakistan available at http://www.pdma.gov.pk/PaRRSA/documents/DNA.pdf”. 152 Destroyed means completely damaged. 153 Damaged means partially destroyed. 154 Following the Preliminary Damage and Needs Assessment (DNA) an exchange rate of US$1 = Rs. 80 is used in this section. The amounts reported in million US$ have been rounded off to the first decimal place.

2. Focus Areas and Groups

The focus area of the project is Charsadda155 and the five districts of Malakand Division namely Buner, Lower Dir, Shangla, Swat, and Upper Dir.

Initially, KPRP’s focus group was the population affected by the destruction caused by the conflict, in particular those affected by the loss of essential services (health, education, and drinking water supply systems) in the five districts of Malakand Division mentioned above.

Subsequently, USAID added the reconstruction of two irrigation headworks (Amandara and Munda) and two bridges (at Khwazakhela and Mingora), which were damaged by the massive floods of 2010. There are no specific focus groups for these interventions, but the beneficiaries include farmers, traders, tourists and other irrigation and road users both inside and outside Malakand Division.

The Upper Swat Canal, which takes off from Amandara in Malakand District, irrigates parts of Malakand District and the Mardan and Swabi Districts of KP, which are outside Malakand Division. The Lower Swat Canal begins at Munda, which is located in Charsadda District, and irrigates this district as well as parts of Mardan and Nowshera Districts, which are also outside Malakand Division.

The Khwazakhela Bridge, which is located in the Swat District of Malakand, connects Chitral, Upper Dir and Lower Dir to the rest of the country. The Mingora Bridge, which is also located in Swat, connects Upper Swat and Shangla to the rest of the country. C. INTENDED RESULTS AND THE THEORY OF INTERVENTION

In July 2009, GOP lifted restrictions on the return of internally displaced persons (IDPs) as a first step toward increasing confidence among residents. The government initiated an early recovery process, in particular, to address the needs of IDPs and also reestablish its administrative control in the affected districts.156 It started a reconstruction and rehabilitation program to demonstrate to the people its ability to respond to their needs. USAID approved KPRP to assist GOP initiatives to reconstruct facilities in conflict-affected areas and lay the groundwork for the return to normalcy. USAID expects that KPRP’s interventions for education, health, drinking water supply systems will benefit both male and female segments of the focus population.

KPRP is assisting GOKP to rehabilitate and reconstruct infrastructure for provision of essential services, which were disrupted by infrastructure destruction and damage during the insurgency and counter-insurgency related conflict. As articulated in the results framework of Development Objective 3, USAID expects that the provision of essential services will contribute towards improving state legitimacy, governance, economic opportunities and changing the local populations’ attitudes and behaviors. These intended results are expected to contribute toward the overall stability of this conflict affected area.

Figure 1 below illustrates the KPRP theory of intervention, depicting how project interventions are expected to lead to KPRP’s intended results and the higher-level USAID Development Objective 3.

155 The Munda headworks are located in Charsadda which is outside Malakand Division. 156 KPRP Activity Approval Document, December 2009.

FIGURE 1: KPRP THEORY OF INTERVENTION

Increased stability

Improvements in state legitimacy, governance, economic opportunities and people's attitudes and behavior

Enhanced access to essential services

Key infrastructure facilities reconstructed, rehabilitated and strengthened

According to the KPRP results framework, the project goal is “to accelerate reconstruction and recovery efforts in Malakand Division.” Project objectives, as reported in the January-March 2013 quarterly report are:

 To support the rehabilitation and reconstruction of social services infrastructure including health, education, water and sanitation, governance, irrigation and communication;

 To build the capacity of the Provincial Reconstruction Rehabilitation and Settlement Authority (PaRRSA); and,

 To support the line departments of GOKP. According to the KPRP Results Framework, the project has the following intermediate results (IRs):  IR 1: Increased enrollment at focus public schools;

 IR 2: Improved public access to water and sanitation services in focus communities;

 IR 3: Improved health services at focus health facilities;

 IR 4: Enhanced government capacity to provide public services.

D. APPROACH AND IMPLEMENTATION

KPRP is undertaking reconstruction and rehabilitation of destroyed and damaged infrastructure with a “build back better” approach, which is elaborated in the description of project components below:

1. Reconstruction157 and rehabilitation158 of destroyed and damaged government schools: KPRP is supporting the rehabilitation and reconstruction of 116 destroyed and damaged schools. These schools are being reconstructed and rehabilitated according to the new improved designs (including boundary walls and toilets), higher earthquake resistance, and adherence to high-quality construction standards. KPRP is also providing bags to students attending these schools and furniture such as tables and chairs for teachers, and student desks once the schools are completed. 2. Reconstruction of Basic Health Units (BHUs) and provision of supplies to health facilities: KPRP is providing funding to support maternal and child health interventions including:

157 Reconstruction means building a new similar structure at the site of a previous structure that has been demolished for whatever reason. 158 Rehabilitation means restoration to a previous condition; repair is similar and generally indicates a lesser degree of work than rehabilitation, e.g., painting, replacing broken glass, etc. So far KPRP has focused only on reconstruction and rehabilitation may be taken up in future.

 Reconstruction and rehabilitation of destroyed and damaged BHUs and staff residences. These BHUs are being reconstructed and rehabilitated according to a new improved design, higher earthquake resistance, and adherence to high-quality construction standards.  Provision of furniture, medical equipment and supplies to improve the maternal and neonatal health status of the community. The original target for this purpose was 17 BHUs, out of which one was destroyed and had no space for the equipment and furniture, so the target was reduced to 16. During implementation, government health officials realized that available furniture, equipment and supplies exceeded the requirements and capacity of the focus BHUs and decided to provide the surplus to 28 additional health facilities. 3. Reconstruction of Drinking Water Supply Systems (DWSSs): KPRP is funding the reconstruction of destroyed water supply systems (tube well and hand pumps) in Buner and Swat. In addition to the installation of tube wells and distribution networks for water supply, these systems include installation of better electricity management equipment such as voltage regulators, stabilizers and transformers. 4. Rehabilitation of Munda and Amandara Headworks: KPRP is providing support to GOKP to rehabilitate the Munda and Amandara Headworks, which were damaged during the 2010 floods. The rehabilitation activities include improvements in the design and structure of the headworks, which will increase their discharge. 5. Reconstruction of Khwazakhela and Mingora Bridges: KPRP is providing support for the reconstruction of the Khwazakhela and Mingora bridges in Swat, which were damaged during the 2010 floods. The reconstruction of these bridges includes a new improved design, which can better withstand earthquakes and allows greater water flow during floods. 6. Construction159 of Integrated Tehsil Complex: The second AAD amendment added a component for governance support to enable construction of an integrated tehsil complex at Kabal, Swat. 7. Capacity building of GOKP: KPRP is providing funding for operational support to enable PaRRSA to recruit, retain, and deploy additional staff as needed by provincial authorities to carry out project activities. It is also providing financial assistance to support the operational costs of the project, including cost of utilities, transportation and communications. The project does not provide any direct assistance for capacity building of line departments, but USAID expects the line departments to enhance their capacity by adopting good practices introduced by KPRP.

According to the progress report for July-September 2013, education is the largest component of the project in terms of the amounts obligated (Figure 2).

159 Construction includes building a new structure that is distinctly different from any previous structure that may have existed at the site.

FIGURE 2: KPRP BUDGET ALLOCATION

Munda HW Khawazakhela Bridge 10% 6% Amandara HW 7%

Water & Sanitation 4%

Tehsil Complex 3%

CB Budget 4% Health Facilities Schools 7% 59%

According to the agreement between USAID and PaRRSA, dated December 17, 2009, PaRRSA has the primary responsibility for coordination and dialogue with USAID on all project activities. PaRRSA has also established a Project Management Unit (PMU)160 in Swat, which is responsible for monitoring and coordination of all reconstruction and rehabilitation activities by KPRP and other donors.

The project is executed through GOKP agencies including the Communication & Works (C&W) Department, the Flood Damage Restoration Directorate (FDRD) of the Irrigation Department, the Public Health Engineering Department (PHED) and the Pakhtunkhwa Highways Authority (PKHA).

The C&W Department is responsible for preparing the basic project document required of government agencies, namely, the Planning Commission Pro Forma No. 1 (PC-1), for reconstruction and rehabilitation of schools and health facilities. For other components of KPRP, such as drinking water supply, headworks, bridges and the integrated tehsil complex, the relevant departments mentioned above are responsible for preparing the PC-Is.

In response to implementation issues, including delays, GOKP established a dedicated Project Unit within the C&W Department in Swat in June 2012. The purpose of this Project Unit is to expedite planning, approval and construction. In pursuit of this, GOKP has delegated certain powers, including powers for technical and financial approval of PC-Is, opening of tenders, and administrative approvals, to the Project Unit’s Project Director.

Construction work is undertaken by local contractors. This reflects an agreement between USAID and GOKP to use local contractors in order to benefit the local economy.

PaRRSA and USAID have jointly prepared the performance management plan, including the performance indicators. They have also jointly developed monitoring and evaluation plans to monitor project progress.

USAID has contracted a full time engineering and monitoring firm called Al-Kasib Group of Engineering Services (AGES) to provide third party monitoring support for all infrastructure activities except Amandara headworks, which is monitored by another engineering and monitoring firm, called Associates in Development (AiD). These firms have been engaged to fully supervise local contractors’ construction161 activities. They provide monthly monitoring reports to USAID, which shares them with PaRRSA, which undertakes any course corrections that may be required.

160 The USAID project called the Assessment and Strengthening Program (ASP) has contributed to this PMU’s capacity by providing training to its administrative staff. 161 AGES and AiD supervisors are present at the construction sites during construction activities.

Implementation and monitoring arrangements for all components are summarized in Table 2 and illustrated in Figure 3. E. CURRENT STATUS OF ACTIVITIES

According to the KPRP quarterly progress report for July-September 2013, the following is the current status of the project’s activities:  Forty-five schools serving approximately 8,000 students have been completed, equipped, and handed over to the GOKP Department of Education. The list of all schools and their construction status is attached as Appendix 3. According to the report, 24 more schools are near completion, with physical progress ranging from 90 to 99 percent (and the main building 100 percent) completed.

 Construction work on the five destroyed and 13 damaged BHUs has been started and is at various stages of reconstruction and rehabilitation. The list of the 18 focus BHUs that are being reconstructed and rehabilitated and their completion status is attached as Appendix 4.

TABLE 2: IMPLEMENTATION AND MONITORING ARRANGEMENTS BY COMPONENTS

Monitoring Component Executing Agency Host Department162 Firms Reconstruction and rehabilitation C&W Department Education Department AGES of Schools Reconstruction and rehabilitation C&W Department Health Department AGES of BHUs Reconstruction of Drinking Water Public Health Engineering Public Health AGES Supply Systems Department/Municipal Engineering Committee Department/Municipal Committee Rehabilitation of Munda Flood Damage Provincial Irrigation AGES Headworks Restoration Directorate Department (FDRD) Rehabilitation of Amandara Flood Damages Provincial Irrigation Associates in Headworks Restoration Directorate Department Development (FDRD) (AiD) Reconstruction of Khwazakhela Pakhtunkhwa Highways Pakhtunkhwa Highways AGES and Mingora Bridges Authority (PKHA) Authority (PKHA) Construction of Integrated Tehsil Details not available Details not available AGES Complex

The list of 44 health facilities strengthened with furniture, medical equipment and supplies is attached as Appendix 5.163 These facilities include 16 of the focus BHUs as well as 28 other health facilities that were provided equipment and supplies that exceeded the requirements and capacity of the focus BHUs. The list of equipment and medical supplies given to these health facilities is attached as Appendix 6.  Survey and assessment of 13 water supply systems in Buner has been completed and reconstruction started. The list of these under-construction systems is attached as Appendix 7. The project implementation letter for reconstruction of 10 drinking water supply systems in district Swat has been signed and the pre-qualification of contractors is in process.  The progress on reconstruction of Amandara headworks and Munda headworks is 95 and 90 percent, respectively.

162 The “host department” is the department designated to take over completed facilities and provide services. 163 A document provided by KPRP reports that 47 health facilities have been provided furniture and medical equipment, but the list actually adds up to 44.

 The progress on reconstruction of Khwazakhela and Mingora Bridges is 70 and 55 percent, respectively.  A master plan for the Integrated Tehsil Complex in Kabal has been finalized. Drawings and designs have been completed and are currently being reviewed by USAID. However, the Project Implementation Letter (PIL) for this activity has not been signed and, therefore, the executing agency for this project is yet to be identified.

FIGURE 3: ILLUSTRATION OF KPRP GOVERNMENT-TO-GOVERNMENT MODALITY

USAID Assistance Agreement / PaRRSA PIL / $ Deliverables

$ $ Contracts Contracts

Information Information Deliverables

A&E Firms C&W PHED PKHA FDRD (AGES/AiD) for for for for

Education DWSS Bridges Headworks

Supervised Designs, Construction & Procurement & Monitoring Management Submittal

and Deliverables

Local Contractors and Sub-Contractors for Construction Activities

Drawing on the KPRP quarterly report for July-September 2013, Tables 3-6 report the current status of KPRP activities in greater detail.

TABLE 3: KPRP STATUS OF SCHOOL CONSTRUCTION Status of Construction Total District Under Target Completed Dropped construction Buner 13 7 6 - Lower Dir 20 8 11 1 Upper Dir 3 - 3 - Swat 80 30 48 2 Total 116 45 68 3

TABLE 4: KPRP STATUS OF BHU CONSTRUCTION Extent of Damage Status District Target Under Destroyed Damaged Completed Construction Swat 11 4 7 - 11 Lower Dir 6 - 6 - 6 Buner 1 1 - - 1 Total 18 5 13 - 18

TABLE 5: KPRP STATUS OF DRINKING WATER SUPPLY SYSTEMS CONSTRUCTION Completion Status Target District Completed Under Construction Water Hand Water Hand Water Hand Supply Pumps Supply Pumps Supply Pumps Buner 12 20 - 2 12 18 Swat 8 120 - - 8 120 Total 20 140 - 2 20 138

TABLE 6: KPRP STATUS OF FLOOD DAMAGED INFRASTRUCTURE Completion Sr. No Description of Project Components Status

1 Rehabilitation of flood damaged Amandara Headworks 95% 2 Rehabilitation of flood damaged Munda Headworks 90% 3 Reconstruction of Khwazakhela Bridge 70% 4 Reconstruction of Mingora Bridge 55%

II. RATIONALE FOR EVALUATION

A. EVALUATION PURPOSE

At roughly the half-way point of KPRP implementation, USAID would like to assess the project’s performance in order to determine whether mid-term corrections in implementation are warranted. Evaluation findings may also contribute to designing future projects of a similar nature, particularly government-to-government (G2G) projects. In addition to these broad objectives, USAID would like the evaluation to address the following issues:

For a variety of reasons, implementation has been slower than expected. The February and March 2011 AGES progress reports attribute delays in the beginning of the project to unprecedented floods, severe weather in some parts of the project area, issues in site clearance, delays in the release of funds to the C&W Department, delays in survey work by the supervisory consultant, informal sub-letting of contracts by the selected contractors to subcontractors, contractor capacity to manage multiple contracts, cancellation of some contracts, and the fact that advance payment is not available and payment to contractors is made only upon completion of milestones, which challenges the contractors’ financial capacity.

There have also been delays and coordination issues in project management. Over the course of the project to date, USAID and GOKP have changed administrative processes in an attempt to alleviate implementation bottlenecks. For example, USAID and GOKP agreed to establish a Project Unit in the C&W Department to expedite implementation. To aid in designing future G2G projects, USAID wishes to know how well this structure has been working and whether this structure is effective in moving towards improved implementation including site identification, planning, approval, communication, reporting, fund transfers, technical oversight and, monitoring. More broadly, USAID wants to know whether any lessons are emerging about how best to implement projects directly through the host government.

The direct and indirect capacity building effects of KPRP need to be assessed. For example, the operations of PaRRSA were funded by the United Nations Development Program and this assistance came to an end in 2010. Since then, USAID has been providing financial assistance to PaRRSA to pay for staff, utilities, transportation and communications. USAID does not provide such direct support to GOKP line departments, but anecdotal evidence suggests that line departments have adopted some of the good practices for construction that KPRP has introduced. These instances of enhanced capacity need to be documented in the evaluation.

While hard data are not available, it is conjectured that students who were unable to attend a destroyed school might have attended a tent school, or gone to school in another location (for example, in a neighboring village), or dropped out of school for some time. KPRP’s contribution to increased access and improved schools needs to be assessed in the context of the situation that prevailed after the schools had been destroyed. Comparison with the number of children going to schools that were not destroyed will be useful for contextualizing the contributions of KPRP.

Existing anecdotal evidence suggests that school characteristics in the project areas may limit enrollment, particularly for girls. For example, if schools are too small to accommodate the number of school-aged children in the catchment area, parents may choose not to enroll some or all of their children. Similarly, if schools lack water, toilets, or boundary walls (particularly important for girls’ schools), parents may hesitate to enroll their children. If this is true, then the government may be able to increase enrollment by improving the quality of school facilities rather than merely building more schools with the same characteristics as existing schools. If possible, USAID would like the evaluation to determine whether and to what extent improved facilities influence parents’ willingness to send children to school.

To address these objectives the evaluation will:

a. Assess project performance to date to determine whether mid-course corrections are needed. b. Assess whether and how the changes made to-date in administrative processes have affected project implementation. c. Analyze the extent to which the project has, to date, achieved its results (understood basically, but not strictly, in terms of the KPRP IRs). Analysis of sex-disaggregated data and issues of girls’ enrollment in schools will be included in this part of the assessment. d. Analyze the extent to which the project has, to date, built the capacity of PaRRSA and relevant GOKP line departments. The assessment of PaRRSA will focus on the effects of inputs provided by the project. Capacity building of line departments will be assessed in terms of their adoption of the good practices in construction introduced by the project.

B. AUDIENCE AND INTENDED USE

The results of this evaluation will be shared within USAID and with the implementing partners. USAID will develop a dissemination plan in accordance with the Evaluation Policy as specified in Automated Directives System (ADS) Chapter 203. A debriefing on the final evaluation findings will be provided to USAID staff from relevant offices as well as to staff of various departments of Government of KP, including Communication & Works, Education, Health, Irrigation, and Highways.

C. EVALUATION QUESTIONS

To address the objectives mentioned above, the evaluation will answer the following three key questions.

1. In what ways and to what extent have the administrative processes, site identification, coordination, and monitoring of subcontractors, particularly through the G2G modality, influenced project implementation to date?

Explanation: The answer to this question will assess how well the current G2G modality, which was amended at the beginning of course of implementation, has worked in terms of needs-assessment processes for damaged infrastructure; identification of suitable site selection; proper planning for implementation; approval processes for the sub-projects or schemes; contracting out the sub-projects or schemes; coordination and communication among various stakeholders; reporting mechanisms; monitoring and evaluation mechanisms; process of funds transfers among GOP agencies/departments as well as to contractors/subcontractors; established systems and overall capabilities of GOP agencies/departments to provide proper and adequate technical oversight; USAID relationship with the GOP and all of the other stakeholders; the relationship of GOP agencies/departments and contractors/subcontractors to the outside monitoring and evaluation firm; and the delivery of outputs, among other things.

This question will further examine if there has been, and the subsequent impacts of, any practice of sub-awarding schemes or sub-projects by a recipient contractor to an unapproved subcontractor. The evaluation will examine the efficiency of the current mode of implementation and will also determine if modifications in administrative processes are warranted to facilitate implementation over the next half of the project.

2. In what ways and to what extent have the project’s schools, supplies to basic health units (BHUs), bridges and irrigation components164 contributed to the achievement of intended results to date through completed and almost completed activities?

164 Capacity building is excluded from this question because it is addressed separately in question 2.

Explanation: The evaluation will assess the extent to which completed activities have contributed to the intended results, that is, improved access to schools (which can also be termed as improved schools in view of their build-back-better features), access to improved health services, access to irrigation water supplied through the Amandara and Munda Headworks, and access to surface travel by means of the Khwazakhela Bridge. The evaluation should contextualize the reported results with reference to the number and nature of outputs completed by the project.

For reconstructed schools, the evaluation should consider the number of children going to school and infer the KPRP contribution by comparing the 2011 and 2013 numbers of children going to these and undamaged schools in the respective districts. The project’s contribution should also be analyzed by finding out where the currently- enrolled students went to school (or did not) during the time the destroyed schools were not reconstructed, and how this touched the affected families. The evaluation should also shed light on the extent to which parental choices about sending children to school have been influenced by the build-back-better features (such as boundary walls, toilets and higher number of class rooms) of KPRP-financed schools.

As the project has not yet completed the repair or reconstruction of any health facilities, it cannot be said that it has helped increase access in this sector. Thus, the evaluation will focus on access (in relevant catchment areas) to improved health services in the BHUs that have received furniture, equipment and medical kits.

The evaluation will also document the access to irrigation water and surface transport in terms of the number of beneficiaries, acreage, traffic flows and other available information on access.

The project had no explicit gender objectives or targets. Nevertheless, it may have generated sex-differentiated results. For instance, if the project reconstructs schools with characteristics more suited to girls (e.g., boundary walls) it may increase girls’ enrollment more than boys. In addition, the project implementation approach may engage and affect both men and women in varying ways and possibly mitigate or enhance gender inequities, including decision making and community engagement. This question will identify and explore likely gender-specific effects of project activities.

3. In what ways and to what extent has KPRP built the capacity of PaRRSA and other government line departments to date to provide public services?

Explanation: For PaRRSA, the evaluation will assess the contribution of the assistance provided by KPRP.165 For GOKP line departments, the project does not include any direct capacity building activity. The evaluation will identify and analyze the extent to which improved construction practices introduced by KPRP have been officially accepted and notified by the line departments, as reflected in relevant technical documents and standards, and implemented in identifiable projects and districts of KP. If evidence suggests very limited or lack of capacity being built through KPRP to provide public services, this evaluation question will endeavor to determine why.

165 This assistance consists of operational support to enable PaRRSA to recruit, retain and deploy additional staff as needed by provincial authorities, and financial assistance to support the operational costs of the project, including the cost of utilities, transportation and communications.

III. DESIGN AND METHODOLOGY

A. EVALUATION PROCESS

The evaluation process consists of the following five main stages: Stage 1: Preparation and planning by MEP staff; Stage 2: Initial review of priority documents by evaluation team; working from their home stations, evaluation team members will review all project background documents. Stage 3: Team work in Islamabad; upon deployment in Islamabad, the evaluation team will participate in a team planning meeting facilitated by the MEP Evaluation Manager and undertake specific preparatory tasks as a team. Stage 4: Fieldwork; as soon as possible during the evaluation team’s Islamabad-based activities, MEP will administer the survey, through a contracted service provider, so that data will be available for analysis in time for the evaluation team’s report writing activity. Simultaneously, the evaluation team will begin its fieldwork during which it will also provide MEP with initial findings, conclusions and possibly recommendations. The following table summarizes the data collection methods and fieldwork locations, with the methods described in greater detail later in this section: Stage 5: Data analysis and report writing;

TABLE 7: DATA COLLECTION METHODS AND LOCATIONS Data Collection Method Location Initial review of key documents Consultant’s home station Qualitative Data Collection – Phase I Peshawar and district headquarters of Swat,  Individual interviews Lower Dir and Buner Districts of Malakand  Collect and take photographs showing facilities after they Division were destroyed/damaged and reconstructed/rehabilitated Qualitative Data Collection – Phase II Sites and villages in Swat, Lower Dir and Buner  Focus group discussions Districts of Malakand Division Sample Survey (simultaneously with qualitative data collection) Communities in Swat, Lower Dir, and Buner  Community perceptions survey of a sample of users of Districts located within 3 km of KPRP KPRP built schools and the schools not destroyed in the supported schools, and the nearest schools not conflict (included to enable comparisons). destroyed during the conflict (included to enable comparisons).

B. EXISTING DATA

Documents available on the operations of the project include, but are not limited to:

 Activity Approval Document, December 2009 and Implementation Letter Number MLK-01 dated Dec 17, 2009  Action Memorandum for Mission Director, March 2, 2011 and Implementation Letter No. MLK-08  Activity Approval Document, Amendment Two May 16, 2011 and Action Memorandum for Mission Director, May 19, 2011  Activity Agreement, June 10, 2011 and Action Memorandum for Mission Director, Jun 10, 2011;  PIL No. 391-014-PIL-01 dated March 21, 2012 for Enhancing PaRRSA Capacity and Revised Capacity Building Budget

 Performance Management Plan  USAID Branding and Marking Guidelines  USAID Monitoring Reports  9 Quarterly Progress Reports  21 monthly progress reports by Al-Kasib Group of Engineering Services  Contract Agreement with Al-Kasib Group of Engineering Services for Monitoring and Evaluation Services for KPRP  Contract Agreement with Asian Trading Company for provision of School Supplies and Furniture to the newly constructed schools of Malakand Division  Preliminary Damage and Needs Assessment (DNA), November 2009  Terms of Reference for AGES USAID will help MEP to obtain the following information and documents from PaRRSA and other government organizations well before the start of the field work for evaluation:

 Traffic count (vehicles and passengers) data for the Khwazakhela and Mingora bridges;  Data on cultivable command area (CAA), water discharge, design improvements and benefits for the population served by the Munda and Amandara Headworks;  School profiles of 43 completed schools prepared by PaRRSA;  Minutes of the Project Steering Committee meetings;  Minutes of meeting between the Chief Secretary and Director FATA KP USAID;  School level data on enrollment (if possible from 2005-06 to 2012-13) available in the Education Management Information System (EMIS);  Health facility users (utilization) data (if possible 2006-07 to 2012-13) available from the District Health Information Systems (DHIS);  List of petty contractors and local vendors who were involved in the construction activities of the project;  PaRRSA Annual Reports;  KPRP Workplan;  KPRP data on quarterly expenditures and physical progress of all components; and,  PaRRSA’s agreement with the C&W Department for construction work.

C. DATA COLLECTION METHODS

The evaluation team will apply a mixed-methods approach, using both quantitative and qualitative techniques to collect data from multiple sources to ensure multiple levels of triangulation. Specifically, the evaluation will use quantitative data from a sample survey in Malakand as well as secondary sources (from relevant government departments) located in Malakand and Peshawar. The evaluation team will also conduct focus group discussions with teachers, parents of children attending KPRP-assisted schools, and local contractors (elaborated in Table 8). All groups will be separated by sex except for the one with local contractors who are anticipated to all be male. It will also conduct 31 individual interviews with USAID and government managers, monitoring firms and female health service providers (please refer to Table 9 for detail). The detailed data collection methodology is described below for the three evaluation questions separately

The sample survey, in conjunction with primary qualitative data and secondary quantitative data, is intended to address the education component of evaluation question one. The survey will elicit the responses of the mothers and fathers of school-going children about whether and where the children attended school before and after schools were reconstructed, and whether and to what extent parents’ decisions to send their children to school are influenced by the build-back-better features included in KPRP-assisted schools. This will require comparison between the KPRP reconstructed schools (the intervention group) and schools not damaged during the conflict (the control group).

In view of the requirements mentioned above, a sample of 400 each for the schools reconstructed by KPRP and schools not destroyed during the conflict is proposed. The total sample will be 800 respondents (equal number of mothers and fathers of children attending these two categories of schools). For statistics drawn from the entire sample, the sample size will allow inference within a margin of error of plus or minus 5 percent, assuming a design effect of 1.5, and a confidence level of 95 percent. For comparison between beneficiaries and non-beneficiaries, the level of statistical significance will depend on the final sample size and the magnitude of observed differences between the two groups. Greater differences are more likely to be statistically significant. Establishing the significance of smaller differences will require a larger sample. The present sample size does not allow inter-district comparisons.

The sample will include an equal number of parents of school-going children from each focus school. All respondents will be selected randomly. Thus, the survey design is multistage, with probability disproportionate to the size of the population. Thus, to ensure the representativeness of the sample, it will be weighted according to the current enrollment of each of the focus schools.

For question one, the answers for the remaining components (health, bridge and headworks) will be based on secondary data, individual interviews and focus group discussions with relevant stakeholders. Evaluation questions two and three will be answered through secondary data and relevant individual interviews.

Qualitative data will be collected through individual interviews and focus group discussions. Each of the 10 focus group discussions will have 8-12 participants and will be conducted with the following groups of project beneficiaries:

 Parents of children attending schools reconstructed and rehabilitated with KPRP assistance;  Head teachers of KPRP reconstructed/rehabilitated schools; and,  Local contractors involved in reconstruction and rehabilitation activities.

MEP will identify locations for focus group discussions and convey participants’ requirements to PaRRSA for onward communication to the departments concerned (Health and Education) in advance of the field work, so that participants can be mobilized in time for the evaluation team’s field work. MEP will select local contractors for participation in a focus group discussion from the list of contractors provided by PaRRSA and/or other agencies. Table 8 shows the focus group discussion sampling frame:

TABLE 8: FOCUS GROUP DISCUSSION SAMPLING FRAME Project Number of focus group Component/ Participants District discussions Theme Male Female Total Swat 1 1 2 Parents of children attending KPRP built Lower Dir 1 1 2 schools Education Buner 1 1 2 Head teachers166 from all the three Swat 1 1 2 districts Rehabilitation and Local contractors of schools and bridge Swat167 2 - 2 reconstruction components Total 10

MEP will conduct individual interviews with officials from USAID, PaRRSA, the monitoring firms, the C&W Department, the Health, Education and Irrigation Departments, and the Pakhtunkhwa Highways Authority. The sampling frame for individual interviews is summarized in Table 9.

D. DATA ANALYSIS METHODS

Quantitative Data Analysis

Following the completion of the survey, the data will be tabulated and subsequently analyzed for differences across gender and intervention/control group.168 Statistical analysis of survey data will include a combination of the following:

 descriptive statistics of the characteristics of the overall sample and the two groups in the sample; and,  frequency distributions, cross-tabulations, analysis of variance (ANOVA) and multivariate techniques, as appropriate, to analyze differences across groups in parental preferences and choices.

166 At least three to four head teachers from each district will participate in the focus group discussion. If a head teacher has been appointed recently, indicating lack of information about the school, MEP will request the senior most teacher to participate in the focus group discussion. 167 Focus group discussion with contractors in Swat will include participants from Swat, Lower Dir and Buner. 168 Where there are too few observations for male-female comparison, the level of statistical significance of the difference between men and women will be lower than the standard used in high-quality research.

TABLE 9: INDIVIDUAL INTERVIEWS SAMPLING FRAME

Project Number of Individual Interviews Components/ Individuals Lower Total Organizations Swat Buner Islamabad Peshawar Dir Female health care Health providers from KPRP 3 2 2 - - 7 strengthened BHUs District Education - Education 2 1 1 - 4 Officers Senior officials from Khwazakhela Pakhtunkhwa Highway - - - 4 4 Bridge Authority (PKHA) Senior officials from Flood Damage Restoration Headworks - - - 4 4 Directorate (FDRD) in Irrigation Department Senior official from 4 USAID - - - - 4 KP/FATA office Senior officials responsible PaRRSA 1 - - 5 6 for managing KPRP169 Senior officials of the Project Unit Project Unit established in 5 - - 3 8 C&W Department AGES Focal person for KPRP - - - 4 4 AiD Focal person for KPRP - - - 1 - 1 Total 4242

The trends in physical and financial progress over time, as reported in progress reports and project records, will be related to changes in the administrative process in order to answer part of evaluation question three. Information on administrative changes will be obtained through individual interviews and project documents.

The Khwazakhela Bridge, and the Amandara and Munda Headworks serve large populations in and outside of Malakand Division. Secondary data on some of the effects of these activities will be obtained from relevant government departments as indicated above. The evaluation team will use irrigation data to estimate the population and area of crops that have benefitted from the rehabilitation of the headworks. It will use secondary traffic data to estimate the number of vehicles, travelers and freight carriers (if possible) that have benefitted from the Khwazakhela Bridge.

Qualitative Data Analysis

The evaluation team will employ rigorous analytical methods appropriate to the focus group discussion and individual interviews. The team will use a structured approach based on identifying key themes, coding the responses according to the identified themes and reporting the relative importance of responses by theme. The analysis of the qualitative data will yield an in-depth understanding of the beneficiary perceptions about the effectiveness of the project, direct capacity building of PaRRSA personnel, indirect capacity building of line department staff, administrative changes in project implementation processes and the government-to-government (G2G) modality.

169 The individuals from PaRRSA will include project manager, chief infrastructure, planning officer and monitoring and evaluation officer at Peshawar, and senior planning officer and administrator at PMU in Swat.

E. METHODOLOGICAL STRENGTHS AND LIMITATIONS

The scope of this evaluation is limited to the evaluation questions and does not extend to other aspects of the project. The evaluation will employ multiple levels of triangulation in terms of information sources and data collection methods to ensure the reliability and validity of findings. In addition, the evaluation will use a mixed methods approach combining qualitative and quantitative methods. These are the main strengths of the methodology.

Through individual interviews and focus group discussions, qualitative methods will provide valuable insight into project performance and help capture the diversity of community perceptions about KPRP project components. However, purposive sampling used in qualitative methods has the shortcoming of selection bias and is, therefore, not representative or generalizable to the population at large. To address this shortcoming, this evaluation will use a probability sampling technique in the survey to ensure representative and generalizable survey results. Further, the findings, conclusions and recommendations emerging from this evaluation will be based on this convergent mixed method design, which relies on a combined analysis to ensure the reliability and validity of results.

IV. EVALUATION PRODUCTS

A. DELIVERABLES

The evaluation will produce the following deliverables:

1. Finalized draft SOW prepared in collaboration with USAID/Pakistan FATA/KP Office and USAID/Pakistan Program Office. The SOW will meet USAID expectations for quality evaluation Statements of Work;

2. Survey questionnaire and instruments for focus group discussions and individual interviews; 3. A debriefing discussion with USAID and the IP on initial findings (either separately or together, as decided by USAID);

4. A draft evaluation report; 5. A final evaluation report consistent with USAID standards for evaluation reports (outlined below) and no more than 30 pages (excluding annexes); and,

6. A presentation to USAID and the IP on the final report. 7. Field work will not start until USAID has approved the final SOW and commented on the instruments. B. REPORTING GUIDELINES

The evaluation report will follow standard guidelines as laid out in Appendix 1 of USAID’S Evaluation Policy and operationalized in ADS 203.3.1.8 (Documenting Evaluations), reproduced in Appendix 2. The evaluation report will follow the structure given below (the section titles and order are illustrative):

 Title page  Table of contents and table of tables and charts  List of acronyms  Acknowledgements  Project summary  Map showing the location of project activities  Executive summary (ideally not to exceed five pages)

 Evaluation purpose and evaluation questions (2-3 pages)  Project background. This information provides important context for understanding the evaluation purpose, questions, methods, findings and conclusions and includes: . the problem statement; . the theory of intervention; . the design of the project; . the project’s results framework; and, . project implementation, including the current status of the project.

 Evaluation methods and limitations (1-3 pages), describing in detail the evaluation design and methods with explanation as to why they were chosen, with additional information provided in the annexes, if so required.  Findings and conclusions. This section (or sections) will include findings and conclusions for each evaluation question. If there are a large number of findings, there will be a synthesis or summary of findings for each question that establishes the connection with the conclusions that follow.  Main conclusions and recommendations. This section will recapitulate the main conclusions, including those that form the logical basis for recommendations. It will highlight a few key recommendations, or clusters of recommendations, that include actionable statements of what remains to be done, consistent with the evaluation’s purpose, and based on the evaluation’s findings and conclusions. It will provide judgments on what changes need to be made for future USAID financial and cooperative development programming. This section will also recommend ways to improve the performance of future USAID programming and project implementation; ways to solve problems this project has faced; identify adjustments/corrections that need to be made; and recommend actions and/or decisions to be taken by management.  Annex

. Evaluation Statement of Work . Evaluation Methods and Limitations . Data Collection Instruments . Bibliography of Documents Reviewed . List of individuals and agencies contacted . Disclosure of Any Conflicts of Interest . Statement of Differences (only if applicable) . Evaluation Team Bios

V. TEAM COMPOSITION

The evaluation team will be led by an experienced evaluator and include an engineer and male and female researchers for focus group discussions and individual interviews, so that two team members can act as moderators/interviewers and two as note-takers at all times during the field work. All four team members will be Pashto speakers and two of them will be women. The team’s experience will also include familiarity with GOKP and donor assisted projects, including infrastructure projects, as well as sample surveys and statistical methods. Individual team member qualifications are given below.

A full-time Evaluation Specialist from MEP will be the Team Leader/Evaluator and lead the team throughout the evaluation in accordance with the USAID Evaluation Policy and directives. S/he is expected to possess prior experience in public sector administration within the KP government and the evaluation of USAID projects. S/he should have a master’s degree in social sciences or another relevant field, demonstrated team leadership, report writing abilities, communication skills, and fluency in English, Urdu and Pashto. USAID will review CVs of proposed candidates for this position and provide approval.

A Civil Engineer/Project Management Specialist will be a key member of the evaluation team. S/he is expected to have demonstrated experience in planning and implementation of infrastructure projects preferably in Khyber Pakhtunkhwa province of Pakistan. S/he should have a degree in civil engineering, experience in working with public sector entities such as the C&W department, Public Health Engineering and Highways Departments, report writing abilities, communication skills, and fluency in English, Urdu and Pashto. USAID will review CVs of proposed candidates for this position and provide approval.

A Researcher/Moderator (female) is expected to possess eight to twelve years of relevant experience in research and evaluation. She should have a master’s degree in social sciences and demonstrated experience in qualitative research methods and analysis, familiarity with gender issues, and must have fluency in English, Urdu and Pashto.

A Researcher/Note taker (female) will assist the Researcher/Moderator in note-taking during individual interviews and focus group discussions with female participants. She is expected to possess at least five to ten years of relevant experience in qualitative research, especially note-taking for individual interviews and focus group discussions. She must be fluent in English, Urdu and Pashto.

A Researcher/Note taker (male) will assist the Team Leader in note-taking during individual interviews and focus group discussions. He is expected to possess at least five to ten years of relevant experience in qualitative research, especially note-taking for individual interviews and focus group discussions. He must be fluent in English, Urdu and Pashto.

A Resource Person/Infrastructure Specialist from KP will work with the four-person evaluation team on a part-time basis. S/he will provide inputs on the design of instruments and analysis of data. S/he is expected to have diverse experience in infrastructure projects in KP and knowledge of GOKP institutions and procedures. USAID will review CVs of proposed candidates for this position and provide approval.

Disclosure of Conflict of Interest: All evaluation team members will provide a signed statement attesting to a lack of conflict of interest, or describing an existing conflict of interest relative to the project being evaluated.

VI. EVALUATION MANAGEMENT

A. LOGISTICS

In terms of logistics, this assignment requires:

 Travel to the project locations by the team members and the MEP staff responsible for the evaluation;

 Travel cost reimbursement to the focus group discussion participants (and individual interviews, if the evaluation team cannot access certain locations in view of travel restrictions); and,

 Access to project-related information and individuals representing various stakeholders; and,

 Organizing the focus group discussions and survey. The evaluation team will work under contract to MEP, which will be responsible for all travel, lodging, and other arrangements related to the team’s work. Prior to MEP issuing invitations, USAID will send a written notice to relevant government officials and other key participants to inform them about the evaluation and introduce them to MEP.

MEP has responsibility for identifying all key individuals and discussion group participants, inviting them to participate, organizing meeting schedules, arranging venues, and handling any necessary transport and lodging arrangements for participants.

B. SCHEDULING

TABLE 10: EVALUATION SCHEDULE

Jan. 2014 Feb. 2014 Mar. 2014 Apr. 2014 May. 2014 Activity W1 W2 W3 W4 W1 W2 W3 W4 W1 W2 W3 W4 W1 W2 W3 W4 W1 W2 W3 W4 Review of documents Draft of instruments Draft of initial chapters

Team Planning Meeting (TPM)

Qualitative data collection at provincial and district level

Qualitative data collection at community level Qualitative data analysis

Field work – Phase II Survey including training of enumerator, pretesting of instrument, data entry and analysis Report writing Internal review and revision

FCR workshop with USAID/IP

Branding and editing

Submission of final draft report

C. BUDGETING Table 11 reports estimates of the Level of Effort (LOE) of the team.

TABLE 11: LEVEL OF EFFORT OF TEAM MEMBERS

Level of Effort (days) Team Civil Researcher/ Researcher/ Researcher/ Evaluation Evaluation Director Coordinator Technical Senior Director G2G Tasks Leader Engineer Moderator Note Taker Note Taker Manager Co-Manager Survey Unit Survey Unit Director Advisor Evaluation Resource (LTTA) (STTA) (Female) (Female) (Male) (LTTA) (LTTA) (LTTA) (LTTA) (LTTA) Evaluation (LTTA) Person (STTA) (STTA) (STTA) (LTTA) (LTTA) Pre-arrival

 Review of 2 1 1 5 5 1 documents  Draft of 3 1 1 3 3 1 1 1 1 instruments  Draft of initial 4 1 1 1 chapters Team Planning 5 5 3 5 5 1 1 1 1 Meeting Training on qualitative data collection and 2 2 2 2 2 2 2 2 analysis Translation of data 2 1 2 collection instruments Field work – FGDs, 10 10 10 10 10 5 10 2 Individual Interviews Training/Pre-test for 1 1 1 3 3 Survey Data Analysis 5 5 3 5 5 5 5 2 Draft final report 10 5 3 3 3 1 3

Internal review and 3 1 1 revision FCR workshop 3 1 1 1 1 1 1

Final report 5 1 1 1 1

Presentation to USAID and 1

implementers Total 56 33 22 12 12 31 36 9 9 5 7 11 5

SOW Appendix 1: Draft Getting To Answers (G2A) Matrix

Type of Data Collection Data Evaluation Question Answer/ Analysis Evidence Method Sources Sampling Methods 1. In what ways and to what extent have Descriptive  Approximately 11 out  Individual interviews with government  Purposive Content the administrative processes, site Analytical of 31 individual officials, project managers and stakeholders sampling for analysis of identification, coordination, and interviews  FGDs with contactors and sub-contractors individual individual monitoring of sub-contractors, interview  2 out of 10 FGDs  Review of relevant project records such as interviews particularly through the G2G transcripts and  Review of relevant reports, official memos, notifications, modality, influenced project project record and minutes of meetings indicating changes in the secondary data implementation to date? secondary data administrative processes. 2. In what ways and to what extent have Descriptive  Household Survey  Survey with users of KPRP built schools and  Probability Statistical the project’s schools, supplies to basic Analytical  8 out of 31 individual public schools not damaged in the conflict sampling for Analysis of health units, bridges and irrigation interviews  FGDs with parents of children attending survey of survey data components170 contributed to the disaggregated  8 out 10 Focus Group schools and head teachers 800-900 achievement of intended results to by sex Discussions  Individual interviews with health care parents of date through completed and almost children  Review of relevant providers of strengthened BHUs completed activities? attending project record and  Individual interviews with those involved in Content both secondary data reconstruction to determine the build back analysis of categories of better design FGDs and schools individual  Review of relevant project records such as  Purposive interview reports, enrollment data, traffic count data, sampling for transcripts and data on land irrigated by the headworks and FGDs and secondary data data on beneficiaries of the DWSSs etc. individual interviews 3. In what ways and to what extent has Descriptive  Approximately 12 out  Individual interviews with government  Purposive Content KPRP built the capacity of PaRRSA Analytical of 31 individual officials, project managers and stakeholders sampling for analysis of and other government line interviews  Review of relevant project records such as individual FGD departments to date to provide public  Review of relevant reports, official memos and notifications to interviews transcripts and services? project record and institutionalize good practices introduced by secondary data secondary data the project

170 Capacity building is excluded from this question because it is addressed separately in question 2.

SOW Appendix 2: Reporting Guidelines

1. The evaluation report must represent a thoughtful, well-researched, and well organized effort to objectively evaluate what worked in the project, what did not work, and why.

2. Evaluation reports must address all evaluation questions included in the scope of work. The evaluation report should include the evaluation statement of work as an annex. All modifications to the statement of work, whether in technical requirements, evaluation questions, evaluation team composition, methodology or timeline need to be agreed upon in writing by the technical officer.

3. Evaluation methodology must be explained in detail and all tools used in conducting the evaluation such as questionnaires, checklists, and discussion guides will be included in an annex in the final report.

4. When evaluation findings address outcomes and impact, they must be assessed on males and females. 5. Limitations to the evaluation must be disclosed in the report, with particular attention to the limitations associated with the evaluation methodology (selection bias, recall bias, unobservable differences between comparator groups, etc.).

6. Evaluation findings must be presented as analyzed facts, evidence, and data and not based on anecdotes, hearsay, or simply the compilation of people’s opinions. Findings should be specific, concise, and supported by strong quantitative or qualitative evidence.

7. Sources of information must be properly identified and listed in an annex. 8. Recommendations must be supported by a specific set of findings and should be action-oriented, practical and specific, with defined responsibility for the action.

Note: These guidelines are taken from ADS 203.3.2.8 (Documenting Evaluations) - http://transition.usaid.gov/policy/ads/200/203.pdf - which is based on Appendix 1 of USAID Evaluation Policy: Criteria to Ensure the Quality of the Evaluation Report.

SOW Appendix 3: List of Schools with Their Construction Status

Sr. Construction Name of the School District Level of School Gender No. Status

GGCMS Kharere Chum Govt. Girls Community Model 1 Swat Girl Completed Matta School 2 GGMS Baboo Swat Govt. Girls Middle School Girl Completed 3 GGMS Bara Bandai Swat Govt. Girls Middle School Girl Completed

4 GGMS Gulibagh Swat Govt. Girls Middle School Girl Completed

5 GGMS Ningolai Swat Govt. Girls Middle School Girl Completed 6 GGMS Sersenai Swat Govt. Girls Middle School Girl Completed

7 GGMS Shakkar Dara Swat Govt. Girls Middle School Girl Completed

8 GGPS # 1 Charbagh Swat Govt. Girls Primary School Girl Completed 9 GGPS Jehan Abad Swat Govt. Girls Primary School Girl Completed

10 GGPS Kharere Matta Swat Govt. Girls Primary School Girl Completed

11 GGPS Mian Mira Arkot Swat Govt. Girls Primary School Girl Completed 12 GGPS Niligram Ronyal Swat Govt. Girls Primary School Girl Completed

13 GGPS Ningolai Swat Govt. Girls Primary School Girl Completed

14 GGPS Zangay Charbagh Swat Govt. Girls Primary School Girl Completed

15 GGPS Araq Swat Govt. Girls Primary School Girl Completed

16 GGPS Bar Qala Bishbanr Swat Govt. Girls Primary School Girl Completed

17 GMS Alamganj Swat Govt. Middle School (Boys) Boy Completed 18 GMS Aligrama Swat Govt. Middle School (Boys) Boy Completed

19 GMS Charbagh Swat Govt. Middle School (Boys) Boy Completed

20 GMS Kuz Shawar Swat Govt. Middle School (Boys) Boy Completed 21 GMS Nawagai Swat Govt. Middle School (Boys) Boy Completed

22 GPS Charbagh Swat Govt. Primary School (Boys) Boy Completed

23 GPS Ehingaro Dherai Swat Govt. Primary School (Boys) Boy Completed

24 GPS Hazara Swat Govt. Primary School (Boys) Boy Completed

25 GPS Kota Swat Govt. Primary School (Boys) Boy Completed

26 GPS Koza Bandai # 2 Swat Govt. Primary School (Boys) Boy Completed 27 GPS Kuz Shawar Swat Govt. Primary School (Boys) Boy Completed

28 GPS Shin Swat Govt. Primary School (Boys) Boy Completed

Sr. Construction Name of the School District Level of School Gender No. Status

29 GPS Gut Swat Govt. Primary School (Boys) Boy Completed

30 GPS Sher Palam Swat Govt. Primary School (Boys) Boy Completed Govt. Girls Higher Secondary 31 GGHSS Manglawar Swat Girl Dropped School 32 GGPS Ashar Banr Swat Govt. Girls Primary School Girl Dropped Govt. Girls Community Model Under 33 GGCMS Qala Swat Girl School Construction Under 34 GGHS Tirat Qandil Swat Govt. Girls High School Girl Construction Under 35 GGMS Manyar Swat Govt. Girls Middle School Girl Construction Under 36 GGMS Telegram Swat Govt. Girls Middle School Girl Construction Under 37 GGMS Araq Swat Govt. Girls Middle School Girl Construction Under 38 GGMS Bar Shawar Swat Govt. Girls Middle School Girl Construction Under 39 GGMS Kishawara Swat Govt. Girls Middle School Girl Construction Under 40 GGMS Shinkad Swat Govt. Girls Middle School Girl Construction Under 41 GGPS Chuprial Swat Govt. Girls Primary School Girl Construction Under 42 GGPS Garhi Chuprial Swat Govt. Girls Primary School Girl Construction Under 43 GGPS Manglawar Swat Govt. Girls Primary School Girl Construction Under 44 GGPS Mangra Sakhra Swat Govt. Girls Primary School Girl Construction Under 45 GGPS Asharay Malam Jaba Swat Govt. Girls Primary School Girl Construction Under 46 GGPS Awesha Chuprail Swat Govt. Girls Primary School Girl Construction Under 47 GGPS Barkalay Bishbanr Swat Govt. Girls Primary School Girl Construction Under 48 GGPS Bishbanr Swat Govt. Girls Primary School Girl Construction Under 49 GGPS Danai Shalpin Swat Govt. Girls Primary School Girl Construction Under 50 GGPS Ditpanai Swat Govt. Girls Primary School Girl Construction Under 51 GGPS Ghuzpatay Biha Swat Govt. Girls Primary School Girl Construction Under 52 GGPS Kass Khass Swat Govt. Girls Primary School Girl Construction GGPS Khwar Oba Bar Under 53 Swat Govt. Girls Primary School Girl Thana Construction Under 54 GGPS Kishawra Swat Govt. Girls Primary School Girl Construction

Sr. Construction Name of the School District Level of School Gender No. Status

Under 55 GGPS Lalkoo Sakhra Swat Govt. Girls Primary School Girl Construction Under 56 GGPS Malam, Kishawara Swat Govt. Girls Primary School Girl Construction Under 57 GGPS Mangar Kot Swat Govt. Girls Primary School Girl Construction Under 58 GGPS Paindai Sakhra Swat Govt. Girls Primary School Girl Construction Under 59 GGPS Qala Swat Govt. Girls Primary School Girl Construction GGPS Qasim Nasapa Under 60 Swat Govt. Girls Primary School Girl Darmai Construction GGPS Qayum Abad Under 61 Swat Govt. Girls Primary School Girl Lalkoo Construction Under 62 GGPS Saeed Abad Swat Govt. Girls Primary School Girl Construction Under 63 GGPS Seer Telegram Swat Govt. Girls Primary School Girl Construction Under 64 GGPS Shingrai Swat Govt. Girls Primary School Girl Construction Under 65 GGPS Shinkad Swat Govt. Girls Primary School Girl Construction GGPS Speny Oba Under 66 Swat Govt. Girls Primary School Girl Kishawara Construction Under 67 GGPS Sumbat Mandoor Swat Govt. Girls Primary School Girl Construction Under 68 GGPS Telegram Swat Govt. Girls Primary School Girl Construction GGPS Toor Toot Under 69 Swat Govt. Girls Primary School Girl Kishawra Construction GGPS Yakhtangai Aka Under 70 Swat Govt. Girls Primary School Girl Maroof Bamikhel Construction Under 71 GGPS Balasur Chuprail Swat Govt. Girls Primary School Girl Construction Under 72 GHS Bar Shawar Swat Govt. High School (Boys) Boy Construction Under 73 GMS Peochar Swat Govt. Middle School (Boys) Boy Construction Under 74 GMS Shakardara Swat Govt. Middle School (Boys) Boy Construction Under 75 GMS Barkalay Saidu Sharif Swat Govt. Middle School (Boys) Boy Construction Under 76 GPS Peochar Swat Govt. Primary School (Boys) Boy Construction Under 77 GPS Bar Shawar Swat Govt. Primary School (Boys) Boy Construction Under 78 GPS Bara Bandai # 2 Swat Govt. Primary School (Boys) Boy Construction GPS Chum Bara Under 79 Swat Govt. Primary School (Boys) Boy Durashkhela Construction

Sr. Construction Name of the School District Level of School Gender No. Status

Under 80 GPS Shakaradara Pir Kalay Swat Govt. Primary School (Boys) Boy Construction 81 GCMS Moranai Lower Dir Govt. Community Model School Boy Completed 82 GGPS Kotkay Lal Qila Lower Dir Govt. Girls Primary School Girl Completed

83 GMS Kumbar Lower Dir Govt. Middle School (Boys) Boy Completed

84 GPS Atoo Lower Dir Govt. Primary School (Boys) Boy Completed 85 GPS Dara Sher Khanai Lower Dir Govt. Primary School (Boys) Boy Completed

86 GPS Gumbatai Lower Dir Govt. Primary School (Boys) Boy Completed

87 GPS Mulyano Banda Lower Dir Govt. Primary School (Boys) Boy Completed 88 GPS Takatak Lower Dir Govt. Primary School (Boys) Boy Completed

89 GPS Lamotai Lower Dir Govt. Primary School (Boys) Boy Dropped GGHS Haya Serai Under 90 Lower Dir Govt. Girls High School Girl Balambat Construction Under 91 GGMS Khal Colony Khal Lower Dir Govt. Girls Middle School Girl Construction Under 92 GGPS Takatak Lower Dir Govt. Girls Primary School Girl Construction Under 93 GHS Sangolai Lower Dir Govt. High School (Boys) Boy Construction Under 94 GHS Shekolai Tormang Lower Dir Govt. High School (Boys) Boy Construction Govt. Higher Secondary School Under 95 GHSS Zaimdara Lower Dir Boy (Boys) Construction Under 96 GPS Ashrogai Lower Dir Govt. Primary School (Boys) Boy Construction Under 97 GPS Dapoor Lower Dir Govt. Primary School (Boys) Boy Construction Under 98 GPS Kotkay Lal Qilla Lower Dir Govt. Primary School (Boys) Boy Construction Under 99 GPS Machine Koorona Lower Dir Govt. Primary School (Boys) Boy Construction Under 100 GPS Mirgam Bala Lower Dir Govt. Primary School (Boys) Boy Construction Under 101 GPS Chiragalai Upper Dir Govt. Primary School (Boys) Boy Construction Under 102 GGHS Ganorai Upper Dir Govt. Girls High School Girl Construction Under 103 GPS Gatoo Upper Dir Govt. Primary School (Boys) Boy Construction 104 GPS Torwarsak Buner Govt. Primary School (Boys) Boy Completed

105 GPS Jowar No. 1 Buner Govt. Primary School (Boys) Boy Completed 106 GPS Gulbandai Buner Govt. Primary School (Boys) Boy Completed

Sr. Construction Name of the School District Level of School Gender No. Status

107 GPS Bekhanai Buner Govt. Primary School (Boys) Boy Completed

108 GPS Topi Buner Govt. Primary School (Boys) Boy Completed

109 GGPS Awanai Chagurzai Buner Govt. Girls Primary School Girl Completed 110 GGMS Topai Buner Govt. Girls Middle School Girl Completed Under 111 GPS Maradu Buner Govt. Primary School (Boys) Boy Construction Under 112 GPS Mairagai Buner Govt. Primary School (Boys) Boy Construction Under 113 GPS Kot Soray Buner Govt. Primary School (Boys) Boy Construction Under 114 GGPS Baikhanai Buner Govt. Primary School (Boys) Boy Construction Under 115 GMS Gul Bandai Buner Govt. Middle School (Boys) Boy Construction Under 116 GHS Gul Bandai Buner Govt. High School (Boys) Boy Construction Source: The list of focus schools and their construction status is taken from Al-Kasib Group of Engineering Services (AGES) regional office, Swat.

SOW Appendix 4: List of Under-construction Health Facilities

Sr. Name of Health Facility District Extent of Damage Construction Status No.

1 BHU Chongai Shamozai Swat Destroyed Under construction 2 BHU Nazarabad Swat Destroyed Under construction 3 BHU Miankilli Swat Destroyed Under construction 4 BHU Kishwara Swat Destroyed Under construction 5 BHU Topai Buner Destroyed Under construction 6 BHU Kotanai Swat Damaged Under construction 7 BHU Bandai Swat Damaged Under construction 8 BHU Bashbanr Swat Damaged Under construction 9 BHU Mashkomai Swat Damaged Under construction 10 BHU Shalpin Swat Damaged Under construction 11 CD Taligram Swat Damaged Under construction 12 CD Parrai Swat Damaged Under construction 13 BHU Chinar Kot Dir Lower Damaged Under construction 14 BHU Deedan Pura Dir Lower Damaged Under construction 15 BHU Moranai Dir Lower Damaged Under construction 16 BHU Shagai Dir Lower Damaged Under construction 17 BHU Zamdara Dir Lower Damaged Under construction 18 BHU Kitiari Dir Lower Damaged Under construction

Source: The list of focus BHUs and their construction status is taken from KPRP quarterly report of January-March, 2013 quarter.

SOW Appendix 5: List of Health Facilities Strengthened by KPRP

District Swat District Buner District Lower Dir

1. BHU Bandai 1. BHU Bagh 1. BHU Chanar Kot 2. BHU Banjoot 2. BHU Battra 2. BHU Deedan Pura 3. BHU Barshawar 3. BHU Dokada 3. BHU Shagai 4. BHU Choongai 4. BHU Elai 4. BHU Zaindara 5. BHU Dureshkhila 5. BHU Gagra 5. BHU Moranai 6. BHU Gulibagh 6. BHU Gokand 6. RHC Asbanr 7. BHU Kishwara 7. BHU Koria 7. RHC Gulabad 8. BHU Meragai 8. BHU Malka 8. RHC Lal Qala 9. BHU Mian Kalay 9. BHU Pander 9. RHC Z. Talash 10. BHU Nazar Abad 10. BHU Serqala 10. THQ Chakdara 11. BHU Ronial 11. BHU Topai 12. BHU Talang 12. CH Chamla

13. CH Madyan 13. CH Pacha Kalia

14. CH Manglore 14. CH Tota Lai

15. RHC Chuprial 15. RHC Dewana Baba 16. RHC Develai 16. RHC Jowar 17. RHC Khazana 17. RHC Nagri 18. THQ Matta Source: The list of health facilities strengthened with KPRP provided furniture, equipment and health supplies is taken from PaRRSA PMU, Swat.

SOW Appendix 6: List of Equipment and Furniture Provided to Health Facilities Strengthened by KPRP171

Sr. No. List of Items A Furniture 1 Office Table with 3 Drawers 2 Office Chair 3 Office Rack Wooden 4 Wooden Stool 5 Wooden Bench B Equipment 1 Steel cabinet Model A107 2 Dressing Cart Model 1182SS 3 Surgical Examination Light Model 434 4 Enema Kit, SS Model AMZ011 5 Examination Torch Reusable AA batteries Model 10-50R 6 Fetoscope De Lee Model 3241 7 Refrigerator Medical use, with auto stabilizer Model S405 8 Mucus Extractor Model H010 9 Surge Protector Model APC-USA 10 Pillow+Cover Model 65-09 C Medical Equipment 1 Dilation & Curettage Diagnostic Set 2 Vaginal Speculum Small / Medium 3 Dressing Drum 9x9 SS 4 Kidney Tray 8” SS 5 Kidney Tray 10” SS 6 Kidney Tray 12” SS 7 Mercury Blood Pressure Focal Japan 8 Stethoscope MDF USA Adult / Pediatric 9 Otolaryngology (ENT) Diagnostic Set ADC USA 10 Basin SS with stand 11 Electric Water Sterilizer 12 Oxygen Gas Cylinder with gauge & key with trolley 13 Needle Holder forceps 14 Steel Bucket 15 Bowel 16 Artery Forceps 17 Curved Artery forceps 6” 18 Curved Artery Forceps 8” 19 Clinical Thermometer Flat 20 Dressing Tray 8*10 21 Waste paper bucket

171 Not everything in the list was provided to all the health facilities covered by this intervention.

Sr. No. List of Items 22 Dressing Scissor 6.5” 23 Emergency Box with complete contents as per order 24 Tongue depressor steel 15cm 25 Tongue depressor steel 19 cm 26 Cord Clamp 27 Baby weighing machine 28 Weighing scale adult 29 Delivery Kit with complete contents as per order 30 Tissue forceps toothed 6” 31 Tissue forceps plain 6” 32 Scissor Straight 6” 33 Scissor curved 6” 34 Scissor blunt point 6” 35 Revolving Stool 36 Examination Couch 37 Delivery Table 38 Foam Mattress 4” with cover 39 Bad sheet 60” * 100” Cotton 40 Bed pan male 41 Bed pan female 42 Bed side locker 43 IV Stand 44 Wheel Chairs 45 X-Ray View Box 46 Needle Cutter 47 Beds 48 Surgical Blades 49 Surgical Handles 50 Breast Pump Kit 51 Screen for folding (Bedside screen) 52 Patient Trolleys Source: The list of furniture, equipment and health supplies provided to health facilities is taken from PaRRSA PMU, Swat.

SOW Appendix 7: List of Under Construction Drinking Water Supply Systems in Buner

S. No. Name of Scheme District Completion Status

1 DWSS Maskipur Buner Under construction 2 DWSS Qabla Maira Buner Under construction 3 DWSS Dewana Baba Buner Under construction 4 DWSS Sunigram Buner Under construction 5 DWSS Shalbandai Buner Under construction 6 DWSS Soor Kamar Maria Buner Under construction 7 DWSS Chowngai Kohay Dara Buner Under construction 8 DWSS Aso Dara Buner Under construction 9 DWSS Jang Dara Buner Under construction 10 DWSS Bagra Maira Buner Under construction 11 DWSS Jowar Bazargai Buner Under construction 12 DWSS Dagai Buner Under construction 13 DWSS Banda Krrapa Buner Under construction

Source: This list of focus drinking water supply systems and their construction status is taken from KPRP quarterly report of January-March, 2013 quarter.

ANNEX II: SOURCES OF INFORMATION

Focus Group Discussion (FGD) Participants

TABLE 4: SCHOOLS INCLUDED IN FOCUS GROUPS OF PARENTS AND HEAD TEACHERS OF CHILDREN IN KPRP- ASSISTED SCHOOLS

KPRP Schools Included in Focus KPRP Schools Included in Focus Group Discussions with Head Teachers Groups with Parents of Children Focus Groups Focus Groups Female Male with Mothers with Fathers Government Girls Government Girls Primary School (GGPS), Mingola, Swat Government Primary School (GPS) No. 1, Charbagh, Swat Community Model Primary School Government Girls High School (GGHS), Mingloly, Swat Government Primary School (GPS), Kota, Swat School (GGCMS), (GPS), Sheen, Swat Qala, Swat Government Girls High School (GGHS), Migily, Swat Government Primary School (GPS), Hazara, Kabal, Swat Government Girls High School (GGHS), Sar Sainai, Swat Government Primary School (GPS), Alingaro Dehrai, Swat Government Girls Government Government School (GGPS), Awany, Chagharzai, Buner Government Primary School (GPS), Maradu, Buner Primary School Middle School Government Girls High School (GGHS), Topai, Chagharzai, Government Primary School (GPS), Kot Soray, Buner (GGPS), Awarain, (GMS), Tor Buner Buner Warsak, Buner Government Girls Primary School (GGPS), Baikhani, Government Primary School (GPS), Topai, Buner Chagharzai, Buner Government Girls High School (GGHS), Sairi Haya, Maidan, Government Primary School (GPS), Mairagai, Buner Lower Dir Government Girls Government Government Girls Middle School (GGMS), Khall Colony, Lower Government Primary School (GPS), Baikhanai, Buner Primary School Middle School Dir (GGPS), Kotky LaL (GMS), Kambar, Government Girls Community Model School (GGCMS), Morani, Government Primary School (GPS), Takatak, Lower Dir Qila, Lower Dir Lower Dir Lower Dir (Two participants) Government Primary School (GPS), Mulyano Banada, Lower Dir Government Primary School (GPS), Machine Korona, Lower Dir Government Primary School (GPS), Kotkay, Lower Dir

TABLE 5: TOTAL NUMBER OF FOCUS GROUP DISCUSSIONS AND PARTICIPANTS BY DISTRICT

FGD Swat Buner Lower Dir Total

1 FGD with Fathers of KPRP School Children (Government Primary School (GPS), Sheen, Swat) 11 0 0 11 2 FGD with Mothers of KPRP School Children (Government Girls Community Model School (GGCMS), 10 0 0 10 Qala, Swat) 3 FGD with Fathers of KPRP School Children (Government Middle School (GMS), Tor Warsak, Buner) 0 8 0 8 4 FGD with Mothers of KPRP School Children (Government Girls Primary School (GGPS), Awarain, 0 9 0 9 Buner) 5 FGD with Fathers of KPRP School Children (Government Middle School (GMS), Kambar, Lower Dir) 0 0 10 10 6 FGD with Mothers of KPRP School Children (Government Girls Primary School (GGPS), Kotky LaL 0 0 11 11 Qila, Lower Dir) Sub-total for Parents 21 17 21 59  Mothers 10 9 11 30  Fathers 11 8 10 29 7 FGD with Male Head-teachers 4 5 4 13 8 FGD with Female Head-teachers 5 2 5 12 Sub-total for Head Teachers 9 7 9 25 9 FGD with Contractors (Group 1) 12 0 0 12 10 FGD with Contractors (Group 2) 6 0 7 13 Sub-total for Contractors 18 0 7 25 Total 117 65 95 277

List of Persons Interviewed

TABLE 6: INTERVIEWS IN ISLAMABAD, APRIL 1, 2014 TO APRIL 5, 2014

Number of Name of Organization Position of Interviewee Interviewees USAID/Pakistan Current and previous KPRP Program Managers 4 Associates in Development Person assigned to provide information on the Amandara 1 Headworks Sub-total 5

TABLE 7: INTERVIEWS IN PESHAWAR, APRIL 7, 2014 TO APRIL 12, 2014

Number of Name of Organization Position of Interviewee Interviewees AGES Team Leader, Partner, Planning Officer, and In-charge WATSAN 4 PaRRSA Program Manager, Chief Planning Officer, Financial Advisor; Chief 5 Infrastructure, and Planning Officer C&W Department Current and former Project Directors, Project Unit, KPRP 3 FDRD, Irrigation Deputy Director, Assistant Director, Accounts Officer, and Ex 4 Department Project Director Sub-total 16

TABLE 8: INTERVIEWS IN MALAKAND, APRIL 15, 2014 TO APRIL 23, 2014

Number of Name of Organization Position of Interviewee Interviewees District Education District Education Officers, Swat, Buner and Lower Dir 4 Department District Health Department Male and Female Medical Officers and Male Medical Technicians 7 at the Basic Health Units (BHUs) Subcontractors Subcontractors from Swat, Buner, and Lower Dir 4 C&W Department Deputy Director PU, Assistant Director PU, Divisional Accounts 5 Officer PU, and Infrastructure Coordinators, Lower Dir and Swat PaRRSA In-charge PMU PaRRSA, Mingora 1 Sub-total 21

ANNEX III: DATA COLLECTION INSTRUMENTS

Khyber Pakhtunkhwa Reconstruction Program (KPRP) Mid-term Performance Evaluation

Individual Interview Instrument 1 – for Interviews with USAID, Government and Monitoring Firms on Question 1

Introduction

Thank you very much for meeting us today. My name is ______and I represent the USAID Monitoring and Evaluation Program (MEP). Our project helps USAID by monitoring and evaluating USAID projects in Pakistan. It is implemented by Management Systems International (MSI), a US consulting firm. USAID has asked MSI to conduct a mid-term evaluation of KPRP and identify possible improvements for the remainder of the project. I would like to ask you a few questions about KPRP and Mr. ______will take notes. The main subject of this discussion is how KPRP has been implemented so far. This covers all aspects of implementation including needs assessment, site selection, planning, implementation, quality assurance, monitoring, financial management and so on, the responsibilities for these as well as the relationships among the various organizations that are involved in managing KPRP.

Confidentiality

With your permission, we would like to record this discussion so that we accurately capture what you say and do not miss any important point. Please be assured that your responses will be kept strictly confidential and not shared with anyone. In case we need to mention something you have said in our evaluation report, you will not be identified by name, position or organization but only as an official or manager or, where appropriate, as an official or manager responsible for a specific aspect of management, such as planning, implementation, quality assurance, monitoring, financial management and so on.

Individual Interview Code: nn

Main Topic of Discussion Evaluation Question Number 1 Name of Organization172 Position of Individual With KPRP Since: Month: Year: Date of Interview Venue City

Name and Position of Interviewer Name and Position of Note-taker

Question 1 What was your role with KPRP? [Prompt: Use the attached sheet “Dimensions of Evaluation Questions” to focus the interview on the individual’s areas of responsibility]

Answer

Question 2 Did other people in your organization have a key role in working with KPRP? For which processes?

Answer

172 The highlighted information will be kept separate from the notes (in a document called “List of Sources”) to maintain confidentiality.

Question 3 What can you tell us about the needs assessment for destroyed or damaged schools?173 [Prompt: Going back to the start of KPRP], Answer

Question 4 How have school sites been selected in KPRP since the project started? [Prompt: It may be useful to also explore how well has this process worked]

Answer

Question 5 How have sub-projects been approved in KPRP since the project started? [Prompt: Need to understand the role of the District Reconstruction and Rehabilitation Committee and why approvals were given so speedily in the first year]

Answer

Question 6 What planning process has been followed for schemes/sub-projects since KPRP started? [Prompt: Ask about all the steps after approval up to and including preparation of tender documents.]

Answer

Question 7 Has the planning process changed over time? [Prompt: Ask for references to specific rules and documents such as FARA, PIL and government documents] [Prompt: Has the Project followed the planning process required under KPRP? If not, why not?] [Prompt: Ask for references to specific rules and documents such as FARA, PIL and government documents] [Prompt: Are most respondents really going to know what planning systems were actually required under PKRP? Will they simply refer to normal GOP planning processes? And more broadly: does it matter for the evaluation?]

Answer

Question 8 How are sub-projects or schemes contracted out (after tenders are announced) in your component? [Prompt: Ask about changes over time and whether they were officially notified]

Answer

Question 9 There are reports that contractors sometimes sub-let (assign) or sell their contracts to subcontractors. Have you seen this happening? Is this an important issue? [Prompt: What are the implications of this assignment of contracts?] [Prompt: Is it permissible according to the regulations governing KPRP? How widespread is it? What are its results in terms of efficiency, quality and other factors?]

Answer

Question 10 How has the Project arranged for technical oversight during implementation since it started and how has it changed over time? [Prompt: To what extent has the project arranged technical oversight as required officially under KPRP? Ask for references to specific rules and documents such as FARA, PIL and government documents] Answer

173 Questions 3 and 4 are for those individuals in USAID, PaRRSA, C&W, the Department of Education and AGES who are or were associated with the school reconstruction component.

Question 11 AGES and PaRRSA reports describe design issues, poor quality, slow pace of construction and a failure to meet targets. How do you respond to these reports? [Prompts: Refer to the list provided in Attachment 2] Answer

Question 12 How do you report physical and financial progress for the components with which you deal? [Prompt: Ask about changes over time and whether they were officially notified]

Answer

Question 13 What can you tell us about the quality of the reporting process for KPRP? Prompt: For the first year of the project, [April 2010 to March, 2011] the quarterly progress reports are missing. The Program financial status documentation is missing until the Quarterly report for July to September, 2012. Can you help us understand why this happened? Three QPRs also remain in draft form and were not finalized.]

Answer

Question 14 What can you tell us about the Project’s system of monitoring and evaluation from the inception of KPRP to date? Has it helped improve the quality of design, construction, and implementation processes? Prompt: Includes spot checks by PaRRSA, CRISP, AGES and AiD. Answer

Question 15 Some reports say that CRISP’s performance was unsatisfactory and that it was not adequately staffed. Did you see evidence of unsatisfactory performance from CRISP? Prompt: Was this issue addressed during CRISP’s contract? How? Answer

Question 16 How are funds transferred from the GOP to your organization? Does money get transferred from your department to other government organizations?

Answer

Question 17 How are contractor bills/invoices certified and how do they get paid? [Prompt: KPRP has used a system of payments against milestones? How well has this worked? Ask about all the milestones and the percentage of total payment they trigger]

Answer

Question 18 Does the system of certification by the consultants (AGES, AiD) help achieve quality and efficiency? If yes, how? If not, why? Answer

Question 19 KPRP has utilized a Project Unit in C & W. How did this affect project coordination and implementation? Prompt: What led to the creation of the PU in the C&W Department? Answer

Question 20 How does your organization coordinate with other stakeholders? [Prompts: How and to what extent have your coordination and relationships with government organizations affected implementation? Have management and/or coordination issues affected implementation? The stakeholders include USAID, PaRRSA, C&W, FDRD, PHED, PKHA, the Departments of Health and Education, the two monitoring firms (AiD and AGES) and the Project Steering Committee chaired by the Director General of PaRRSA.]

Answer

Question 21 What is the role of AGES/AID in your work? And what was your experience with them? [Prompt: For Amandara, it is AiD, not AGES. Has coordination with AGES been an issue for you? Any potential issues of conflict of interest involving AGES?]

Answer

Question 22 What Project implementation issues have worked well? Why? Which ones have not worked well? Why? [Prompt: Has FAR changed project implementation modalities across the departments?]

Answer

Question 23 Do the Project’s implementation modalities need to be changed? Which? Why? How? [Prompts: Ask about efficiency, meeting targets, quality assurance and issues brought up earlier in the interview]

Answer

Attachment 1 to Individual Interview Instrument 1 – Dimensions of Evaluation Questions174

Question 1: In what ways and to what extent have the administrative processes, site identification, coordination, and monitoring of sub-contractors, particularly through the G2G modality, influenced project implementation to date?

1. Identification and selection of suitable sites 2. Proper planning for implementation 3. Approval of the sub-projects or schemes 4. Contracting out the sub-projects or schemes 5. Prevalence and impacts of engaging unapproved sub-contractors 6. Technical oversight 7. Reporting (physical and financial, monthly and quarterly reporting) 8. Monitoring and evaluation (PaRRSA, AGES and AiD - spot checks) 9. Funds transfers among GOP agencies/departments 10. Funds transfer to contractors/sub-contractors 11. Coordination and communication among various stakeholders 12. USAID relationship with the GOP (including the PU) and all of the other stakeholders 13. Relationship of GOP agencies/departments (including the PU) and contractors/sub-contractors to the outside monitoring and evaluation firm 14. Systems and overall capabilities of GOP agencies/departments (including the PU) to provide proper and adequate technical oversight 15. Delivery of outputs 16. Efficiency of the current mode of implementation

Question 2: In what ways and to what extent have the project’s schools, supplies to basic health units (BHUs), bridges and irrigation components contributed to the achievement of intended results to date through completed and almost completed activities?

1. Health: Access to improved health facilities 2. Communication: Extent to which travel through bridges improved 3. Irrigation: Extent to which access to irrigation water improved 4. Education: Extent to which access to schools improved for boys and girls 5. Where the currently enrolled children went to school when schools were not reconstructed (boys and girls) 6. Effect of changed construction on enrollment

Question 3: In what ways and to what extent has KPRP built the capacity of PaRRSA and other government line departments to date to provide public services?

1. Direct contribution to PaRRSA capacity building 2. Construction practices which KPRP introduced replicated by line departments

174 Please note that the sub-questions we need to answer are more specific – and more numerous – than the dimensions listed here; refer to the table “Deconstructing the Evaluation Questions” at the end of this document.

Attachment 2 to Individual Interview Instrument 1 – Design and Quality Issues Identified in PaRRSA and AGES Reports

1. Disability concerns were not addressed in many schools. 2. Toilets were not provided in the administration blocks of schools. 3. Drinking water supply is a problem in and temporary arrangements made by the teachers from the local community are not adequate and sustainable. 4. Latrines in the schools have been found closed as water is not available. There is improper and unsanitary use of toilets and open defecation was observed in some schools. 5. Improper slab concrete or roof treatment, causing seepage and deterioration of finishing material. 6. Improper finishing of doors and windows. 7. Installation of poor quality of wooden doors. 8. Cracks were found in wooden doors. 9. Poor quality distemper and paints were used. 10. GPS Sheen School has a large enrollment (600) and the school building is not sufficient for the children.

[More can be added]

Khyber Pakhtunkhwa Reconstruction Program (KPRP) Mid-term Performance Evaluation

Individual Interview Instrument 2 – for Sub-contractors in Malakand

Introduction

Confidentiality

Individual Interview Code: nn

Main Topic of Discussion Evaluation Question Number 1 Name of Organization Position of Individual With KPRP Since: Sub-contractor Month: Year: Date of Interview Venue City

Name and Position of Interviewer Name and Position of Note-taker

Question 1 What is the purpose of sub-contracting in the KPRP Project? Answer

Question 2 What is/was/ your role as a sub-contractor in KPRP? [Prompt: ask about division of labor and sharing responsibility at the sub-project/scheme level]

Answer

Question 3 How did you become a sub-contractor? How were you selected? [Prompts: ask about the process and if there is any documentation]

Answer

Question 4 During the course of your work, what interaction(s) have you had with officials from the C&W Department? Answer

Question 5 What interactions did you have with CRISP and AGES?

Answer

Question 6 What has your experience been working with KPRP? [Prompt: ask about issues, payments,

monitoring, learning and so on]

Answer

Question 7 What could be done to improve your work with KPRP?

Answer

Khyber Pakhtunkhwa Reconstruction Program (KPRP) Mid-term Performance Evaluation

Individual Interview Instrument 3 – for Health Service Providers at BHUs in Malakand

Introduction

Thank you very much for meeting us today. My name is ______and I represent the USAID Monitoring and Evaluation Program (MEP). Our project helps USAID by monitoring and evaluating USAID projects in Pakistan. It is implemented by Management Systems International (MSI), a US consulting firm. USAID has asked MSI to conduct a mid-term evaluation of KPRP and identify possible improvements for the remainder of the project. I would like to ask you a few questions about KPRP and Mr. ______will take notes. The main subject of this discussion is the assistance this BHU has received through KPRP and how it has affected the BHU and the people of the area.

Confidentiality

With your permission, we would like to record this discussion so that we accurately capture what you say and do not miss any important point. Please be assured that your responses will be kept strictly confidential and not shared with anyone. In case we need to mention something you have said in our evaluation report, you will not be identified by name or position but only as an or manager official or, where appropriate, as an official or manager responsible for a specific aspect of management, such as planning, implementation, quality assurance, monitoring, financial management and so on.

Individual Interview Code: nn

Main Topic of Discussion Evaluation Question Number 2 Name of Organization Position of Individual At this BHU Since: Basic Health Unit Female Medical Office/ Month: Year: Lady Health Visitor Date of Interview Venue City

Name and Position of Interviewer Name and Position of Note-taker

Question 2 What assistance has the BHU received through KPRP?175

[Note: it might be better to start with Question # 2, and then go to Q # 1.] Answer

Question 1 What health services did this BHU provide before receiving assistance from KPRP. [Prompts: Emergency care, maternal care, obstetric care, mother and child care, surgery, outdoor patient checkup, hospitalization]

Answer

Question 3 Has the BHU’s services changed after receiving KPRP assistance? How?

Answer

175 Check the response against the list of supplies provided through KPRP to this BHU.

Question 6 Has the provision of furniture, supplies and equipment affected the quality of health care you provide in this BHU? How?

Answer

Question 5 Has KPRP assistance affected those who come to use the BHU? How? Has it improved access? How? [Prompt: Ask about differences across women, men and children]

Answer

Question 6 What could be done differently to improve the results of KPRP assistance? [Prompt: Ask about differences across women, men and children] Answer

Khyber Pakhtunkhwa Reconstruction Program (KPRP) Mid-term Performance Evaluation

Individual Interview Instrument 4 – for District Education Officers in Malakand

Introduction

Thank you very much for meeting us today. My name is ______and I represent the USAID Monitoring and Evaluation Program (MEP). Our project helps USAID by monitoring and evaluating USAID projects in Pakistan. It is implemented by Management Systems International (MSI), a US consulting firm. USAID has asked MSI to conduct a mid-term evaluation of KPRP and identify possible improvements for the remainder of the project. I would like to ask you a few questions about KPRP and Mr. ______will take notes. The main subject of this discussion is how the schools reconstructed with KPRP assistance have affected children of school going age in this district.

Confidentiality

With your permission, we would like to record this discussion so that we accurately capture what you say and do not miss any important point. Please be assured that your responses will be kept strictly confidential and not shared with anyone. In case we need to mention something you have said in our evaluation report, you will not be identified by name or position but only as an official or, where appropriate, as an education department official.

Individual Interview Code: nn

Main Topic of Discussion Evaluation Question Number 2 Name of Organization Position of Individual In Present Position Since: District Education Department District Education Officer Month: Year: Date of Interview Venue City

Name and Position of Interviewer Name and Position of Note-taker

Question 1 What happened to school-going children and teachers after their schools were destroyed ? How did children, parents and teachers respond to the situation? [Prompt: Ask separately for boys and girls if they were in another school, another district, dropped out, etc.]?

Answer

Question 2 Were difficulties encountered by teachers, children and their parents before the reconstructed schools were completed? [Prompt: Ask about differences across women, men and children]

Answer Question 3 How do you compare the features of the old school with the new schools constructed with KPRP assistance?

Answer

Question 4 Have the new features of the KPRP reconstructed schools affected student access? [Prompt: are there changes in enrollment, drop out, attendance, or other aspects that help to show access? Ask separately for boys and girls]

Answer

Question 5 How do parents view the new features of the KPRP-assisted schools? [Prompt: boundary walls, toilets and number of class rooms; ask separately for views affecting girls’ and boys’]

Answer

Question 6 Did the project engage parents and the community in taking decisions about the schools? How? [Prompt: Ask about mothers and fathers as well as male and female members of the community]

Answer

Question 7 What could KPRP do differently in reconstructing the schools?

Answer

Khyber Pakhtunkhwa Reconstruction Program (KPRP) Mid-term Performance Evaluation

Individual Interview Instrument 5 – for Interviews with USAID and PaRRSA on Question 3 (Direct Capacity Building – PaRRSA)

Introduction

Thank you very much for meeting us today. My name is ______and I represent the USAID Monitoring and Evaluation Program (MEP). Our project helps USAID by monitoring and evaluating USAID projects in Pakistan. It is implemented by Management Systems International (MSI), a US consulting firm. USAID has asked MSI to conduct a mid-term evaluation of KPRP and identify possible improvements for the remainder of the project. I would like to ask you a few questions about KPRP and Mr. ______will take notes. The main subject of this discussion is how KPRP has helped with the capacity building of PaRRSA. Are you aware that USAID has been providing capacity building assistance to PaRRSA?176

Confidentiality

Individual Interview Code: nn

Main Topic of Discussion Evaluation Question Number 3 Name of Organization Position of Individual With PaRRSA Since: Month: Year: Date of Interview Venue City

Name and Position of Interviewer Name and Position of Note-taker

Question 1 In your opinion, what is the role and purpose of PaRRSA? [Prompts: Why is USAID providing capacity assistance to PaRRSA? Does PaRRSA need capacity assistance?] Answer

Question 2 Has KPRP assistance affected the operations of PaRRSA? How? [Prompt: for example, planning, coordination, monitoring and financial management] Answer

Question 3 Can you identify the two or three areas of responsibility or management functions of PaRRSA that have been most affected? How? Answer

Question 4 Is there a link between assistance to PaRRSA and the implementation of KPRP?

Answer

176 The interview will be aborted if the individual is not aware of USAID assistance for capacity building.

Question 5 Are there important gaps in PaRRSA’s capacity that have not been filled by USAID assistance through KPRP? [Prompt: Focus on resources as well as areas of responsibility/management functions]

Answer

Question 6 Have other donors or other program (including USAID’s Assessment and Strengthening Program) helped fill any of these gaps? How and to what extent?

Answer

Question 7 What could be done to improve PaRRSA? [Prompt: What could KPRP do differently to build the capacity of PaRRSA?] [Based on your experience with PaRRSA, are there important capacity gaps that have not been addressed so far? If yes, please describe.]

Answer

Khyber Pakhtunkhwa Reconstruction Program (KPRP) Mid-term Performance Evaluation

Individual Interview Instrument 6 – for Interviews with USAID and Line Departments on Question 3 (Indirect Capacity Building – C&W, FDRD, PKHA, PHED and TMA Swat)

Introduction

Thank you very much for meeting us today. My name is ______and I represent the USAID Monitoring and Evaluation Program (MEP). Our project helps USAID by monitoring and evaluating USAID projects in Pakistan. It is implemented by Management Systems International (MSI), a US consulting firm. USAID has asked MSI to conduct a mid-term evaluation of KPRP and identify possible improvements for the remainder of the project. I would like to ask you a few questions about KPRP and Mr. ______will take notes. The main subject of this discussion is your department’s experience with the construction practices followed in KPRP.

Confidentiality

Individual Interview Code: nn

Main Topic of Discussion Evaluation Question Number 3 Name of Organization Position of Individual In this position since: Month: Year: Date of Interview Venue City

Name and Position of Interviewer Name and Position of Note-taker

Question 1 As a government official engaged in infrastructure development, please tell us about the differences, if any, that you have observed between the government’s typical177 construction practices and those found in KPRP.

Answer

Question 2 Do these differences represent improvements over the government’s typical approach? If yes, how?

Answer

Question 3 Has your department’s staff learned from these approaches? [Prompt: Ask about numbers and functions (design, planning, tendering/contracting, supervision, quality assurance, etc.) of those said to be engaged in learning]

177 “Normal” means those practices that are followed in projects funded by the government’s own resources – the Annual Development Plan (ADP) – and not by donor resources.

Answer

Question 4 Has this learning extended to other work of the department outside the KPRP project?

Answer

Question 5 Is your department implementing any improved construction practices introduced by KPRP in projects funded through the ADP? [Prompt: Ask for project name, specific location and documentary evidence.]

Answer

Question 6 Are there improved practices that your department has not adopted? If yes, then why?

Answer

Question 7 What would enable your department to implement additional improved construction practices, using the government’s own resources, over the next 2-3 years?

Answer

KPRP EVALUATION – DECONSTRUCTING THE G2G QUESTION

Data Collection and Analysis Requirements for Findings, Conclusions and Recommendations → negative) prevailing needed in changes in changes rationale forrationale Modifications arrangements arrangements arrangements arrangements

Nature of and Description of Assessment of implementation implementation implementation implementation (positive and/or Processes, Relationships and Other Effects of changes Dimensions of Evaluation Question 1 ↓ Findings Conclusions Recom- mendations 1. Needs assessment for damaged infrastructure 2. Identification and selection of suitable sites 3. Proper planning for implementation 4. Approval of the sub-projects or schemes 5. Contracting out the sub-projects or schemes 6. Prevalence and impacts of sub-awarding schemes or sub-projects by a recipient contractor to an unapproved sub- contractor 7. Technical oversight 8. Reporting (periodic basis) 9. Monitoring and evaluation (spot checks) 10. Funds transfers among GOP agencies/ departments 11. Funds transfers for contractors/subcontractors Relationships 12. Coordination and communication among various stakeholders 13. USAID relationship with the GOP (including the PU) and all of the other stakeholders 14. Relationship of GOP agencies/ departments (including the PU) and contractors/sub-contractors to the outside monitoring and evaluation firm Other Dimensions - Different view of the technical oversight dimension: Systems and overall capabilities of GOP agencies/departments (including the PU) to provide proper and adequate technical oversight 15. Delivery of outputs 16. Efficiency: timeliness of the current mode of implementation and efficiency of the process Notes:

100

1. Documents suggest that not all processes have changed over time and that some processes and relationships have experienced only one or two changes over time, but this needs to be triangulated through relevant individual interviews, FGDs and official notifications (to be obtained during individual interviews).

2. Delivery of outputs and efficiency are consequences rather than processes and do not require description as such.

101

Khyber Pakhtunkhwa Reconstruction Program (KPRP) Mid-term Performance Evaluation

Focus Group Discussion (FGD) Guide – Number 1 For Contractors Engaged in the Construction of KPRP-assisted Infrastructure

FGD Code: FGD nn

Main Topic of Discussion Evaluation Question Number 1 Date of FGD Venue City Group One/Two Name and Position of Moderator Name and Position of Note-taker

Introduction

Thank you very much for joining us today for this focus group discussion. My name is ______and I represent the USAID Monitoring and Evaluation Program (MEP). This program helps USAID by independently monitoring and evaluating USAID projects in Pakistan. It is implemented by Management Systems International (MSI), a US consulting firm. USAID has asked us to conduct a mid-term evaluation of KPRP and identify possible improvements for the remainder of the project duration. USAID is keen that this should be a learning exercise for USAID as well as the government. The evaluation report will be disclosed to the public when it has been finalized.

The main subject of this discussion is how you have participated in construction activities supported by KPRP. In this connection, we would like to discuss your participation in all steps of the process, from the time of pre-qualification and tendering to the actual construction, its supervision and payment.

We would like to encourage all of you to discuss these subjects frankly as a group and in so doing share your thoughts with us. I will facilitate discussion by asking a few questions, one at a time, and try to ensure that everyone feels free to participate in the discussion. Please allow everyone to express their views and focus on the questions. My colleague ______will take notes.

Confidentiality

With your permission, we would like to record this discussion so that we accurately capture what you say. Please be assured that your responses will be kept strictly confidential. MEP will not share the data with anyone and will not identify individual responses in the evaluation report by name.

We can now start with by introducing everyone. Please mention your name, the district where you have worked for KPRP, and the component for which you were engaged.178

Thank you for your participation.

178 Before the start of the FGD, the note-taker will go around and record the following information for each participant: name, age, district of current residence, previous experience with current KPRP client (yes/no), KPRP component(s) for which contracted, number of sub-projects for which contracted, month and year when first contracted, and district(s) in which working with KPRP.

102

Questions Prompts General Question/Ice Breaker 1. As construction contractors working in Malakand,  Speed of reconstruction can you tell us how reconstruction is going on in  Remaining priorities and unfinished work this region?  Agencies involved (government, donors, NGOs) Specific Questions 2. What process did you go through before you were  Pre-qualification engaged to work with KPRP?  Open bidding  Tendering/selection process 3. What was your experience of the tendering/  Efficiency/timely decisions selection process of KPRP?  Fairness  Transparency  Better/worse than other tendering experiences 4. Once you were selected and contracted by KPRP,  Mobilization advance/financial arrangements what made it easy or difficult for you to start  Availability of workers working on the scheme for which you were  Availability of materials contracted?  Availability of sub-contractors  Role of local communities 5. What has been your experience so far in the  Progress has been rapid/slow construction process?  Payments have been timely/delayed  Supervision has been tight/lax  Sub-contractors have/have not worked as expected 6. Which of these organizations have you dealt with  Supervision and technical guidance during construction – the client organization;  Payment and financial matters CRISP, AGES or AiD; and the local administration?  Dispute resolution For what purposes? 7. How do you compare KPRP with your other  Efficiency/timeliness experiences as contractors?  Supervision and quality control  Size of schemes/sub-projects  Design of schemes/sub-projects  New techniques of construction  Performance of sub-contractors 8. Considering your experiences in KPRP and other  In terms of efficiency/speed of work projects, how do you think KPRP can make your  In terms of quality of work work better?

103

Khyber Pakhtunkhwa Reconstruction Program (KPRP) Mid-term Performance Evaluation

Focus Group Discussion (FGD) Guide – Number 2 For Mothers and Fathers of Children Enrolled in KPRP-assisted Schools

FGD Code: FGD nn

Main Topic of Discussion Evaluation Question Number 2 Date of FGD Venue Village/City Group Mothers/Fathers Name and Position of Moderator Name and Position of Note-taker

Introduction

The main subject of this discussion is how the schools reconstructed with KPRP assistance have affected you and your children. In this connection, we would like to talk about how you and your children were affected when the schools were destroyed, how you feel about the new schools, and how the new schools have affected your thinking about sending your daughters and sons to school.

Confidentiality

We can now start with by introducing everyone. Please mention your name and the district in which you are currently residing.179

Thank you for your participation.

179 Before the start of the FGD, the note-taker will go around and record the following information for each participant: name, age, district of current residence, number of girls and boys currently in school, and if the family had to leave their village/town during the militancy or counter- insurgency (yes/no).

104

Questions Prompts

General Question/Ice Breaker 1. Can you tell us please what you think of the state of  State of reconstruction education in your district and what is influencing it?  Trends in enrollment (both girls and boys)  Role of teachers  Role of government Specific Questions 2. What kinds of changes did you and your children  Dislocation experience just after your children’s school was  Loss of income or livelihood destroyed?  Effect on children’s education  Mental stress 3. How and to what extent did you deal with these  Moved to camps problems?  Moved in with friends/relatives in other locations  Remained at home  Continued/discontinued children’s education  Got help from government and other agencies 4. What were the difficulties encountered by you and  School building incomplete or non-existent your children after you returned to your home and  Lack of furniture, water and other facilities before the school in your community was  Distance to alternative school completed?  Cost of schooling  Lack of teachers 5. How and to what extent did you continue your  Makeshift arrangements in the under-construction children’s education before the school in your school community was reconstructed?  Temporary shelter outside the school  Another government school  Private school  Differences in treatment of girls and boys 6. To what extent and why have changes in the last 2-3  Steps taken by the government years affected your willingness to send your  Reconstruction of school children to the school constructed with KPRP  Features of new school assistance?  Differences in treatment of girls and boys 7. How do you compare the features of the old  Classrooms (number, size, lighting, ventilation, etc.) (destroyed) school of your community with the new  Other physical infrastructure (toilets, boundary school constructed with KPRP assistance? walls, etc.)  Water (for drinking and washing) 8. To what extent, if any, have the new features of the  Classrooms (number, size, lighting, ventilation, etc.) KPRP reconstructed schools affected enrollment,  Other physical infrastructure (toilets, boundary drop out, and attendance in the school serving your walls, etc.) community? Why or why not?  Water (for drinking and washing) 9. We have read reports of certain deficiencies and  Faster reconstruction would like to know what KPRP could have done  Larger building, more rooms differently in reconstructing the schools.  Better building and physical facilities (toilets, boundary walls, etc.)  Availability of water (for drinking and washing)

105

Khyber Pakhtunkhwa Reconstruction Program (KPRP) Mid-term Performance Evaluation

Focus Group Discussion (FGD) Guide – Number 3 For Male and Female Head Teachers of KPRP-assisted Schools

FGD Code: FGD nn

Main Topic of Discussion Evaluation Question Number 2 Date of FGD Venue City Group Male/Female Name and Position of Moderator Name and Position of Note-taker

Introduction

The main subject of this discussion is how the schools reconstructed with KPRP assistance have affected children of school-going age and their parents in and around the school where you teach. In this connection, we would like to talk about how children and parents were affected when the schools were destroyed, what changes you have seen in the new schools, and how parents and children feel about the new schools.

Confidentiality

We can now start with by introducing everyone. Please mention your name and the district where you are teaching.180

Thank you for your participation.

180 Before the start of the FGD, the note-taker will go around and record the following information for each participant: name, age, position in school (head teacher, senior teacher, other), district where teaching, level at which teaching (primary, middle, high school) and whether present in the same school before it was destroyed (yes/no).

106

Questions Prompts

Specific Questions 2. What kinds of changes did you and the children you  Dislocation were teaching experience just after your school was  Loss of income or livelihood destroyed?  Effect on children’s education  Mental stress 3. How and to what extent did the children and their  Moved to camps families deal with these problems?  Moved in with friends/relatives in other locations  Remained at home  Continued/discontinued children’s education  Got help from government and other agencies 4. What were the difficulties encountered by the  School building incomplete or non-existent children and their families after they returned to  Lack of furniture, water and other facilities their homes and before the school in your  Distance to alternative school community was completed?  Cost of schooling  Lack of teachers 5. How and to what extent did the children continue  Makeshift arrangements in the under-construction their education before the school in your community school was reconstructed?  Temporary shelter outside the school  Another government school  Private school  Differences in treatment of girls and boys 6. What kinds of changes have you observed in the last  Steps taken by the government 2-3 years that have affected parents’ willingness to  Reconstruction of school send their children to the school constructed with  Features of new school KPRP assistance?  Differences in treatment of girls and boys 7. How do you compare the features of the old  Classrooms (number, size, lighting, ventilation, etc.) (destroyed) school of your community with the new  Other physical infrastructure (toilets, boundary school constructed with KPRP assistance? walls, etc.)  Water (for drinking and washing) 8. To what extent, if any, have the new features of the  Classrooms (number, size, lighting, ventilation, etc.) KPRP reconstructed schools affected enrollment,  Other physical infrastructure (toilets, boundary drop out, and attendance in your school? Why or walls, etc.) why not?  Water (for drinking and washing) 10. We have read reports of certain deficiencies and  Faster reconstruction would like to know what KPRP could have done  Larger building, more rooms differently in reconstructing the schools.  Better building and physical facilities (toilets, boundary walls, etc.)  Availability of water (for drinking and washing)

107

ANNEX IV: BIOS OF TEAM MEMBERS

Mr. Tariq Husain has worked in development for 32 years, including 25 years as a consultant. He has worked in most parts of Pakistan and in 18 other countries. Much of his experience revolves around conceptualizing initiatives, evaluating operations and results, and helping institutions adapt to change. High-level contributions include Pakistan’s national reports for the Social Summit (1995) and Rio+10 (2002); Pakistan’s first progress report on the Millennium Development Goals (2003); corporate evaluation policies (of the International Fund for Agricultural Development, IFAD, and the United Nations Development Program, UNDP); evaluation of IFAD’s 17-country program in Asia-Pacific and of its contributions to decentralization in Eastern Africa; assignments for the UNDP-assisted South Asia Poverty Alleviation Program; assistance to an 18-agency Working Group on Agriculture, Rural Development and Poverty to operationalize the One UN approach in Pakistan; advice to policy makers on local governance in Pakistan and other countries; and a review of ombudsman institutions in Pakistan (for the Asian Development Bank). Sector and thematic experience includes governance, institutional development, community mobilization, rural development, agricultural research and extension, water management, social forestry, community infrastructure, social sectors, environment, gender, microfinance and the informal economy. As Director of Evaluation in MEP since July 2011, Mr. Husain has led, managed or contributed to most of the evaluations undertaken by MEP at the request of USAID. Before entering consulting, Mr. Husain worked for six years with the Aga Khan Rural Support Program, heading its monitoring and evaluation function. He is the author of 15 international publications and more than 30 articles on development and governance in the print media. He obtained his bachelor’s degree from Columbia University, New York, and a master’s and doctorate in economics from the University of Chicago.

Aftab Ismail Khan is a full-time Evaluation Specialist in the MEP Evaluation Unit. He has more than 18 years of experience and expertise in project management, designing and conducting evaluations, strategic planning, process analysis, report writing, sustainable enterprise and livelihoods development, and poverty alleviation. He has served as: a Team Leader in the evaluation of the USAID Housing Uniform Assistance Subsidy Project (HUASP); a Housing Specialist with the National Disaster Management Authority (NDMA), Pakistan; a National Consultant for UNDP/Ministry of Environment, Pakistan; and an Advisor, Social Protection with the Earthquake Reconstruction and Rehabilitation Authority (ERRA), Pakistan. He holds a bachelor’s degree (equivalent) from Pepperdine University, Malibu, California and a master’s in Business Administration from Preston University, Wyoming, USA.

David Altus Garner is a strategic planner with more than three decades experience planning, designing, implementing, and evaluating projects in Asia, Africa, and the Middle East. He has worked in more than 20 countries, and has extensive experience inside host country departments and ministries. He has often worked in fragile states and conflict prone countries at various levels, including: (i) Cabinet-level advisor; (ii) extensive work with energy programs; (iii) various sub-national assignments; (iv) hands on work with agriculture and rural development at village levels; plus (v) substantial institutional development experience inside host country departments and ministries at national and sub-national levels. Education: Master’s in Public Administration, Harvard University. Undergraduate Degree: Political Science, University of California at Berkeley.

Ahmed Ali Khattak has more than 14 years of experience in the areas of strategic planning, monitoring and evaluation, and project design. He has good understanding of government and development sector in Pakistan with expertise in designing and monitoring projects in the areas of Infrastructure, Governance and Environment, Public Health Engineering, Irrigation, Sustainable Development, Education and Health. Mr. Khattak has worked as an M&E Specialist, Governance Support Project, Multi Donor Trust Fund, KP Province, as Manager M&E Research, Provincial Program Office, for the USAID funded Assessment and Strengthening Program (ASP), and as Director M&E, P&D Department, GOKP. He has also served as a visiting faculty at Iqra University, Abasyn Universities, and Institute of Management Sciences at Peshawar.

108

ANNEX V: ADDITIONAL INFORMATION FOR EVALUATION QUESTION 1

Needs Assessment and Identification of Sub-projects for KPRP-assisted Reconstruction

The needs assessment report that has relevance to KPRP is the “Pakistan North West Frontier Province and Federally Administered Tribal Areas Preliminary Damage and Needs Assessment, Immediate Restoration and Medium Term Reconstruction in Crisis Affected Areas,” prepared By Asian Development Bank and World Bank for Government of Pakistan, Islamabad, Pakistan November 2009 (referred to in this annex as DNA 2009). This report in its Preface (p. vii) states that in 2009, at the request of the Government of Pakistan (GoP), a team of experts led by the Asian Development Bank (ADB) and the World Bank (WB) conducted a preliminary Damage and Needs Assessment (DNA) of the districts of Buner, Lower and Upper Dir, Shangla, and Swat in the North West Frontier Province (NWFP), now called Khyber Pakhtunkhwa (KP), and two tribal agencies called Bajaur and Mohmand in the Federally Administered Tribal Areas (FATA) of Pakistan.

The report states (p. vii) that “the DNA was carried out in close coordination with the government, development partners, civil society, and other stakeholders active in the affected area. The DNA estimated the damage and reconstruction costs resulting from the crisis.” The report also mentions (p. 7) that “the DNA findings and reconstruction needs included promoting build-back-better solutions to immediate infrastructure and service delivery recovery improvement in specifications but not scope. However, addressing the pre-crisis development deficits is beyond the scope of the DNA.”

According to Annex 1 (education sector) of the report, 411 schools and facilities such as residences and hostels were “affected” by militancy, out of which 231 were “fully damaged” and 180 were “partially damaged.” (This evaluation, following its SOW, uses the word “destroyed” for “fully or completely damaged” and “damaged” for “partially damaged.”) A table in the annex, reproduced below with additional calculations, shows that 411 (8 percent) of the 5,250 schools in Malakand were “affected,” that is, either destroyed or damaged, of which 210 (51 percent) were girls’ schools and 236 (57 percent) were primary schools. The affected schools included 24 percent of the girls’ middle schools, 35 percent of the girls’ high and higher secondary schools, and 28 percent of the boys’ high and higher secondary schools.

TABLE 9: NUMBER OF AFFECTED SCHOOLS IN MALAKAND DIVISION, 2009

Boys’ Schools Girls’ Schools Total Affected Affected Affected Total Total Total School Level Number Percent Number Percent Number Percent Primary 3,079 98 1.5 1,309 138 10.5 4,388 236 5.4 Middle 335 29 8.7 172 42 24.4 507 71 14.0 High/Higher 269 74 27.5 86 30 35.0 355 104 29.3 Secondary Total 3,703 201 5.4 1,566 210 13.4 5,250 411 7.8 Source: “Pakistan North West Frontier Province and Federally Administered Tribal Areas Preliminary Damage and Needs Assessment, Immediate Restoration and Medium Term Reconstruction in Crisis Affected Areas,” prepared by Asian Development Bank and World Bank for Government of Pakistan, Islamabad, Pakistan, November 2009, Annex 1.

Note that the numbers reported in the DNA report included residences and hostels associated with the schools, whereas PaRRSA’s documents focus only on school buildings. “PaRRSA at a Glance” (p. 21) says that 273 schools in Malakand were “Reported,” 313 were “Verified,” 180 were “Fully Damaged,” 78 “Partially Damaged,” and 51 “Repaired.” It does not explain the meaning of “Reported,” “Verified,” and “Repaired.” The minutes of the second

109

and fourth meetings of the KPRP Project Steering Committee suggest that verification means the physical verification of the condition of a building by PaRRSA staff.181 A PaRRSA field officer explained to the evaluation team that physical verification took two months and led to the finding that some schools reported in the DNA as destroyed were actually in the damaged category.

KPRP assistance focused on reconstructing destroyed schools and not on repairing damaged schools. Reconstruction provides much greater opportunity for building-back-better than repair. For KPRP, the GOKP Elementary and Secondary Education Department (E&SED), in consultation with PaRRSA, identified 122 destroyed schools for reconstruction182, generally excluding schools with low enrollment and those located in far-flung areas where the cost of transportation was very high.183 The total number of schools selected for KPRP assistance was based on the funding made available by USAID/Pakistan.

“PaRRSA at a Glance” (p. 22) reports that, in 2012, 116 of the 180 destroyed schools in Malakand Division were being reconstructed with the assistance of USAID/Pakistan, 63 with the help of other donors and international non-governmental organizations, and one through a GOKP initiative called the Chief Minister’s Package. At the time of writing of this report, USAID/Pakistan had authorized funds for the reconstruction of 122 schools.

Annex 2 (health sector) of the DNA report says that 63 (29 percent) of the 217 health facilities in Malakand Division were destroyed or damaged; 19 destroyed and 44 damaged. Fourteen of the destroyed and 34 of the 44 damaged facilities were Basic Health Units (BHUs). Other damaged facilities included three secondary care facilities (civil hospitals) and seven small units (see Table 10).

TABLE 10: NUMBER OF AFFECTED HEALTH FACILITIES IN MALAKAND DIVISION, 2009

Number Type of Health Institution Destroyed Damaged Total Tertiary Care Hospital 0 0 0 Secondary Care – District Headquarters and Civil Hospitals 0 3 3 First Level Care Facilities (Rural Health Centers, Basic Health Units, and 14 34 48 Mother and Child Health Centers) Other Health Facilities (Dispensaries, First Aid Posts, etc.) 5 7 12 Total 19 44 63 Source: “Pakistan North West Frontier Province and Federally Administered Tribal Areas Preliminary Damage and Needs Assessment, Immediate Restoration and Medium Term Reconstruction in Crisis Affected Areas,” prepared by Asian Development Bank and World Bank for Government of Pakistan, Islamabad, Pakistan, November 2009, Annex 2.

Senior PaRRSA officials informed the evaluation team that PaRRSA staff carried out physical verification and found that the actual number of health facilities damaged and destroyed was less than the number reported in DNA 2009.184 Some of the damaged facilities had suffered minor damage, which the Department of Health repaired with its own available resources. The department identified 18 facilities for KPRP assistance according to its priorities and the availability of funds from KPRP and other sources, and these are the 18 that were included in KPRP.

181 Minutes of the second KPRP Project Steering Committee Meeting, held on June 4, 2010, and minutes of the fourth KPRP Project Steering Committee Meeting, held on May 14, 2012, Provincial Disaster Management Authority (PDMA)/ Provincial Reconstruction Rehabilitation & Settlement Authority (PaRRSA), Government of Khyber Pakhtunkhwa, Peshawar. 182 “Information Provided by PaRRSA and USAID/Pakistan to the Evaluation Team At a Meeting Held at PaRRSA, Peshawar, on June 5, 2014,” MEP evaluation team note. 183 Information provided to the evaluation team by the Chief Planning Officer of PaRRSA in an email dated May 21, 2014. 184 “Information Provided by PaRRSA and USAID/Pakistan to the Evaluation Team At a Meeting Held at PaRRSA, Peshawar, on June 5, 2014,” MEP evaluation team note.

110

According to PaRRSA’s tenth Quarterly Progress Report (July-September 2013, p. 13), KPRP assistance has focused on the reconstruction of five destroyed BHUs and rehabilitation of 13 damaged health facilities. The latter include 10 BHUs (five each in Lower Dir and Swat Districts), two civil dispensaries (in Swat) and one Mother and Child Health (MCH) Center (in Lower Dir).185

Annex 9 (water and sanitation sector) of DNA 2009 also provides information on the water supply schemes damaged in Malakand Division (see Table 11). These schemes are owned by two GOKP departments, the Local Government and Rural Development Department (LG&RDD) and the Public Health Engineering Department (PHED). The LG&RDD schemes are mainly hand pumps, open wells and protected springs, and the DNA counts each one of them as a scheme. All the PHED schemes are tubewell-based with distribution networks and household and/or street connections. DNA 2009 reported that 451 (30 percent) of the 1,508 schemes in Malakand Division were destroyed or damaged, including 111 LG&RDD schemes that were destroyed (no PHED schemes were destroyed).186 The damages were estimated to have affected 1.03 million people in five districts, approximately half of them in Swat District and 31 percent in Buner.

TABLE 11: NUMBER OF AFFECTED WATER SUPPLY AND SANITATION SCHEMES IN MALAKAND DIVISION, 2009

Total Number Affected Number Owner Department Number of Affected As Schemes Destroyed Damaged Total % of Total Local Government and Rural 580 111 189 300 51.7 Development Public Health Engineering 928 0 151 151 16.3 Department Total 1,508 111 340 451 29.9 Source: “Pakistan North West Frontier Province and Federally Administered Tribal Areas Preliminary Damage and Needs Assessment, Immediate Restoration and Medium Term Reconstruction in Crisis Affected Areas,” prepared by Asian Development Bank and World Bank for Government of Pakistan, Islamabad, Pakistan, November 2009, Annex 2.

Sub-project Site Selection

The sixth KPRP Quarterly Progress Report (QPR), July-September 2012, p. 13 says that “reconstruction of 5 schools was dropped due to different reasons.” “Of the five schools dropped from reconstruction, one was dropped because the name of school contained Dar-ul-Uloom, and USAID declined to fund reconstruction of this school. Reconstruction of the other four schools was not feasible either due to space limitation or the location of schools in inaccessible areas.” Senior PaRRSA and USAID/Pakistan officials informed the evaluation team that four of these schools were replaced with others.187 The fifth site actually had two schools, one of which was a Government Middle School (GMS) and the other a religious school, or Dar-ul-Uloom (GDU), which is owned by the GOKP Auqaf Department (which looks after religious properties), and not the Elementary and Secondary Education Department. Both were mentioned together in the documents as one entity, GMS-GDU. KPRP took up the reconstruction of the GMS but not the GDU.

The minutes of the sixth Project Steering Committee meeting (October 11, 2013, p. 2) say that “11 schools were to be dropped and replaced due to various reasons, i.e., land sliding, far flung areas where cost of the

185 Project Implementation Letter (PIL) Number 21, dated August 10, 2012, under Activity Agreement Number 391-014. 186 The DNA report (p. 84) explains that many more LG&RDD schemes were destroyed compared with PHED schemes because the former were “mostly single-component schemes and of small size.” 187 This and the remaining sentences of this paragraph are based on “Information Provided by PaRRSA and USAID/Pakistan to the Evaluation Team at a Meeting Held at PaRRSA, Peshawar, on June 5, 2014,” MEP evaluation team note.

111

transportation of material was too high and contractors were not interested to work.” Senior PaRRSA and USAID/Pakistan officials informed the evaluation team that these schools were replaced with others.188

Project Steering Committee

The GOKP Provincial Disaster Management Authority (PDMA)/ Provincial Reconstruction Rehabilitation & Settlement Authority (PaRRSA) issued a notification (Number PDMA/PaRRSA/NWFP/TF/2010) on January 26, 2010 to establish a Project Steering Committee for KPRP. The notification designated the Director General of the KP Provincial Disaster Management Authority (PDMA), who is also, ex officio, the Director General of PaRRSA, as the Chairman of the Project Steering Committee and the following as its other members:

 Secretary, Elementary and Secondary Education Department, GOKP;

 Secretary, Health Department, GOKP;

 Secretary, Public Health Engineering Department;

 Secretary, Communication and Works Department;

 Secretary, Local Government and Rural Development Department;

 Representative of USAID/Pakistan; and,

 Program Manager, PaRRSA, as the member/secretary of the Project Steering Committee. The notification established the following responsibilities for the Project Steering Committee:

 The Project Steering Committee shall be a decision making forum for reconstruction and rehabilitation activities related to USAID-funded projects.

 It shall provide the requisite facilitation and speed for smooth implementation.

 It shall monitor the progress of action plans.

 It shall give approvals required for smooth and timely implementation of activities.

 It shall meet for quarterly progress review of project activities, or as and when required.

 Approve re-appropriations as and when required.

 Any other issue as and when it arises. Special Arrangements for Sub-project Planning, Approval, and Implementation

Divisional Approval Committee

The Relief, Rehabilitation and Settlement Department, Government of Khyber Pakhtunkhwa, Revised Notification Number SO (RR&SD) Admn/1-2/2013, dated February 28, 2013, notified189 the revised constitution of the Divisional Approval Committee (DAC) for approving schemes costing up to Rs. 100 million. The committee is chaired by the Commissioner of the relevant division and includes the Deputy Commissioner of the relevant district and one representative each of the department concerned and PaRRSA. In KPRP, according to senior PaRRSA officials, it is relevant only for the Integrated Tehsil Complex, which costs more than Rs. 100 million.190

188 This information is based on the MEP evaluation team note mentioned above. 189 A notification is an official written announcement, in this case for a change in the composition of a committee. 190 “Information Provided by PaRRSA and USAID/Pakistan to the Evaluation Team at a Meeting Held at PaRRSA, Peshawar, on June 5, 2014,” MEP evaluation team note.

112

District Reconstruction and Rehabilitation Committees

The Relief, Rehabilitation and Settlement Department, Government of Khyber Pakhtunkhwa, Revised Notification Number SO (RR&SD) Admn/1-2/2013, dated February 28, 2013, notified the revised constitution of District Reconstruction and Rehabilitation Committees (DRRCs) for approving schemes costing up to Rs. 40 million. Each committee is chaired by the Deputy Commissioner of the relevant district and includes the District Officer (Finance and Planning) and one representative each of the department concerned and PaRRSA. In KPRP, according to senior PaRRSA officials, it is relevant for the reconstruction and rehabilitation of schools, health facilities and water supply and sanitation schemes, all of which cost less than Rs. 40 million each.191

Project Unit in the Communication and Works Department

The minutes of the third Project Steering Committee meeting (p. 2) held on March 12, 2011 report that “The Director General of…PaRRSA expressed concern at the disbursement” and that PaRRSA’s Chief of the Infrastructure Section informed the Steering Committee that the physical progress of reconstruction of schools was very slow and the C&W Department needed to accelerate the pace of work (p. 4 of the minutes). In May 2012, the minutes of the Steering Committee meeting (p. 4) note that “The Director General … said that … the provincial government had seriously viewed the slow progress of school reconstruction and a decision had been taken to continue with the dedicated lean execution mechanism in the C&W Department to ensure the timely completion of the project.”

From late-2011 to early-2012, high-level GOKP and USAID/Pakistan officials also expressed concern at the delays in implementing the schools component of KPRP and discussed possible remedies, leading finally to the establishment of a dedicated Project Unit in the C&W Department, based in Malakand. The Project Unit is funded by GOKP by means of a PC-I approved by GOKP. It started functioning in July 2012 and is funded through September 2014 at a total cost of Rs. 122.6 million (see Table 12).

TABLE 12: COST AND DURATION OF THE PROJECT UNIT IN THE C&W DEPARTMENT

Non- Salary Salary as salary Total Project Unit/PC-I Expenses Percent Period Expenses Cost (Rs. Phase of Total (Rs. (Rs. million) Cost million) million) Original, 11 months June 1, 2012-April 30, 2012 19.2 13.7 33.0 58 1st Revision, 22 months May 1, 2013-March 31, 2014 31.5 15.0 46.5 68 2nd Revision, 28 months April 1, 2014-Sept. 30, 2014 27.5 15.7 43.2 64 Total, 61 months June 1, 2012- Sept. 30, 2014 78.2 44.4 122.6 64 Source: Second Revised PC-I/ Detailed Cost Estimate of “Establishment of Project Unit for PaRRSA/USAID Projects in C&W Department”, Project Director, PaRRSA/USAID Projects Unit, C&W Department, Government of Khyber Pakhtunkhwa, Saidu Sharif, February 3, 2014, Section 8 (Project Cost Estimates) and annex on cost abstract.

Documents attached to the PC-I help explain the rationale for establishing the Project Unit and the concerns with and possible reasons for delayed implementation that led to its creation; relevant text is reproduced below verbatim.

191 ibid.

113

From Document 1 attached to the PC-I:

 “Initially the progress of project was not satisfactory due to various reasons, including devastating floods 2010, direct engaging of two consultants with conflict of interest and different payment mechanism to which C&W Department was not accustomed.”

 “Due to delay in project, the Chief Secretary Khyber Pakhtunkhwa intervened in Sept. 2011 and directed Secretary C&W to conduct monthly progress meeting along with Director General PDMA so as to expedite work progress. This resulted in positive improvement.”

 “Director [sic] USAID held meeting with Chief Secretary Khyber Pakhtunkhwa in January 2012, and advised to bring changes in the existing mechanism. The Secretary C&W accordingly requested for removal of one of the consultants which had unduly caused delay in the execution of project. The new measures substantially improved the progress of work, which is appreciated by all stakeholders, including USAID.”

 “It is pertinent to mention here, that the quality of work is of high standard and accepted as such by the client and sponsoring agency. The standards set are much higher in comparison to similar nature works by other agencies in the area.”

 “Since the unit targets were missed, therefore, USAID has once again shown their concern on timely completion of project, the Chief Secretary Khyber Pakhtunkhwa, accordingly convened a meeting on May 9, 2012, in his office, the following attended: Additional Chief Secretary, Secretary C&W Department, Secretary Relief, Director General PDMA.”

 “After detailed deliberation, the Chief Secretary Khyber Pakhtunkhwa has directed to establish a dedicated Unit within C&W Department for this project. The same has also been supported by USAID.” From Document 2 attached to the PC-I:

 “USAID through FARA 1, 2 & 3 is funding reconstruction of 115 schools in Swat and adjoining districts of Malakand Division. Unfortunately and due to various reasons the construction schedule could not be strictly followed by C&W Department, and USAID has been showing serious concerns over delays. The Chief Secretary held a meeting in January this year with Mr. Andrew B. Sisson, who showed concerns and was assured of speeding up the construction work by the Secretary C&W. In this meeting Mr. Sisson pointed at possible further grant of US$ 30 million subject to timely completion of ongoing schemes. USAID’s consultants closely work with C&W Department and report progress to their principals.”

 “The Chief Secretary held a detailed meeting with the Additional Chief Secretary, Secretary C&W Department, DG PaRRSA and the undersigned [Secretary, Relief, Rehabilitation and Settlement Department] on 9th May 2012, and on the same day, the undersigned with approval of the Chief Secretary apprised Mr. Simmons [sic. Sisson] through an email of the decisions taken. Today USAID has responded to that email through a letter …. Interestingly, this letter does not make a mention of the previously indicated amount for further assistance of US$ 30 million but reminds us about expiry of date of FARA I and II for 70 schools to be 31-12-2012.”

 “Now it is all up to C&W Department to concentrate its energies to put the unit on ground at the earliest and to show substantial progress by June 28 to reassure USAID of its ability to deliver, otherwise the informally promised amount of another tranche of US$ 30 million, still redeemable, may be jeopardized or lost.” The C&W Department issued a notification establishing the Project Unit after approval of the PC-I. This was Notification Number SOE/C&WD/14-4/2012, issued by the Secretary, C&W Department, Government of Khyber Pakhtunkhwa, Peshawar, dated June 28, 2012, for the “Establishment of Project Unit for PaRRSA/USAID Projects in C&W Department” at Saidu Sharif, Swat. It says that the Project Unit will start functioning with effect from July 1, 2012. It provides the following Terms of Reference for the Unit (the original text is reproduced verbatim):

 The unit will work directly under the supervision of Secretary C&W Department.

114

 The Project Director will exercise full administrative, technical and financial powers for procurement of works, their bidding, approvals, technical sanction, time extension, enhancement, etc. to minimize the time lapsed in such formalities by various offices to a great extent as is experienced at present contributing in delay towards implementation.

 Hiring of consultants for design, planning and detailed supervision of schemes.

 Timely preparation of DCE/PC-I by the consultants through expert and experienced staff.

 Processing of PC-Is for approval of schemes by the competent forum.

 Transparent tendering through pre-qualified contractors.

 Smooth implementation.

 Effective monitoring and submission of monthly/quarterly progress reports.

 Finalization of contractor agreement, negotiations with the pre-qualified contractors and consultants.

 Overall control of financial management by preparation/submission of accounts to the Accountant General as per instructions of Assignment Account on monthly basis.

 Handing/taking over of completed schemes.

 Field visits for better progress and surprise checks of contractors’ activities.

 Coordination and interaction with various government departments/organizations.

 Any other work/duty assigned by the higher-ups and implementation of their instructions in letter and spirit. Sub-project Planning and Implementation Process The evaluation team drafted a description of the planning and implementation process for the reconstruction and rehabilitation of schools, headworks and bridges under KPRP based on official documents, including Project Steering Committee minutes, and interviews with officials of the following organizations: PaRRSA head office, Peshawar; the PaRRSA Project Management Unit (PMU), Swat; the Project Unit (PU) of the C&W Department, Swat; FDRD; and AGES. The evaluation team shared the draft with relevant officials of PaRRSA on June 2, 2014, and they provided their feedback by June 4, 2014, which is reflected in the description given below.

Planning and Implementation Process for the Reconstruction of KPRP Schools

Planning Process: KPRP followed the routine government planning process but on a fast track basis. For examples, District Reconstruction and Rehabilitation Committees (DRRCs) described above were formed in the affected districts. The DRRCs approved the PC-Is for all 122 schools in a short time. The planning process for the reconstruction of schools included the following steps:

1. The Elementary and Secondary Education Department, GOKP, prepared a list of damaged and destroyed schools and submitted it to PaRRSA and the Deputy Commissioner (DC), Swat’s office.

2. PaRRSA verified the Elementary and Secondary Education Department’s list of damaged and destroyed schools and reassessed the damages by sending its own team to visit the sites.

3. The C&W Department prepared the PC-1s with the assistance of PaRRSA and the Education Department vetted the PC-Is. The PC-Is contained the designs and cost estimates for the schools. PaRRSA developed the designs or modules for the schools with the help of an independent consultant (AGES) and the cost estimates were based on composite schedule rates (CSR) 2009.

115

4. PaRRSA submitted the PC-Is through its Planning Cell for approval to the respective DRRC, chaired by the concerned DC. According with its mandate to approve projects costing less than Rs. 40 million, the DRRCs approved the PC-Is.

5. The tender/bid opening committee was formed, headed by the concerned DC, comprising of the representatives of the concerned department, C&W, Pakistan Army and Executive District Officer (EDO), Finance and Planning, in order to streamline tendering process. Implementation Process: The implementation process for the reconstruction of schools included tendering, contracting, mobilization of resources, supervision, monitoring, reporting, and payments and involved the following steps:

1. The C&W Department prepared and issued tendering documents by publically inviting bids from pre- qualified contractors.

2. Since very few pre-qualified contractors submitted tenders due to the poor law and order situation at the time, and the low rates permitted under government rules, the Executive Engineer (XEN) of C&W reissued the invitation for bids without restricting it to pre-qualified contractors.

3. The bids were received and the tender/bid opening committee opened and evaluated the bids. After approval from the concerned DC, the minutes of the proceedings of the committee were circulated amongst all committee members. Appropriate record keeping of withdrawals, cancellation and selection of bids was carried out to ensure transparency of the process.

4. The XEN awarded the contracts. The contracts were signed based on the CSR 2009 rates. 5. After the issuance of work orders, the contractors mobilized their machinery, equipment, and human resources; while the consultants (CRISP and AGES) established their offices at the district level.

6. Payments were made to the contractors on the basis of milestones completed by them. These milestones are agreed between PaRRSA and USAID/Pakistan for each type of module.

7. The supervisory consultants (CRISP) prepared the Interim Payment Certificates (IPCs) for the contractors after the completion of an activity, submitted it to the AGES Monitoring Coordinator for initial checking and onward to the Deputy Team Leader and Team Leader (AGES), who, after verification submitted it to the PU Project Director. The Project Director submitted the demand based on the IPCs to PaRRSA, which released the demanded amount to C&W for onward payment to contractors.

8. After completion of a school and before handing it over, the standard PC-IV form for the completed school was signed by the concerned District Education Department.192

9. The final payment was released to the contractors after rectification of minor defects if any. 10. In KPRP a multi-pronged technical oversight regime was put in place. The Project Steering Committee undertook oversight at the highest level. USAID hired CRISP and AGES as supervisory and monitoring consultants, respectively, to carry out the day to day supervision of work, monitoring, and maintenance of quality control, including, quality tests on materials, steel and concrete cylinders.

11. AGES as monitoring consultants had its own oversight mechanism through its engineers present at the headquarters as well as the PMU level in Swat. CRISP as the supervisory consultant provided overall technical support to C&W and contractors. The C&W department and PaRRSA also reviewed progress.

12. PaRRSA and AGES prepared progress reports on quarterly and monthly basis, respectively, and submitted them to USAID.

192 The Planning Commission Form-IV (PC-IV) is the project completion report template used for government projects.

116

Planning and Implementation Process for the Rehabilitation of Amandara and Munda Headworks

Planning Process:

1. After the 2010 floods, the GOKP Irrigation Department undertook a rapid assessment of the damages caused to the Amandara and Munda Headworks and constituted the Flood Damage and Restoration Department (FDRD).

2. The FDRD, Irrigation Department, GOKP carried out a Damage Needs Assessment to prepare a reconstruction and restoration plan for the damages to the irrigation infrastructure.

3. FDRD devised a two pronged strategy, which comprised of: (a) temporary measures for an immediate restoration of water flow into the lower and upper Swat canal systems; and (b) permanent rehabilitation of the headworks.

4. FDRD developed the PC-Is for the rehabilitation of the headworks and submitted them to the GOKP approval forum. The Secretary Irrigation gave the administrative approval for the schemes. Implementation Process: The procurement of consultant and contractors was carried out according to routine government procedures193 as follows:

1. FDRD prepared the terms of reference for hiring of consultants, floated the expression of interest, and notified the procurement evaluation/selection committee for hiring of consultants.

2. The committee first opened and ranked the technical proposals and then the financial proposals; 3. The committee after scrutinizing the bids and negotiating with the lowest bidder, selected a consulting firm to undertake the assignment and executed the contract;

4. For hiring of contractors, the consultants prepared the tender documents, shortlisted the contractors, evaluated the bids and awarded the contract;

5. The contract was awarded to the contractor based on the Pakistan Engineering Council (PEC) rules, which required the contractor to prepare the engineering design, carry out the procurement and execute the works. Planning and Implementation Process for the Reconstruction of Bridges

Planning Process: The GOKP assigned the reconstruction of major bridges project including Khwazakhela and Mingora bridges to Pakhtunkhwa Highways Authority (PKHA) for execution and completion. FDRD initiated the tendering process for hiring of contractor and consultants, preparation of detailed cost estimates, administrative approval of PC-1, and technical sanction from competent authorities.

Implementation Process:

1. For the selection of consultant, FDRD through the office of its Director General (DG) floated the Expression of Interest for hiring consultants in newspapers. 2. The DG, FDRD, constituted an evaluation committee for evaluation of technical and financial proposals. 3. The FDRD followed the Planning and Development (P&D) guidelines, “Selection of Consultants, 2001” for prequalification. According to the minutes of evaluation committee meeting, four consulting firms were prequalified and invited to submit their technical proposals. 4. Pre-qualification of consultants was based on a 100-point score (Experience and Standing 40 points, and personnel 60 points; the minimum points required for pre-qualification were 70).

193 NWFP Local Government Planning Manual, 2007.

117

5. The GOKP, follows the “NWFP Procurement of Goods, Services and Civil Works Rules, 2003” and the “Draft Rules of 2008”, for construction related procurement. 6. M/S Consulting Associates obtained the highest score and were awarded the contract. 7. For the selection of contractor, the GOKP Information Directorate, on the request of FDRD, published an advertisement for the prequalification of contractors. Profiles and other documents were opened in the office of DG FDRD in the presence of the evaluation committee, which evaluated each profile based on criteria defined for prequalification of contractors, and prepared a list of prequalified contractors. 8. The FDRD then sent a notice inviting tenders (NIT) from prequalified contractors to the GOKP Information Directorate for publication. 9. The evaluation committee publicly opened the sealed bids, readout loud the quoted rates, evaluated bids and determined the successful bidder on the basis of lowest quoted price. 10. Minutes of the evaluation committee meeting were prepared and shared with the members and work order was issued to successful contractor and a formal contract agreement was executed. Flow of Funds194

For the PaRRSA capacity building component, GOKP released Rs. 20 million from its own resources to PaRRSA. After that, PaRRSA spent available GOKP funds and then received reimbursement for eligible expenses from USAID/Pakistan. PaRRSA did not need to utilize any of the advance provided by USAID for KPRP for its capacity building.

The minutes of the sixth Project Steering Committee meeting (October 11, 2013) say that the GOKP Finance Department had “released Rs. 61 million (in advance of reimbursement purpose) to Health Department to start the activities on the ground.” This was for the “soft” part of the health component – the Malakand Health Recovery and Supplies Component – for which USAID/Pakistan and PaRRSA signed PIL Number 391-014-PIL-20, dated August 8, 2012, for fixed amount reimbursement. It was not for the reconstruction and rehabilitation of health facilities.

The advance of $4.965 million that PaRRSA received from USAID/Pakistan195 is used for the reconstruction and rehabilitation of schools, health facilities and water supply and sanitation (WATSAN) sub-projects. School reconstruction started first and the health and WATSAN components much later. PaRRSA did not use up the entire advance on schools but held back the amounts required for health and WATSAN until they were needed. PaRRSA gives the advance to the executing agency for each component – C&W for schools and health, the Public Health Engineering Department (PHED) for WATSAN in Buner, and the Tehsil Municipal Administration (TMA) for WATSAN in Swat. The executing agency gives monthly expenditure statements based on the actual work done by contractors to PaRRSA. When the executing agency is about to run out of funds, it sends a demand for additional funds to PaRRSA.

Reimbursement for all construction activities is based on the milestones for each activity. The relevant agreements between USAID/Pakistan and PaRRSA spell out the number of milestones and the fixed amounts reimbursable for the completion of each of these. The USAID monitoring and evaluation consultant (AGES) submits milestone completion reports to PaRRSA, which are sent on to USAID/Pakistan as vouchers on form SF 1034 for reimbursement, under the signatures of the Director General of PaRRSA. USAID/Pakistan asks AGES to certify the completion of the milestones in question and then approves reimbursement. Every quarter, PaRRSA reports total accruals to USAID/Pakistan in the prescribed manner. USAID/Pakistan recovers/adjusts part of the advance it provided to PaRRSA from the PaRRSA reimbursement claim.

194 Unless otherwise stated, this section is based on “Information Provided by PaRRSA and USAID/Pakistan to the Evaluation Team at a Meeting Held at PaRRSA, Peshawar, on June 5, 2014,” MEP evaluation team note. 195 This is the advance for “design and mobilization” USAID/Pakistan sub-obligated for KPRP under Implementation Letter Number MLK-02, dated March 26, 2010. It amounts to 13.8 percent of the $36 million obligated under Implementation Letter Number MLK-01, dated December 17, 2009, for four components, namely, school reconstruction, health, water and sanitation, and enhancing government capacity. MLK-01 is the first formal agreement between USAID/Pakistan and GOKP for KPRP.

118

ANNEX VI: ADDITIONAL INFORMATION FOR EVALUATION QUESTION 2

Media and Independent Reports on Education in Malakand Post-2009

1. Gloria Caleb, “A slow return to normalcy,” Op-ed in Dawn (daily newspaper), August 20, 2009 (http://www.paktalibanisation.com/?p=1616)

“The Talibs attacked our school thrice,” says Fazl Mohammad, principal of the high school in Qambar, formerly a Taliban stronghold, some 2 km from Mingora city, “but we did not lose heart.” Here he sits with his staff amidst the rubble under the clear blue sky with the Pakistani flag flying at full mast. Classes have resumed. His commitment to bringing normalcy to Swat is seen in his decision to restart education activities amid the debris as soon as the announcement to reopen schools was made.

Further on, a group of children are seen shaking hands with some soldiers. The atmosphere is relaxed and congenial. Some tents are seen at a distance. Outside Swat these would appear as refugee camps. But here, these are tent schools. “Running a school in tents is extremely difficult,” says Akhtar Hussain, acting principal of the Balogram Secondary School, “but we cannot afford to lose more time. It has been two years since this conflict began and since then education in this region has suffered considerably. In order to move ahead, it is important to make use of all the resources we have,” he asserts.

2. Shamim Shahid, “Attendance thin as schools reopen in Pakistan’s restive region,” Korea Herald, August 2, 2009 (http://www.koreaherald.com/common_prog/newsprint.php?ud=20090803000002&dt=2)

Schools and colleges in Pakistan’s Malakand Division, particularly in Swat district, resumed their activities on Saturday (August 1) after a break of several months amidst students’ thin attendance. Following the Pakistani government’s decision, the schools and other educational institutions opened on Saturday. Most of the educational institutions in Swat district were closed in mid-January when Taliban militants began blowing them up. On the first day of the reopening, the situation was surprising as well as depressing in Mingora…the situation in Government High School No. 1 [in] Mingora city was very miserable where the students and teachers restarted their educational activity with [the] National Anthem on the debris of the destroyed school. The principal of this old educational institution, Fazal Aziz, said the historical building was destroyed by the militants with explosive material on the night between January 19 and 20. Later, the government, according to Fazal Aziz, provided nine tents as alternative arrangement, which were … burnt by the militants on May 2. He said that after the January 19 incident, a large number of students left the school. But the displacement multiplied the miseries when most of the parents shifted to Peshawar and other peaceful areas of the country.

3. Associated Press of Pakistan, “Govt [sic] schools in Malakand to reopen tomorrow,” posted by AAJ News Archive (of AAJ Television), uploaded July 31, 2009 (http://www.aaj.tv/2009/07/govt-schools-in-malakand-to- reopen-tomorrow/)

The government educational institutions in Malakand division will reopen from tomorrow. The classes will be started in tent schools till [sic] the reconstruction of damaged schools for which international agencies are providing funds.

4. M. Ibrahim, “Swat school enrolment increases after Taliban’s downfall,” Central Asia Online, November 9, 2011 (http://centralasiaonline.com/en_GB/articles/caii/features/pakistan/main/2011/11/09/feature-01)

A visit to schools in the area [Matta, barely 15km from Mingora] showed students thronging to their courses and demonstrating keen interest. Girls, ages 4-16, had no problem sharing classrooms with boys and are delighted to resume their schooling.

119

Education is like a beacon for girls, said Zuhra Khanum, a seventh-grader in Gorrha Village. “Education is a must for our future life as it will help express ourselves in a better way,” said Khanum, who wants to become a doctor. Sharing classrooms with boys is not a problem, said Sapna, another seventh-grader in Khanum’s school.

5. Tahir Ali, “Swatis love their schools,” January 22, 2013, (http://tahirkatlang.wordpress.com/2013/01/22/education-in-swat-after-militancy-and-malala/)

Afzal Khan Lala, the famous local nationalist leader from Matta who remained in his hometown despite persistent attacks on him, seconded his thoughts. “In fact, the recognition and awards Malala received, has increased the urge for education and excellence among students.” he says.

The district education officer (female) Swat Dilshad Begum also says Malala is the pride of the area and has motivated others. “There is a renewed urge to get more education. The government has made education free up to the secondary level and is providing free books to students and stipends to girls. The KP government has given plenty of monetary benefits and incentives to all cadres of teachers. We are motivating and preparing them for the task through refresher courses. There cannot be a better opportunity. They should now ensure quality education to their students,” she says. The strength of students has increased. “From 74,904 female students in Swat in 2009, the number increased to around 118,594 students in 2011. Data for this year is being finalized and it says that the female enrolment has gone up.”

The Taliban had started their anti-girls education campaign in 2003 which continued for the next four years. They would announce and eulogize the female students on radio who gave up studying. So convincing was their appeal for the naive girls that in 2004, more than 200 girls of high school Charbagh asked for school leaving certificates and tore them there.

In May 2008, the Kabal GGHSS [Government Girls Higher Secondary School] was the first school to be destroyed.

[D]uring their peak days, the Taliban first asked girls to observe strict purdah (veil) on their way to school. Later, they banned girl education and ordered girls’ schools to be shut down by January 15, 2009, and threatened to attack the students and teachers who didn’t follow the edict.

Data on Enrollment and Schools from the Education Management Information System (EMIS)

TABLE 13: EIGHT EDUCATIONAL INDICATORS FOR GOVERNMENT SCHOOLS IN BUNER, LOWER DIR, AND SWAT, 2007-08 TO 2012-13

Indicator 2007-08 2008-09 2009-10 2010-11 2012-13

Enrollment (in hundred thousands) 6.2 6.0 6.2 7.1 7.8 Girls’ enrollment (in hundred thousands) 2.4 2.4 2.5 3.0 3.4 Number of schools (in thousands) 3.7 3.7 3.7 3.7 3.5 Number of girls’ schools (in thousands) 1.2 1.2 1.2 1.2 1.3 Number of classrooms (in thousands) 12.0 11.8 11.6 11.9 14.4 Enrollment per school (in hundreds) 1.7 1.6 1.6 1.9 2.2 Number of classrooms per school 3.2 3.3 3.2 3.2 4.1 Number of students per classroom (in tens) 5.2 4.8 5.1 5.7 5.4 Source: Education Management Information System, Elementary and Secondary Education Department, Government of Khyber Pakhtunkhwa, reports available at www.kpese.gov.pk.

120

TABLE 14: EIGHT EDUCATIONAL INDICATORS FOR GOVERNMENT PRIMARY SCHOOLS IN BUNER, LOWER DIR, AND SWAT, 2007-08 TO 2012-13

Indicator 2007-08 2008-09 2009-10 2010-11 2012-13 Enrollment in primary schools (in hundred thousands) 4.8 4.7 4.8 5.5 6.0 Girls’ enrollment in primary schools (in hundred thousands) 2.1 2.0 2.1 2.5 2.8 Number of primary schools (in thousands) 3.1 3.1 3.0 3.1 2.8 Number of girls’ primary schools (in thousands) 1.0 1.0 1.0 1.2 1.0 Number of classrooms in primary schools (in thousands) 8.5 8.4 8.3 8.5 10.0 Enrollment per primary school (in hundreds) 1.6 1.5 1.6 1.8 2.1 Number of classrooms per primary school 2.8 2.9 2.9 2.9 3.5 Number of students per primary school classroom (in tens) 5.7 5.0 5.4 6.5 6.0 Source: Education Management Information System, Elementary and Secondary Education Department, Government of Khyber Pakhtunkhwa, reports available at www.kpese.gov.pk.

TABLE 15: ENROLLMENT (IN ’000) IN GOVERNMENT SCHOOLS IN BUNER, LOWER DIR, AND SWAT, 2007-08 TO 2012-13

2007-08 2008-09 2009-10 2010-11 2012-13 Indicator % % % % No. No. No. No. No. Change Change Change Change Total enrollment 621 604 -3 623 3 710 14 780 10 Enrollment in primary 484 467 -4 481 3 550 14 596 8 schools Enrollment in higher 137 137 0 142 4 160 13 184 15 level schools Girls’ enrollment 245 238 -3 252 6 301 19 344 14 Girls’ enrollment in 206 199 -3 211 6 252 19 283 12 primary schools Girls’ enrollment in 39 39 0 41 5 49 20 61 24 higher level schools Boys’ enrollment 376 366 -3 371 1 409 10 436 7 Boys’ enrollment in 278 268 -4 270 1 299 11 313 5 primary schools Boys’ enrollment in 98 98 0 101 3 110 9 123 12 higher level schools Source: Education Management Information System, Elementary and Secondary Education Department, Government of Khyber Pakhtunkhwa, reports available at www.kpese.gov.pk.

121

TABLE 16: PERCENTAGE OF GOVERNMENT SCHOOLS IN BUNER, LOWER DIR, AND SWAT WITH BASIC FACILITIES, 2007-08 TO 2012-13

Percentage of Schools Facilities Available 2007-08 2008-09 2009-10 2010-11 2012-13 Electricity 53 56 55 59 64 Water 62 60 57 59 61 Latrine 78 78 76 80 87 Boundary Wall 69 71 71 74 79 Source: EMIS annual reports from 2007-08 to 2012-13 available at www.kpese.gov.pk.

Data on Enrollment Trends, 2006-2011, from the Pakistan Social and Living Standards Measurement Survey

All information in this section of the annex is based on data taken from the Pakistan Social and Living Standards Measurement Surveys, which are conducted by the Pakistan Bureau of Statistics, Islamabad, and available at: http://www.pbs.gov.pk/content/pakistan-social-and-living-standards-measurement. The bureau conducts these sample surveys at the district and national levels in alternate years throughout Pakistan.

Pakistan Social and Living Standards Measurement Survey data on the Gross Enrollment Rate (GER) for girls and boys of three age groups, corresponding to primary, middle, and matric levels196, are reported in Table 17 and illustrated in Figures 6-8. Definitions are provided in the notes below Table 17. The first reporting period is academic year 2006-2007, the second one is 2008-2009 (2009 was the height of conflict) and the third is 2010- 2011 (post-conflict). The most significant finding is that the GER for girls in all three age groups declined sharply in Swat between 2006-2007 and 2008-2009, and then increased to more than pre-conflict levels by 2010-2011 for the primary and matric level age groups and close to the pre-conflict level for the middle school age group. For the primary level age group, it came down from 74 percent in 2006-2007 to 54 percent in 2008-2009 and increased to 85 percent in 2010-2011. For the matric level, for the same three school years, it came down from 24 percent to 14 percent and then increased to 26 percent. The most significant finding for boys is that there was large drop in the GER in the matric-level age group in Swat, from 79 percent to 39 percent, and then a large increase to 69 percent.

196 The primary level is grades 1 to 5, middle is grades 6-8 and matric (short for matriculation) is grades 9 and 10.

122

TABLE 17: COMPARISON OF GROSS ENROLLMENT RATES1 FOR PRIMARY, MIDDLE, AND MATRIC LEVELS (ALL TYPES OF SCHOOLS) 2, 2006-2011

Buner Lower Dir Swat Age Group and Sex 2006- 2008- 2010- 2006- 2008- 2010- 2006- 2008- 2010- 07 09 11 07 09 11 07 09 11 Male 94 100 1081 91 98 97 96 93 98 Primary School Age Female 59 70 72 68 76 91 74 54 85 Group3 Total 77 87 91 80 88 95 86 73 92 Male 73 69 67 75 69 91 75 72 76 Middle School Age Female 38 17 16 27 28 44 37 22 34 Group4 Total 58 46 45 51 50 68 58 48 55 Male 63 70 49 48 59 97 79 39 69 Matric Level Age Female 38 8 13 22 38 37 24 14 26 Group5 Total 51 43 32 37 50 69 53 29 47 Notes: 1 The gross enrollment rate can be greater than 100 percent as a result of grade repetition and entry at ages younger or older than the typical age at that grade level. 2 All types of schools include private and government schools. 3 Gross enrollment rate in primary schools: [Number of children attending primary level (classes 1-5) divided by number of children aged 5-9 years in the population] multiplied by 100. Enrollment in katchi (the pre-primary school year) is excluded. 4 Gross enrollment rate in middle schools: [Number of children attending middle level (classes 6-8) divided by number of children aged 10-12 years in the population] multiplied by 100. 5 Gross enrollment rate in matric level: [Number of children attending matric level (classes 9-10) divided by number of children aged 13-14 years in the population] multiplied by 100.

FIGURE 8: GROSS ENROLLMENT RATES (GER) FOR ALL AGE GROUPS IN BUNER DISTRICT (ALL TYPES OF SCHOOLS), 2006-2011

2010-11 2008-09 2006-07

49 Matric Level - Boys 63 70

13 Matric Level - Girls 8 38 67 Middle Level - Boys 69 73 16 Middle Level - Girls 17 38 Primary Level - Boys 100 108

Age Age GroupSex and 94 72 Primary Level - Girls 59 70 0 10 20 30 40 50 60 70 80 90 100 110 GER (Percentage)

123

FIGURE 9: GROSS ENROLLMENT RATES (GER) FOR ALL AGE GROUPS IN LOWER DIR DISTRICT (ALL TYPES OF SCHOOLS), 2006-2011

2010-11 2008-09 2006-07

97 Matric Level - Boys 59 48

37 Matric Level - Girls 38 22 91 Middle Level - Boys 69 75 44 Middle Level - Girls 28 27 97 Primary Level - Boys 98 Age Age GroupSex and 91 91 Primary Level - Girls 76 68 0 10 20 30 40 50 60 70 80 90 100 GER (Percentage)

FIGURE 10: GROSS ENROLLMENT RATES (GER) FOR ALL AGE GROUPS IN SWAT DISTRICT (ALL TYPES OF SCHOOLS), 2006-2011

2010-11 2008-09 2006-07

69 Matric Level - Boys 39 79

26 Matric Level - Girls 14 24 76 Middle Level - Boys 72 75 34 Middle Level - Girls 22 37 98

Age Age GroupSex and Primary Level - Boys 93 96 85 Primary Level - Girls 54 74 0 10 20 30 40 50 60 70 80 90 100 GER (Percentage)

Impacts of Build-Back-Better on Enrollment

The objective of this analysis is to estimate the impact on enrollment associated with the “build-back-better” approach to rehabilitating/reconstructing damaged schools. Building back better means that rehabilitated/reconstructed schools will be larger (when necessary) than the schools they replace and will have electricity, latrines, water, and boundary walls.

124

Rigorously assessing impact requires comparing what happened under the build-back-better approach to what would have happened in the same schools had they remained in their pre-damaged state. However, we cannot simultaneously observe enrollment in an individual school when it has been rehabilitated/reconstructed and when it has not. The essence of impact assessment is to estimate the counterfactual, i.e., what would have happened without build-back-better.

Data

In this case, we do not have pre-damage data on the schools rehabilitated/reconstructed under KPRP. We do, however, have EMIS data on almost 4,000 schools in Buner, Dir Payan, and Swat. The data cover 3 academic years (2005-06, 2011-12, and 2012-13) and are disaggregated by school type (i.e., boys, girls, and mixed). The data include information on enrollment and the presence of electricity, latrines, water, and boundary walls. The data do not include information on the number of classrooms, another key component of building-back-better. With these data we can examine the impact on enrollment of changes in facilities, excluding number of classrooms, over the time period relevant to KPRP investments. The 2005-06 data represent pre-damage enrollment, and the 2012-13 data represent post-rehabilitation/reconstruction enrollment.

Table 18 summarizes the number of observations in the EMIS dataset for the 2 academic years relevant to this analysis, 2005-06 (pre-damage) and 2012-13 (post-rehabilitation/reconstruction).

TABLE 18: NUMBER OF OBSERVATIONS BY DISTRICT, SCHOOL TYPE, AND YEAR

Boys Girls Mixed Total District 2005-06 2012-13 2005-06 2012-13 2005-06 2012-13 2005-06 2012-13 Buner 161 170 118 189 408 424 687 783 Dir Payan 512 464 433 489 520 596 1,465 1,549 Swat 388 381 252 313 871 885 1,511 1,579 Total 1,061 1,015 803 991 1,799 1,905 3,663 3,911

The strategy for estimating the impact of improvements in facilities is to estimate changes in enrollment as a function of changes in the availability of facilities. To prepare the raw data for analysis, we took the following actions:

 Dropped observations on mixed-sex schools. One objective of the analysis is to estimate changes in enrollment by sex. It is not clear, however, how mixed-sex schools are managed, and anecdotal evidence introduced elsewhere in this report suggests that boys and girls shared classrooms and facilities. Sex- disaggregated estimates will be more easily interpretable if they represent single-sex schools. For the same reasons, we dropped records from schools that changed from single-sex to mixed-sex or vice versa.

 Dropped observations from the 2011-12 academic year in order to focus the analysis clearly on the pre- damage and post-rehabilitation/reconstruction periods.

 Identified and dropped outliers that probably represent errors in the data. We identified outliers by inspecting scatter plots of percentage change in enrollment between 2005-06 and 2012-13.

 Table 19 illustrates the distribution of percentage change in enrollment data, the threshold selected for identifying outliers, and the percentage of cases dropped as a result of applying the threshold.

125

TABLE 19: DISTRIBUTION OF PERCENTAGE CHANGE IN ENROLLMENT

School Number of Standard % Mean Median Min Max Threshold Type Observations Deviation Dropped Boys 2,671 47% 26% -96% 3,400% 129% 300% 1.91% Girls 2,398 149% 60% -97% 17,300% 504% 700% 3.04%

 Dropped records that did not contain data from both the 2005-06 and 2012-13 academic years. Without data from both years, it is not possible to observe changes in enrollment or facilities. Table 20 illustrates the results of data preparation on the number of observations available for the analysis.

TABLE 20: COMPOSITION OF SAMPLE

Original Dataset Modified Dataset Academic Year Academic Year School Type 2005-06 2011-12 2012-13 2005-06 2012-13 Boys 1,061 780 1,015 682 682 Girls 803 808 991 523 523 Mixed 1,799 1,844 1,905 1,230 1,230

Analysis

Visual inspection of the data shows substantial increases in enrollment, particularly in girls’ schools, and increases in the percentage of schools with each of the four facilities. Almost all girls’ schools reported having latrines and boundary walls IN 2005-06, so the change in these facilities for girls’ schools was not quite as dramatic as changes in the other facilities (Figure 11). Buner reported the greatest percentage change in enrollment for both boys’ and girls’ schools, and Swat reported the lowest.

126

FIGURE 11: CHANGES IN FACILITIES AND ENROLLMENT BY SCHOOL TYPE (2005- 06 TO 2012-13)

The analysis estimated the following equation for boys’ and girls’ schools separately.

∆푒푛푟표푙푙푚푒푛푡 = 훼 + 훽1∆푤푎푡푒푟 + 훽2 ∆푙푎푡푟푖푛푒푠 + 훽3∆푒푙푒푐푡푟푖푐푖푡푦 + 훽4∆푤푎푙푙푠 + 훽5∆퐵푢푛푒푟 + 훽6∆푆푤푎푡

In this equation, the symbol ∆ represents change in the associated variable between 2005-06 and 2012-13; the four facilities’ variables are coded “1” if the school had the facility and “0” otherwise; the dummy variables for Buner and Swat are coded “1” if the school was in the district and “0” otherwise.

The estimated coefficient α represents the underlying trend in enrollment between 2005-06 and 2012-13. The coefficients β represent the marginal effect of adding the associated facility. Table 21 reports the regression results. The p-value represents the statistical significance of the coefficient.

127

TABLE 21: REGRESSION RESULTS

Boys Girls Coefficient Coefficient (p-value) (p-value) 0.2621*** 0.4652*** Constant (0.0000) (0.0000) 0.0881* 0.0985* Indicator of adding electricity (0.0630) (0.1110) -0.0416 0.0695 Indicator of adding water (0.3830) (0.2740) 0.0878** 0.1419 Indicator of adding latrines (0.0340) (0.1730) 0.0683* -0.1507 Indicator of adding walls (0.1040) (0.3390) 0.1403** 0.2672** Indicator of Buner District (0.0110) (0.0010) 0.0309 -0.0295 Indicator of Swat District (0.4170) (0.6260) * Statistically significant with p<=0.10 ** Statistically significant with p<=0.05 *** Statistically significant with p<=0.01

The constant terms for both types of schools show large and statistically significant average increases in enrollment between 2005-06 and 2012-13: 26 percent for boys’ schools and 46 percent for girls’ schools. The coefficient for Buner District shows a significantly greater increase in enrollment in Buner than in other districts: 14 percent greater in boys’ schools and 27 percent greater in girls’ schools.

Findings with respect to the impacts on enrollment of changes in facilities relative to what it would have been otherwise include the following:

 Adding electricity increased enrollment by 9 percent in boys’ schools and 10 percent in girls’ schools.

 Adding latrines increased enrollment in boys’ schools by 9 percent.

 Adding boundary walls increased enrollment in boys’ schools by 7 percent.

 Adding electricity in girls’ schools increased enrollment by 10 percent.

 The lack of a significant effect of latrines and boundary walls in girls’ schools does not mean they are not important determinants of enrollment. Almost all girls’ schools reported having these facilities in 2005-06 (92 percent reported having latrines, and 97 percent boundary walls), so there was little room for improvement and thus few observations on changes. Limitations

The primary limitation of this analysis is the lack of data on number of classrooms. If changes in the number of classrooms are correlated with changes in other facilities, which is a reasonable assumption with KPRP improvements, then the estimated effects of the individual facilities on enrollment are biased.

128

Conclusions

Because we do not have data on pre-damage conditions in the KPRP-assisted schools, we do not know the specific changes in facilities that took place during reconstruction/ rehabilitation and can therefore not estimate the increase in enrollment in KPRP-assisted schools. On the basis of our analysis, we can conclude only that, if KPRP reconstruction/rehabilitation provided facilities that were not there before, the intervention would have increased enrollment in KPRP-assisted schools relative to what it would have been if the schools had been built back to their original specifications. The causal effects of changes in individual facilities, however, will not accurately reflect the effect of that facility if the change in the facility is correlated with a change in the number of classrooms (a variable omitted from the analysis due to lack of data).

129

ANNEX VII: ADDITIONAL INFORMATION FOR EVALUATION QUESTION 3

This annex presents information obtained by the evaluation team, and its findings, on one aspect of evaluation question 3, that is, whether GOKP with its own resources has replicated the PaRRSA-KPRP school design. “Own resources” means the GOKP’s Annual Development Plan (ADP), rather than any externally-funded initiative. As mentioned in the main report, the PaRRSA-KPRP designs consist of several modules for the academic, administrative and lavatory blocks, and the selection of a particular combination of modules depends on the site and requirements (particularly the level and enrollment) of the school in question. The points noted below summarize the information provided to the evaluation team by various sources between February 2 and April 17, 2014. The two findings emerging from this information are written in boldface.

1. Minutes of the Second Meeting of the Provincial Steering Committee of PaRRSA, February 2, 2010: “Chief Secretary desired that standard drawings and designs prepared by PaRRSA with the collaboration of C&W [Construction and Works Department], Education and Health etc., to be adopted in the rest of the province.”

2. PaRRSA “Monitoring & Evaluation Report – Schools Reconstructed in District Buner under USAID Supported KPRP,” May 2013, p. 6: “In Topi it was observed that the school funded from ADP [Annual Development Plan] and executed by C&W was somehow identical to the design PaRRSA has implemented.”

3. Reported by Mr. Waseem Kundi, Chief, Planning, PaRRSA, during the kick-off meeting of February 19, 2014, as recorded in the minutes of this meeting: “PaRRSA has commissioned the Pakistan Army to construct schools in the Bannu District of KP using the KPRP school building design but construction has not yet started.”

4. USAID official interviewed by the evaluation team, April 2014: In many cases, the GoKP adopted USAID funded schools’ modular designs and then replicated them in Government-funded schools.

5. PaRRSA official interviewed by the evaluation team, April 2014: The schools reconstructed under PaRRSA’s auspices are excellent and of the highest standards. These schools are being replicated by other donors.

6. Email of February 28, 2014, from Naseer Khan, Team Leader, AGES, to Tariq Husain, Director Evaluation, MEP, copied to Bahrobar, KPRP Program Manager, USAID/Pakistan: “We don’t know anything about this replication outside Malakand Division. None of these Five Schools [in which the KPRP design has been adopted] has been implemented through C&W and any other Government Department but they have been implemented directly by donor.” List of five donor-replicated schools provided by AGES is reproduced in the following table.

130

TABLE 22: LIST OF GOVERNMENT SCHOOLS IN MALAKAND WHERE THE KPRP MODULAR SCHOOL BUILDING DESIGN HAS BEEN REPLICATED

Distance Completion Status Location Funded Through from (Check as Appropriate) Name of (Including Replicated Saidu ADP School Name of Modules Under Sharif (Check if Donor Completed District) Construction (km) Applicable) Kanju Village M3 & M2 8 - UN- √ - Girls Kanju, HABITAT Primary Tehsil School Kabal, Swat Kuz Village T3 45 - IDEA - √ Shawar Salaind Kuz Girls Shawar, Middle Tehsil School Matta, Swat Bahadar Village T4 & LB 48 - Qatar √ - Banda Bahadar Charity Girls Banda, Primary Tehsil School Matta, Swat Tootkai Village T3 25 - Qatar √ - Girls Tootkai, Charity Primary Tehsil School Babozai, Swat Chamtalai Village T3 40 - Save The √ - Girls Chamtalia, Children Middle Tehsil School Khwaza Khela, Swat

7. Advised by Bahrobar, KPRP Program Manager, USAID/Pakistan, in an email to Tariq Husain, Director Evaluation, MEP, dated April 21, 2014, copied to several others in USAID and MEP: Visit Government Higher Secondary School (GHSS) Koza Bandai, Government High School (GHS) Ningolai and GHSS Bar Shawar to “see if the designs have similarity with the designs of schools undertaken by KPRP.” Clarification in a follow-up email the same day: “All these three schools in Swat are ADP Schemes constructed by C&W, to the best of my knowledge.”

8. Information obtained by Ahmed Ali Khattak, Civil Engineer/Project Management Specialist, MEP KPRP Evaluation Team, on April 22, 2014:

 Reported by Incharge, PaRRSA PMU, Saidu Sharif, and Infrastructure Coordinator, PaRRSA PMU, Saidu Sharif: GHSS Koza Bandai has not followed any of USAID’s modules and design.

 Reported by Incharge, PaRRSA PMU, Saidu Sharif, and Infrastructure Coordinator, PaRRSA PMU, Saidu Sharif: GHS Ningolai has been funded by UAE and has not replicated PaRRSA’s modules.

 Reported by Deputy Director, Assistant Director and Sub-divisional Officer, C&W Buildings Division, Swat, Matta: GHSS Bar Shawar is an “upgradation” from secondary school to higher secondary school and might have followed the KPRP schools’ architecture (but not the design), but there is no supporting documentary evidence available with the C&W Buildings Division, Swat.

131

9. On April 25, Ahmed Ali Khattak and the Infrastructure Coordinator of the C&W Project Unit visited GHSS Bar Shawar and observed the following:

 There are three framed structures – one Government Primary School (GPS), one Government Middle School (GMS) and one GHSS – at different stages of construction. The primary and middle schools are being constructed by C&W with KPRP assistance, while the GHSS is being built by C&W with government (Annual Development Plan) funds. The GPS is 95-98 percent complete and has not yet been handed over to the Education Department.

 From an architectural point of view, the GHSS looks similar to the two KPRP-assisted school structures. The Infrastructure Coordinator thought that it looked like the T3 module of KPRP, except that the T3 module has two rooms on the ground floor and the GHSS has three.

 There are more columns in the GHSS than in the GMS, and they are more closely and not equally spaced. There is more steel reinforcement in the GMS columns than in GHSS.

 The beams are thicker in the GHSS then they are in the KPRP-assisted GMS.

 There are five windows in each room of the GHSS and three in each GMS room. Finding: The MEP evaluation team did not find any evidence that GOKP line departments had replicated the PaRRSA-KPRP school design with government funds by December 2013, the evaluation cut-off date.

On April 17, 2014, the GOKP Provincial Development Working Party, the highest project-approval forum in the province, approved the PC-I197 for a project of the Elementary and Secondary Education Department for the construction of 100 primary schools in the province, in which the department, following a recommendation of the core working group on education reform established by GOKP, had recommended replication of a six-room primary school design along the lines of some of the PaRRSA-KPRP modules. The modules selected for replication are the M3 module for the academic block, combined with the M2 module for the administrative block and the lavatory block (with development works). As the following table shows, the GOKP schools’ administrative block, however, is 61 percent the size of the PaRRSA-KPRP M2 module in terms of covered area, and the GOKP schools’ lavatory block is 30 percent larger. The reason for the latter is that the lavatory block includes a toilet for special children, a requirement that PaRRSA-KPRP also introduced during project implementation but is not reflected in the design information, which is taken from a KPRP document that dates back to 2011.

197 “130334 – Establishment of 100 Primary Schools (B&G) on need basis in Khyber Pakhtunkhwa (Phase-III) ADP No. 12, 2013-14 to 2015- 16,” Elementary and Secondary Education Department, GOKP, February 2014.

132

TABLE 23: DESIGN AND COST COMPARISONS BETWEEN THE PARRSA-KPRP AND GOVERNMENT-ADOPTED PRIMARY SCHOOL MODULES

KPRP GOKP Module Features and Cost December 2011 February 2014 Academic Block:  Module number M3 M3  Number of floors Two Two  Number of rooms Six Six  Covered area (square feet) 4,838.5 4,838.5  Roof Reinforced concrete Reinforced concrete  Staircase One One  Cost (in Pakistani rupees) including 10,545,840 9,568,714 contingencies [9% less] Administrative Block:  Module number M2 M2  Number of floors One One  Number of rooms Two One  Covered area (square feet) 802.4 491.3  Roof Reinforced concrete Reinforced concrete  Cost (in Pakistani rupees) including 2,329,380 1,178,109 contingencies [49% less] Lavatory Block with Development Works:  Covered area (square feet) 347.3 452  Cost (in Pakistani rupees) including 3,227,310 3,116,116 contingencies [3% less] Total:  Covered area (square feet) 5,988.2 5,781.8  Cost (in Pakistani rupees) including 16,102,530 13,862,939 contingencies [14% less] Source for PaRRSA-KPRP: Amendment Number 1, Modified Fixed Amount Reimbursement Agreement for Reconstruction of Four- and Six-Room Module Schools in the Malakand Division, FAR Agreement Number 391-MLK-FARA-002-01, dated December 15, 2011. The cost data in this document are in United States dollars. They have been converted to Pakistani rupees in this table at the rate of 90 rupees per dollar, the rate that prevailed in December 2011 (source: United States dollar (USD) Pakistani rupee (PKR) Year 2011 Exchange Rate History – Yahoo Finance). Source for GOKP: Approved PC-I “1330334 – Establishment of 100 Primary Schools (B&G) on need basis in Khyber Pakhtunkhwa (Phase-III) ADP No. 12, 2013-14 to 2015-16,” Elementary and Secondary Education Department, GOKP, February 2014.

133

Annex II of the approved PC-I notes that project cost would increase more than three-fold as the number of classrooms per school is increased from the previously proposed two to six, but that substantial savings were expected in the current year budget that allowed the revised project proposal to go forward.

Finding: In April 2014, GOKP adopted a slightly modified PaRRSA-KPRP school design for constructing 100 primary schools with government funds throughout the province during 2014- 2016. This decision was considered feasible particularly in view of the savings in the current year budget.

134

ANNEX VIII: DISCLOSURE OF ANY CONFLICTS OF INTEREST

Name Tariq Husain Title Director, Evaluation Organization Management Systems International (MSI) Evaluation Position? Team Leader Team member  Evaluation Award Number (contract or other instrument) USAID Project(s) Evaluated Khyber Pakhtunkhwa Reconstruction Program (KPRP) (Include project name(s), implementer name(s) and award number(s), if applicable) I have real or potential conflicts of Yes No  interest to disclose. If yes answered above, I disclose the following facts: Real or potential conflicts of interest may include, but are not limited to: 1. Close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated. 2. Financial interest that is direct, or is significant though indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation. 3. Current or previous direct or significant though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project. 4. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated. 5. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated. 6. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated that could bias the evaluation. I certify (1) that I have completed this disclosure form fully and to the best of my ability and (2) that I will update this disclosure form promptly if relevant circumstances change. If I gain access to proprietary information of other companies, then I agree to protect their information from unauthorized use or disclosure for as long as it remains proprietary and refrain from using the information for any purpose other than that for which it was furnished. Signature

Date May 03, 2014

135

Name Aftab Ismail Khan Title Evaluation Specialist Organization Management Systems International (MSI) Evaluation Position? Team Leader Team member  Evaluation Award Number (contract or other instrument) USAID Project(s) Evaluated Khyber Pakhtunkhwa Reconstruction Program (KPRP) (Include project name(s), implementer name(s) and award number(s), if applicable) I have real or potential conflicts of Yes No  interest to disclose. If yes answered above, I disclose the following facts: Real or potential conflicts of interest may include, but are not limited to: 1. Close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated. 2. Financial interest that is direct, or is significant though indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation. 3. Current or previous direct or significant though indirect experience with the project(s)

being evaluated, including involvement in the project design or previous iterations of the project. 4. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated. 5. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated. 6. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated that could bias the evaluation. I certify (1) that I have completed this disclosure form fully and to the best of my ability and (2) that I will update this disclosure form promptly if relevant circumstances change. If I gain access to proprietary information of other companies, then I agree to protect their information from unauthorized use or disclosure for as long as it remains proprietary and refrain from using the information for any purpose other than that for which it was furnished. Signature

Date May 22, 2014

136

Name David Garner Title STTA, KPRP. Organization Management Systems International (MSI) Evaluation Position? Co-Team Leader Team member x Evaluation Award Number (contract or other instrument) USAID Project(s) Evaluated (Include project name(s), implementer Khyber Pakhtunkhwa Reconstruction Program (KPRP) name(s) and award number(s), if applicable) I have real or potential conflicts of Yes No x interest to disclose. If yes answered above, I disclose the following facts: Real or potential conflicts of interest may include, but are not limited to: 1. Close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated. 2. Financial interest that is direct, or is significant though indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation. 3. Current or previous direct or significant though indirect experience with the project(s) being

evaluated, including involvement in the project design or previous iterations of the project. 4. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated. 5. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated. 6. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated that could bias the evaluation.

137

Name Ahmed Ali Khattak Title STTA, Civil Engineer, Project Management Specialist Organization Management Systems International (MSI) Evaluation Position? Team Leader Team member X Evaluation Award Number (contract or other instrument) USAID Project(s) Evaluated Khyber Pakhtunkhwa Reconstruction Program (KPRP) (Include project name(s), implementer name(s) and award number(s), if applicable) I have real or potential conflicts of Yes No X interest to disclose. If yes answered above, I disclose the following facts: Real or potential conflicts of interest may include, but are not limited to: 7. Close family member who is an employee of the USAID operating unit managing the project(s) being evaluated or the implementing organization(s) whose project(s) are being evaluated. 8. Financial interest that is direct, or is significant though indirect, in the implementing organization(s) whose projects are being evaluated or in the outcome of the evaluation. 9. Current or previous direct or significant though indirect experience with the project(s) being evaluated, including involvement in the project design or previous iterations of the project. 10. Current or previous work experience or seeking employment with the USAID operating unit managing the evaluation or the implementing organization(s) whose project(s) are being evaluated. 11. Current or previous work experience with an organization that may be seen as an industry competitor with the implementing organization(s) whose project(s) are being evaluated. 12. Preconceived ideas toward individuals, groups, organizations, or objectives of the particular projects and organizations being evaluated that could bias the evaluation. I certify (1) that I have completed this disclosure form fully and to the best of my ability and (2) that I will update this disclosure form promptly if relevant circumstances change. If I gain access to proprietary information of other companies, then I agree to protect their information from unauthorized use or disclosure for as long as it remains proprietary and refrain from using the information for any purpose other than that for which it was furnished.

138

U.S. Agency for International Development 1300 Pennsylvania Avenue, NW Washington, DC 20523

139