ASIAN DEVELOPMENT BANK Independent Evaluation Department PROJECT PERFORMANCE EVALUATION REPORT ON KAZAKHSTAN AND THE KYRGYZ REPUBLIC: ALMATY-BISHKEK REGIONAL ROAD REHABILITATION PROJECT In this electronic file, the report is followed by Management’s response, and the Board of Directors’ Development Effectiveness Committee (DEC) Chair’s summary of a discussion of the report by DEC. Performance Evaluation Report Project Numbers: 29568 and 32463 Loan Numbers: 1774 and 1775 Project Performance Evaluation Report (Joint Report) March 2009 Kazakhstan and the Kyrgyz Republic: Almaty– Bishkek Regional Road Rehabilitation Project This joint evaluation report was prepared by the Independent Evaluation Department of the Asian Development Bank and the Evaluation Department of the European Bank for Reconstruction and Development. CURRENCY EQUIVALENTS Asian Development Bank Currency Unit (Kazakhstan) – tenge (T) At Appraisal At Project Completion At Operations Evaluation (August 2000) (October 2007) (August 2008) T1.00 = $0.0070 $0.0082 $0.0084 $1.00 = T142.400 T120.855 T119.680 Currency Unit (Kyrgyz Republic) – som (Som) At Appraisal At Project Completion At Operations Evaluation (August 2000) (October 2007) (August 2008) Som1.00 = $0.0208 $0.02895 $0.0289 $1.00 = Som47.990 Som34.540 Som34.560 European Bank for Reconstruction and Development Currency Unit (Kazakhstan) – tenge (KZT) At Appraisal (October 2000) $1 = €1.17 $1 = KZT (tenge)144 ABBREVIATIONS ADB – Asian Development Bank BME – benefit monitoring and evaluation CAREC – Central Asia Regional Economic Cooperation CBA – cross-border agreement EBRD – European Bank for Reconstruction and Development EIRR – economic internal rate of return EvD – Evaluation Department (EBRD) HGV – heavy goods vehicle IBRD – International Bank for Reconstruction and Development IED – Independent Evaluation Department (ADB) IRI – international roughness index JER – joint evaluation report km – kilometer MOTC – Ministry of Transport and Communications OEM – Operations Evaluation Mission PCR – project completion report (ADB) PIU – project implementation unit PRC – People's Republic of China RSDP – road sector development program SDR – special drawing rights TA – technical assistance TRACECA – Transport Corridor Europe–Caucasus–Asia VOC – vehicle operating cost vpd – vehicle per day NOTE In this report, “$” refers to US dollars and € refers to euros. Key Words kazakhstan, kyrgyz republic, almaty, bishkek, adb, asian development bank, development effectiveness, ebrd, european bank for reconstruction and development, road, roads maintenance, performance evaluation, transport, infrastructure, joint evaluation Director ADB: R.B. Adhikari, Independent Evaluation Department (IED), Independent Evaluation Division 1 EBRD: Fredrik Korfker, Chief Evaluator, Evaluation Department (EvD), European Bank for Reconstruction and Development Team Leaders ADB: N. Singru, Senior Evaluation Specialist, Independent Evaluation Division 2, IED EBRD: W. Gruber, Senior Evaluation Manager, EvD Team Members ADB: R. Lumain, Senior Evaluation Officer, Independent Evaluation Division 2, IED ADB: C. Roldan, Senior Operations Evaluation Assistant, Independent Evaluation Division 2, IED EBRD: A. Gräfin zu Eulenburg, Principal Evaluation Manager, EvD Independent Evaluation Department, PE-722 In preparing any country program or strategy, financing any project, or by making any designation of or reference to a particular territory or geographic area in this document, the Asian Development Bank does not intend to make any judgments as to the legal or other status of any territory or area. CONTENTS Page PREFACE i BASIC DATA iii EXECUTIVE SUMMARY vii MAP I. INTRODUCTION 1 A. Background 1 B. Project Preparation 1 II. DESIGN AND IMPLEMENTATION 2 A. Rationale 2 B. Cost and Financing Arrangements 3 C. Executing Arrangements and Scheduling 4 D. Change in Project Design 5 E. Loan Covenants and Technical Assistance 6 III. ASSESSMENT OF PROJECT OUTPUTS AND PERFORMANCE 7 A. Outputs 7 B. Performance of Contractors and Supervising Consultants 8 C. Performance of the Beneficiary Countries 9 D. Performance of the Multilateral Development Banks 10 IV. ASSESSMENT OF PROJECT OUTCOMES AND IMPACTS 12 A. Overall Assessment 12 B. Relevance 13 C. Effectiveness 14 D. Efficiency 15 E. Sustainability 16 F. Impacts 18 V. ISSUES, LESSONS, AND FOLLOW-UP ACTIONS 24 A. Border Infrastructure 24 B. Lack of Capacity of Implementing Institutions 24 C. Adequacy of Loan Covenants 25 D. Importance of Sufficient Maintenance Allocations 25 E. Project Financing 25 F. Follow-Up Actions 26 In accordance with guidelines formally adopted by the Independent Evaluation Department (IED) of the Asian Development Bank to avoid conflict of interest in its independent evaluations, the IED director general did not review the report and delegated approval of this evaluation to the director of the Independent Evaluation Division 1. Venera Esenalieva and Svetlana Shakirova were the consultants appointed by ADB. John Parry was the consultant appointed by EBRD. To the knowledge of the managements of IED and EvD, the individuals preparing, reviewing, or approving this report had no conflicts of interest. APPENDIXES 1. Design and Monitoring Framework 28 2. Project Cost 31 3. Status of Compliance with Loan Covenants 33 4. Performance of the Asian Development Bank Technical Assistance 43 5. Summary of Physical Accomplishments 44 6. Summary of Road Condition 45 7. Transition Impact Analysis 48 8. Time Taken to Cross the Border 50 9. Economic Analysis 51 10. Impact on Road Safety 65 11. Photographs of the Rehabilitated Regional Road 67 12. Lessons from the Joint Evaluation Exercise 71 Attachments: Management Response DEC Chair Summary PREFACE The Asian Development Bank (ADB) and European Bank for Reconstruction and Development (EBRD) provided parallel financing for the Almaty–Bishkek Regional Road Rehabilitation Project. In the spirit of fostering harmonization among multilateral development banks and to develop synergies in development evaluation practices, the two organizations agreed to carry out a joint evaluation of the Project. In 2007, the Operations Evaluation Department (now the Independent Evaluation Department [IED]) of ADB and the Evaluation Department (EvD) of EBRD entered into a memorandum of understanding to work together to prepare the joint evaluation report (JER). This joint evaluation also contributes to the harmonization agenda under the Paris Declaration of 2005 and is in line with aspirations of the Evaluation Cooperation Group. The memorandum of understanding outlines the methodology employed and the process envisioned. The joint evaluation follows the evaluation criteria developed by the Evaluation Network of the Development Assistance Committee of the Organisation for Economic Co-operation and Development; these criteria are also consistent with the ADB and EBRD evaluation guidelines and policies. However, certain deviations reflect differences in mandates, institutional practices, and history of ADB and EBRD. The JER integrates the evaluation findings and ratings of EvD and IED to provide an overall rating for the Project. The JER is based on surveys and data collection activities carried out jointly by consultants appointed by EvD and IED during June–August 2008. Following this data collection, an EvD and IED joint evaluation mission visited Kazakhstan and the Kyrgyz Republic during August–September 2008. The outputs of these activities were included in the preparation of this JER, which intends to serve the evaluation requirements of the respective organizations. The following figure provides a description of the joint evaluation process. The evaluation was conducted within 2 years following project completion as (i) most of the project highway was completed in 2006, providing sufficient time for the results to be visible in 2008; and (ii) traffic data collected by government agencies in 2007, which provided the basis for the project completion report, needed to be updated independently to understand the outcomes of the Project. ii Joint Evaluation Exercise Phase I: Phase II: Phase III: Consolidation Preparation Execution and Report Preparation ADB Traffic IED Study (ADB) Discussion between Socioeconomic IED Study and EvD (ADB) on the Joint Joint Lessons from the Evaluation Evaluation Evaluation Joint Evaluation Methodology Mission Report Exercise and (ADB and (ADB and (ADB and Finalization EBRD) EBRD) EBRD) of the Technical Methodology EBRD Study EvD (EBRD) April June–July September February–March February–March 2008 2008 2008 2009 2009 ADB = Asian Development Bank, EBRD = European Bank for Reconstruction and Development, EvD = Evaluation Department, IED = Independent Evaluation Department. Sources: ADB IED and EBRD EvD. BASIC DATA (Asian Development Bank) Loan 1774-KAZ: Almaty–Bishkek Regional Road Rehabilitation Project (Kazakhstan Component) Project Preparation and Institution Building TA No. TA Name Type Person- Amount Approval Months ($’000) Date 3530-KAZ Improvement of the Road Sector Efficiencya ADTA 40 750 31 Oct 2000 As per ADB Key Project Data ($ million) Loan Documents Actual Total Project Cost 112.4 122.8 Foreign Exchange Cost 58.9 77.5 Local Currency Cost 53.5 45.3 ADB Loan Amount/Utilization 65.0b 49.8 ADB Loan Amount Canceled 15.2 EBRD Financed 25.0 28.5 European Union TRACECA Financed 0.4 0.4 Key Dates Expected Actual Fact-Finding 1–16 Mar 1999 Appraisal 14 Jun–26 Jul 1999 Loan Negotiations 4–7 Sep 2000 Board Approval 31 Oct 2000 Loan Agreement 4 Jun 2001