Terminal Evaluation of the UN Environment Project “Reducing Greenhouse Gas Emissions with Bus Rapid Transit and Non-Motorized Transport” Terminal Evaluation of the UN Environment-GEF Project “Reducing Greenhouse Gas Emissions with Bus Rapid Transit and Non-Motorized Transport” Final report Evaluation Office of UN Environment April 2018 1 Terminal Evaluation of the UN Environment Project “Reducing Greenhouse Gas Emissions with Bus Rapid Transit and Non-Motorized Transport” Evaluation Office of UN Environment Photos Credits: Front cover: Roland Wong Page 47-48, 53, 56: Roland Wong @UN Environment/(Roland Wong), UN Environment Evaluation Mission (2018) Page 58: Courtesy of TransCaribe Community Outreach Group This report has been prepared by independent consultant evaluators and is a product of the Evaluation Office of UN Environment. The findings and conclusions expressed herein do not necessarily reflect the views of Member States or the UN Environment Senior Management. For further information on this report, please contact: Evaluation Office of UN Environment P. O. Box 30552-00100 GPO Nairobi Kenya Tel: (254-20) 762 3389 Email: [email protected] (Reducing Greenhouse Gas Emissions with Bus Rapid Transit and Non-Motorized Transport) (GEF Project ID 1917) (04/2018) All rights reserved. © (2018) Evaluation Office of UN Environment 2 Terminal Evaluation of the UN Environment Project “Reducing Greenhouse Gas Emissions with Bus Rapid Transit and Non-Motorized Transport” ACKNOWLEDGEMENTS This Terminal evaluation was prepared for the Evaluation Office of UN Environment by Mr. Roland Wong, Climate Change Mitigation Specialist as the Evaluation Consultant. The report benefits from a peer review conducted within Evaluation Office of UN Environment. The Evaluation Office of UN Environment would like to thank the BRT Project teams and in particular, the BRT Project team in Dar es Salaam (Dar Rapid Transit Agency, Dar es Salaam City Council, Tanzania National Roads Agency, and the World Bank Group in Dar es Salaam), and the BRT Project team in Cartagena (the Technological University of Bolivar in Cartagena, TransCaribe, Transambiental, Ciudatos and Despacio) for their contribution and collaboration throughout the Evaluation process. Sincere appreciation is also expressed to other project stakeholders in Columbia, Dar es Salaam and other locations who took time to provide comments to the draft report. The Evaluation Office of UN Environment would also like to thank the Institute for Transportation and Development Policy (namely the Africa Program and Latin America Region) as well as colleagues at the Climate and Technology Branch and the Air Quality and Mobility Unit of UN Environment in Nairobi. Evaluation team Roland Wong – Evaluation Consultant Evaluation Office of UN Environment Saila Toikka – Evaluation Manager Mela Shah – Evaluation Programme Assistant 3 Terminal Evaluation of the UN Environment Project “Reducing Greenhouse Gas Emissions with Bus Rapid Transit and Non-Motorized Transport” ABOUT THE EVALUATION1 Joint Evaluation: No Report Language(s): English Evaluation Type: Terminal Project Evaluation Brief Description: This report is a terminal evaluation of a UN Environment-GEF project implemented between 2005 and 2016. The project aimed to increase overall knowledge of planning, designing and implementing Bus Rapid Transit systems in cities in developing countries, and reduce GHG emissions from the transport sector globally. The evaluation sought to assess project performance (in terms of relevance, effectiveness and efficiency), and determine outcomes and impacts (actual and potential) stemming from the project, including their sustainability. The evaluation has two primary purposes: (i) to provide evidence of results to meet accountability requirements, and (ii) to promote learning, feedback, and knowledge sharing through results and lessons learned among UN Environment, the GEF and their executing partner ITDP and the relevant agencies of the project participating countries. Key words: Project Evaluation; Climate Change; TE; Terminal Evaluation; GEF; GEF Project; Transportation; Bus Rapid Transit; Non-motorized Transit 1 This data is used to aid the internet search of this report on the Evaluation Office of UN Environment Website 4 Terminal Evaluation of the UN Environment Project “Reducing Greenhouse Gas Emissions with Bus Rapid Transit and Non-Motorized Transport” Table of Contents EXECUTIVE SUMMARY ........................................................................................................................... 9 1 INTRODUCTION ............................................................................................................................. 14 1.1 EVALUATION OBJECTIVES ....................................................................................................................... 14 1.2 EVALUATION APPROACH AND METHODOLOGY ........................................................................................... 14 1.3 MAIN EVALUATION CRITERIA AND QUESTIONS ........................................................................................... 15 2 PROJECT BACKGROUND ............................................................................................................... 17 2.1 CONTEXT .............................................................................................................................................. 17 2.2 PROJECT OBJECTIVES AND COMPONENTS ............................................................................................... 21 2.2.1 Objectives ...................................................................................................................................... 21 2.2.2 Components .................................................................................................................................. 22 2.3 TARGET AREAS/GROUPS........................................................................................................................ 22 2.4 MILESTONES IN PROJECT DESIGN AND IMPLEMENTATION ......................................................................... 24 2.5 IMPLEMENTATION ARRANGEMENTS AND PROJECT PARTNERS ................................................................... 24 2.6 PROJECT FINANCING ............................................................................................................................. 25 2.7 CHANGES IN DESIGN DURING IMPLEMENTATION ........................................................................................ 25 2.8 RECONSTRUCTED THEORY OF CHANGE OF THE PROJECT ........................................................................... 26 3 EVALUATION FINDINGS ................................................................................................................ 32 3.1 STRATEGIC RELEVANCE ......................................................................................................................... 32 3.1.1 Alignment with UN Environment’s strategy, policies and mandate ............................................. 32 3.1.2 Alignment with GEF focal areas and strategic priorities .............................................................. 33 3.1.3 Relevance to global, regional and national environmental issues and needs and complementarity to other interventions .................................................................................................... 34 3.2 QUALITY OF PROJECT DESIGN ................................................................................................................ 35 3.3 NATURE OF EXTERNAL CONTEXT ............................................................................................................ 38 3.4 EFFECTIVENESS ..................................................................................................................................... 39 3.4.1 Delivery of Outputs for Component 1: Dar es Salaam BRT .......................................................... 39 3.4.2 Achievement of Outputs for Component 2: Cartagena BRT system ........................................... 42 3.4.3 Achievement of Outputs for Component 3: BRT Planning Guide ................................................ 44 3.4.4 Achievement of direct outcomes as defined in the reconstructed TOC ...................................... 44 3.4.5 Likelihood of impact ...................................................................................................................... 54 3.5 FINANCIAL MANAGEMENT ...................................................................................................................... 65 3.5.1 Completeness of project financial information ............................................................................ 65 3.5.2 Communication between finance and project management staff .............................................. 65 3.6 EFFICIENCY ........................................................................................................................................... 66 3.6.1 Timeliness ..................................................................................................................................... 66 3.6.2 Cost Efficiencies ............................................................................................................................ 67 3.7 MONITORING AND REPORTING ................................................................................................................ 67 3.7.1 Monitoring design and budgeting ................................................................................................