Evaluation of the European Union’s humanitarian assistance in Ukraine, 2014-2018 Final Report Written by Universalia (with support from Landell Mills) July 2020 Final Report - Evaluation of the European Union’s humanitarian assistance in Ukraine, 2014-2018 Directorate-General for Civil Protection and Humanitarian Aid Operations 2020 EUROPEAN COMMISSION Directorate-General for Civil Protection and Humanitarian Aid Operations Contact: [email protected] European Commission B-1049 Brussels Evaluation of the European Union’s humanitarian assistance in Ukraine, 2014-2018 Final Report Directorate-General for Civil Protection and Humanitarian Aid Operations 2020 Neither the European Commission nor any person acting on behalf of the Commission is responsible for the use which might be made of the following information. Europe Direct is a service to help you find answers to your questions about the European Union. Freephone number (*): 00 800 6 7 8 9 10 11 (*) The information given is free, as are most calls (though some operators, phone boxes or hotels may charge you). LEGAL NOTICE This document has been prepared for the European Commission however it reflects the views only of the authors, and the Commission cannot be held responsible for any use which may be made of the information contained therein. More information on the European Union is available on the Internet (http://www.europa.eu). Luxembourg: Publications Office of the European Union, 2020 ISBN 978-92-76-20943-0 doi: 10.2795/403618 © European Union, 2020 Reproduction is authorised provided the source is acknowledged. v Acknowledgements The evaluation team was comprised of Julian Murray (Team Leader), Sharon Truelove, Katerina Stolyarenko, Viktoria Zalesskaya, and Sophie Pénicaud. They were supported in-country by Anna Ivanchenko, in Montréal by Adrian Profitos, Anne-Marie Dawson and Ariane Joazard-Bélizaire, and in Brussels by Tom Barton. The evaluation team would like to thank the Directorate-General for European Civil Protection and Humanitarian Aid Operation (DG ECHO) staff in Kyiv, Brussels and Amman for their considerable engagement throughout the evaluation process. As a consequence of its participatory approach, the evaluation demanded a lot of staff involvement in workshops and feedback discussions. European Union Delegation (EUD) colleagues from the Directorate-General for Neighbourhood and Enlargement Negotiations, Support Group for Ukraine (DG NEAR-SGUA), European External Action Service (EEAS) and the Commission Service for Foreign Policy Instruments (FPI) also made valuable contributions to the data collection and to the validation of overall findings. At the risk of singling out a few people from a large number, the team would like to thank in particular Samuel Marie-Fanon in Kyiv, Cristina Martinez Gallego in Brussels, and the evaluation manager José Manuel Lopez de la Mano, whose flexibility and willingness to experiment greatly helped the work of the evaluation team. vi Contents Abstract v Executive summary vi Résumé exécutif xi 1. Introduction 1 1.1. Evaluation purpose, objectives and scope 1 1.2. Evaluation process 1 1.3. Evaluation overview 1 2. Methodology 2 2.1. Methodological approaches 2 2.2. Data collection methodology 3 2.3. Limitations of the data 3 3. Country and programme context 4 3.1. The crisis in Ukraine and its humanitarian implications 4 3.2. The EU’s humanitarian response 8 4. Findings 13 4.1. Relevance 13 4.2. Coherence 24 4.3. EU value-added 33 4.4. Effectiveness 34 4.5. Efficiency 45 4.6. Sustainability/connectedness 55 4.7. Overview of Evaluation Questions and related Findings 57 4.8. Overall assessment of DG ECHO performance 60 5. Conclusions 62 5.1. Relevance: vulnerability and targeting 62 5.2. Coherence: coordination and the nexus 62 5.3. Effectiveness: programme and advocacy results 63 5.4. Efficiency: a protracted humanitarian crisis in a country of declining needs and interest 64 5.5. Sustainability: resilience and humanitarian exit 64 5.6. The special challenges of the NGCAs 65 6. Recommendations 67 6.1. Inclusion of the elderly and disabled 67 i 6.2. Increasing access and programming in the NGCAs 67 6.3. Working more strategically on the humanitarian-development nexus 68 6.4. Improving efficiency through multi-year programming, localisation and harmonised reporting/monitoring 68 6.5. Preparing deliberately for humanitarian exit from the GCAs 69 Annex 1: Interviewee perceptions of DG ECHO performance 70 Annex 2: Team assessments of DG ECHO performance from document review 71 Annex 3: Results of scorecards completed by key informants 72 Annex 4: Perceptions of beneficiaries as captured in an FGD survey 73 Annex 5: Evaluation Matrix 74 Annex 6: Bibliography 79 Annex 7: List of key stakeholders interviewed 92 Annex 8: Terms of Reference 96 ii Acronyms AAP Accountability to Affected Populations COHAFA Council working party on Humanitarian Aid and Food Aid DG ECHO Directorate-General for European Civil Protection and Humanitarian Aid Operations DG NEAR-SGUA Directorate-General for Neighbourhood and Enlargement Negotiations - Support Group for Ukraine DRC Danish Refugee Council EC European Commission EEAS European External Action Service EECP Entry and Exit Crossing Point EIB European Investment Bank EiE Education in Emergencies ENI European Neighbourhood Instrument EQ Evaluation Question EU European Union EUD European Union Delegation FGD Focus Group Discussion FLAC Free Legal Aid Centre FPI Commission Service for Foreign Policy Instruments FTS Financial Tracking Service GBV Gender-Based Violence GCA Government-Controlled Area HAI HelpAge International HCT Humanitarian Country Team HDG Humanitarian Donor Group HI Humanity and Inclusion (formerly Handicap International) HIP Humanitarian Implementation Plan HNO Humanitarian Needs Overview HRP Humanitarian Response Plan IAF Integrated Analysis Framework IASC Inter-Agency Standing Committee ICRC International Committee of the Red Cross IcSP Instrument contributing to Stability and Peace IDP Internally displaced person IHL International Humanitarian Law INEE Inter-Agency Network on Education in Emergencies INSO International NGO Safety Organisation IOM International Organization for Migration iii IP Implementing Partner JHDF Joint Humanitarian-Development Framework KII Key Informant Interview MdM Médecins du Monde (Médicos del Mundo) MHPSS Mental Health and Psycho-Social Support MinTOT Ministry of Temporarily Occupied Territories and IDPs (since March 2020, renamed Ministry for Reintegration of Temporarily Occupied Territories of Ukraine) MPCT Multi-Purpose Cash Transfers MS Member State NFI Non-Food items NGCA Non-Government-Controlled Area NGO Non-Governmental Organisation NRC Norwegian Refugee Council OHCHR Office of the High Commissioner for Human Rights OSCE Organisation for Security and Cooperation in Europe PIN People in Need PUI Première Urgence Internationale RPBA Recovery and Peace-building Assessment SC Save the Children SMM Special Monitoring Mission TOR Terms of Reference UHF Ukraine Humanitarian Fund UNDP United Nations Development Programme UNHCR United Nations Refugee Agency UNICEF United Nations Children's Fund UNOCHA United Nations Office for the Coordination of Humanitarian Affairs USA United States of America WASH Water, Sanitation and Hygiene iv Abstract This evaluation covers DG ECHO’s programming in Ukraine 2014-2018 (EUR 118.4 million). DG ECHO’s 65 actions were delivered by 21 Implementing Partners and covered Protection, Health, Basic Needs including Food Security, Water, Sanitation and Hygiene, and Education in Emergencies. The evaluation was conducted between August 2019 and May 2020, with fieldwork in Ukraine in December 2019. The evaluation found that DG ECHO was rapid and effective, and overall performed very well. DG ECHO demonstrated system-wide leadership in strategic thinking and coordination. Member states regarded highly the field team’s strength and its access to the Non-Government-Controlled Areas (NGCAs). In Ukraine, DG ECHO supported several innovations: notably system-wide joint and impartial needs assessment, advancing an explicit Joint Humanitarian-Development Framework with the other European Union services, and supporting the creation of the Ukraine Humanitarian Fund. All donors, including DG ECHO, were somewhat slow to adapt their programming to the unusually high proportion of vulnerable elderly persons in the affected population. Government, development and humanitarian actors were also slow to connect and work together in the most-affected Government-Controlled Areas. Although access is difficult, needs in the NGCAs remain high, and should continue to be the focus of advocacy and programming in 2020 and beyond. v Executive summary Objectives and scope This is an independent retrospective evaluation of the Directorate-General for European Civil Protection and Humanitarian Aid Operations’ (DG ECHO's) interventions in Ukraine (2014-2018). The evaluation has three objectives: • Accountability: it assesses the performance of European Union (EU) interventions compared to initial expectations, engages stakeholders and encourages feedback, and offers an independent and objective judgement based on available evidence • Learning: it supports organisational learning by identifying areas for improvement and encouraging the sharing of (good and bad) practices and achievements • Strategy: it aims to make information available in time to support planning for 2021 The geographic scope is the entire region covered by DG ECHO’s Ukraine Humanitarian