EXTERNAL EVALUATION SEPTEMBER 2014 Evaluation of ACF’s Livelihood Projects in Abkhazia funded by UNHCR in 2011-2013 Funded by Photos ©Matteo Modè UNHCR By Matteo Modè This report was commissioned by Action Against Hunger | ACF International. The comments contained herein reflect the opinions of the Evaluators only. This publication has been produced with the assistance of the Office of the United Nations High Commissioner for Refugees (UNHCR). The contents of this publication are the sole responsibility of ACF and can in no way be taken to reflect the views of UNHCR. Table of Contents 1 Executive Summary ............................................................................................................................... 5 2 Introduction .......................................................................................................................................... 7 3 Background and Context ....................................................................................................................... 7 4 Projects description .............................................................................................................................. 8 4.1 2011 project .................................................................................................................................. 9 4.2 2012 project ................................................................................................................................ 10 4.3 2013 project ................................................................................................................................ 11 5 Methodology ....................................................................................................................................... 12 5.1 Data and information sources .................................................................................................... 12 5.2 Selection of interviewees ............................................................................................................ 12 5.3 Evaluation criteria ....................................................................................................................... 14 6 Scope and limitations of the evaluation ............................................................................................. 15 6.1 Scope of the evaluation .............................................................................................................. 15 6.2 Limitations of the evaluation ...................................................................................................... 15 7 Results ................................................................................................................................................. 15 7.1 Relevance .................................................................................................................................... 15 7.2 Effectiveness ............................................................................................................................... 17 7.3 Efficiency ..................................................................................................................................... 23 7.4 Impact ......................................................................................................................................... 24 7.5 Sustainability ............................................................................................................................... 26 7.6 Complementarity/coherence ...................................................................................................... 27 7.7 Coverage ..................................................................................................................................... 28 7.8 DAC results’ matrix and overall performance of projects .......................................................... 29 8 Best practices ...................................................................................................................................... 31 8.1 Best practice 1 – Using previous project expertise and achievements ...................................... 31 8.2 Best practice 2 - Referral of beneficiaries ................................................................................... 31 8.3 Best practice 3 – Specialized training ......................................................................................... 31 9 Conclusions ......................................................................................................................................... 32 10 Recommendations .......................................................................................................................... 32 11 Annexes ........................................................................................................................................... 35 2 11.1 Agenda of field work ................................................................................................................... 35 11.2 List of persons interviewed ......................................................................................................... 36 11.3 List of documents consulted ....................................................................................................... 38 11.4 Terms of reference of the evaluation ......................................................................................... 39 3 Abbreviations ACF – Action Contre la Faim ATC – Agricultural Training Centre DAC – Development Assistance Committee (see OECD) DL – Dividing Line DV – Domestic Violence GA – Grant assistance IDP – Internally Displaced Person (s) IGA – Income Generating Activity INGO – International Non-Governmental Organisation NGO – Non-Governmental Organisation OECD – Organisation for Economic Development and Cooperation SGBV – Sexualised and Gender Based Violence TOT – Training of Trainers UNDP – United Nations Development Programme UNHCR – United Nations High Commissioner for Refugees UNICEF - United Nations International Children's Emergency Fund VTC – Vocational Training Centre 4 1 Executive Summary This project presents the findings of an independent evaluation of three livelihoods projects funded by UNHCR and implemented by ACF in Abkhazia in 2011, 2012 and 2013. These projects were aimed at improving the livelihoods and self-reliance of beneficiaries via vocational and business training opportunities and via the distribution of grants to support the start-up and development of small business. Although these are three separate projects in terms of planning and budget, they share the same approach and delivery modality, with variations in terms of targeting of beneficiaries and activities supported via grant assistance. This evaluation uses the OECD DAC guidelines and principles to assess the performance of the projects according to a set of standardised criteria. Based on the findings, this report presents a set of recommendations and best practices to guide the implementation of future projects. The evaluation found the projects to be highly relevant in the context of high unemployment, lack of economic opportunities and vulnerability faced by people living in the projects’ area, given the focus on improving skills, confidence, livelihoods and self-reliance of beneficiaries. The projects have been highly effective in conducting and delivering a range of training programmes and in supporting the start-up and expansion of small businesses on a range of activities. The projects have made effective use of the experience and the assets developed by ACF under previous projects. At individual and household level, the projects have been moderately effective in improving the livelihoods of beneficiaries in terms of increased income and self-reliance. The projects have been more effective in supporting trade and service related activities taking place in urban and peri-urban areas and less effective with supporting seasonal and livestock related activities. The projects have been moderately efficient in the implementation of training activities and in the selection of beneficiaries and distribution of start-up grants. Some projects’ activities are more time and cost intensive than others, such as selection, verification and monitoring of beneficiaries. Efficiency is also affected by the cost per capita of delivering business, vocational and confidence building training in relation to the number of people trained and beneficiaries receiving grants. The projects had a positive impact in increasing the self-confidence and capacity of beneficiaries via training and provision of technical skills. Beneficiaries have increased their knowledge on business practices, acquired new vocational skills and received specialized training that they have been able to put into practice. On the other hand, the projects had an overall moderate impact on the livelihoods of beneficiaries receiving grants, although there are individual cases where the impact was high. Findings suggest that activities related to trade and services have had a higher impact on livelihoods, while others such as livestock and seasonal activities had less. Findings also suggest that grants had less impact on the livelihoods of more vulnerable people, which are less resilient in the event of a crisis and more prone to sell assets received via in grant assistance. Motivation and ownership have also proved to be important factors behind the successful use of grants for