ICCO Food Security Programme Evaluation – Synthesis report Synthesis Report ACE Europe January 20 11 ICCO Food Security Programme Evaluation – Synthesis report Corina Dhaene and Hannelore Beerlandt With the contribution of a team of evaluators Eustache Wankpo (mission to Benin) Hannelore Beerlandt (IDIS, inception phase, analysis of data, mission to Madagascar) Ramy Razafindralambo (mission to Madagascar) Gerda Heyde (mission to Bangladesh) Shashank Grahacharjya (mission to Bangladesh) Bruno Kervyn (mission to Bolivia) Maria Del Carmen Camacho (mission to Bolivia) Lina Neeb (ACE Europe, analysis of data) Acknowledgement The evaluators would like to thank first of all the respondents in the respective countries: their cooperation, readiness to welcome the evaluators and their willingness to share their perceptions, conclusions, hopes and fears with the evaluators have made this evaluation possible. The evaluators would also like to thank ICCO for the opportunity to execute this evaluation and express their appreciation for the commitment of the ICCO team and the external reference group to the results of the evaluation and their contribution to the quality of this report. Last but not least, ACE Europe thanks the consultants in the evaluation team for their efforts to work within the ‘straight jacket’ of the methodology for this programme evaluation. Finally, it is our hope that this report will contribute to changes in the FS programme to support the effectiveness and sustainability of the very much needed interventions of all NGO partners involved. Corina Dhaene and Hannelore Beerlandt ACE Europe / evaluatie faire trade scholenproject / eindrapport pag. 5/124 Inhoud 1 Executive summary 8 2 Introduction 17 2.1. Context and purpose of the evaluation 17 2.2. Objectives 17 2.3. Evaluation questions 18 2.4. Methods of data collection and limitations of the evaluation 20 2.5. Evaluation team and reference group 28 2.6. Object of evaluation: 28 3 Description of policy theory 43 2.7. Objectives and general approach of the FS programme of ICCO 43 2.8. Theory of change 53 4 Findings of the evaluation 56 4.1. ICCO’s strategy: relevance and coherence 56 4.2. changes in the food security situation 67 4.3. INfluence on policy making: positioning and voice 89 4.4. Assessment of the contribution of ICCO/KIA 103 5 Conclusions 107 5.1. On ICCO’s policy and strategy 107 5.2. On changes in the food security situation 109 ACE Europe / ICCO Food Security Programme - Evaluation / Synthesis report pag. 6/124 5.3. On influence on policy making 111 5.4. On the roles of ICCO 113 5.5. Overall conclusion 114 5.6. Analysis and Identification of dilemma’s 116 5.7. Recommendations: avenues for improvement 118 ACE Europe / ICCO Food Security Programme - Evaluation / Synthesis report pag. 7/124 1 Executive summary Introduction - This report presents the main findings and conclusions of the evaluation of ICCO’s programme on Food Security, executed by ACE Europe and a team of 9 independent consultants at the request of ICCO. The evaluation was supported by an external reference group. This evaluation covers ICCOs’ programme on Food Security between 2007 and the first 6 months of 2010. The total ICCO expenditure 1 for food security (under MFS) during the considered period is 28.603.068 euro. The ICCO food security program (MFS and non MFS) presents about 15% of the total program expenditure of ICCO. The food security program is a sub-program of the ‘Access to Basic Services’ program (and represents 30% of the expenditure of the ABS program, which is the highest portion in this program). With the MFS finances, ICCO has contributed to FS projects in 51 countries and on 4 supra-national levels. In total ICCO has contributed to 419 projects which work at least partly on food security and a total of 278 partners were financed for food security. ICCO selected 20 focus countries for food security in which the organisations of the ICCO alliance together support the food security program. The focus countries for food security are: Afghanistan, Bangladesh, Benin, Bolivia, Burkina Faso, DR Congo, Ecuador, Haïti, India, Madagaskar, Malawi, Mali, Myanmar, Nepal, Pakistan, Paraguay, Peru, Sudan, Uganda and South Africa. An overview of the financial distribution is given in the report: when looking at the continents, Africa and South East Asia take the largest part of the budget with 48% of the budget going to Africa. The aim of the evaluation is to know what the results of the programme have been, more in particular if intended results for the target group have been reached and to judge the involvement of ICCO (both financial and otherwise). The main target groups are understood to be the most vulnerable groups, e.g. small holders, internally displaced people or IDP’s, female headed households, children under 5, …). The evaluation was executed in three phases: inception phase (including definition of sample), field missions, reporting phase and ran between June 2010 and December 2010. Methodology - The evaluation was based on a methodology elaborated during the inception phase. Changes to this methodology and adaptations during field missions are clearly described in this report. Main elements of the methodology are the following: − ACE Europe choose to apply a methodology which is essentially qualitative in nature, taking into account different perceptions of various stakeholders (triangulated), quantitative 1 Based on the monitoring database of ICCO (‘Dynamics’). In this database contracts (projects) are entered. For each project, the percentage of the expenditure that is used for food security is indicated, based on the content of the activities and strategies. This judgement is not based on objective criteria but on human judgement by ICCO staff, closely positioned to the different partners and projects. Only the expenditure for food security, based on these percentages for the time period 2007-July 2010, is considered and calculated under the FS program. ACE Europe / ICCO Food Security Programme - Evaluation / Synthesis report pag. 8/124 data in supporting documents (such as reports of partner organisations) and national statistics (when available and reliable). − ACE Europe formulated 8 evaluation questions with judgement criteria and indicators (based on the evaluation questions mentioned in the ToR drafted by ICCO), which are related to the traditional evaluation criteria (but with less attention to efficiency). The questions are related to the strategy of ICCO (and assessing its clarity, relevance and coherence when looking at implementation through partners), the changes in the food security situation of vulnerable groups in a structural and gender sensitive way and with regards changes in food availability, access to food and utilisation of food, the position of ICCO’s partners and their lobby efforts and the contribution of ICCO (roles played). For reasons explained in this report, questions have been combined or reformulated (when compared to the inception report and the country reports). Overview of evaluation questions EQ 1 - To what extent have the policy and strategies of ICCO offered a specific framework to address the rights and needs related to food security of the most vulnerable groups? (clarity and relevance) EQ 2 - To what extent are the ICCO strategies and policies translated into the cooperation and to what extent have possible synergies in the strategies been used optimally (coherence)? EQ 3-5: to what extent have the interventions allowed to influence food availability, access to food and improved utilisation of food for vulnerable households in a structural and gender sensitive way? EQ 6-7: how are ICCO’s partners positioned (towards their target groups, other NGOs and the lobby targets), what lobby actions have been undertaken and what are the results thereof? EQ 8: what has been the contribution of ICCO? − Combination of desk study, e-questionnaires and field missions (semi-structured interviews, focus group discussions, workshops with partner organisations and indepth work sessions in villages). Through these different instruments, it was possible to triangulate findings. Each country visit was executed by a local/regional consultant and an international consultant. − Application of the PADEV methodology: central in the approach were the sessions with beneficiaries, in which the consultants applied a recently developed method for impact assessment, the PADEV method (Ton Dietz, 2010). to which ICCO has contributed. The PADEV method has been designed to get a bottom-up assessment of development and change in a particular area/community and to get a valuation of the usefulness and impact of specific interventions or initiatives by people from the community. The evaluators have adapted this methodology prior to the field missions and again during the field missions (depending on the availability of beneficiaries and logistic constraints). ACE Europe / ICCO Food Security Programme - Evaluation / Synthesis report pag. 9/124 Object of evaluation - ACE Europe initially planned to cover at least 40% of the FS budget to capture the expected diversity of approaches and contexts. This was almost fully realised by the selection of the 7 countries: Bangladesh, Benin, Bolivia, Malawi, Madagascar, Mali and RD Congo. The sample of countries (4 covered by field visits and 3 through e-questionnaires and skype interviews) covers 36,8% of the expenditure for FS (MFS, 2007-July 2010) or a total of 10.406.807 euro. Within each country a number of partners were selected upon clearly specified criteria to be directly involved; the budget and time frame of this evaluation did not make it possible to visit/include all ICCO partners. This sample of partners represented 21,52% of the total expenditure for FS (6.156.652 euro), The findings of the evaluation in the visited countries however have been validated by all ICCO partners in the respective countries through national validation workshops. The evaluators conclude therefor that the sample allowed them to appreciate the variety in ICCO programmes, to highlight some general tendencies and to be conclusive on the whole FS programme.