Evaluation SNV New strategy, new results? Client: Ministry of Foreign Affairs ECORYS Nederland BV SEOR BV Willem Cornelissen Maaike Oosterbaan Rotterdam, 13 March 2006 SEOR BV ECORYS Nederland BV P.O. Box 1738 P.O. Box 4175 3000 DR Rotterdam 3006 AD Rotterdam The Netherlands Watermanweg 44 3067 GG Rotterdam T +31 (0)10 408 2220 The Netherlands F +31 (0)10 408 9650 www.seor.nl T +31 (0)10 453 88 00 F +31 (0)10 453 07 68 E [email protected] W www.ecorys.com Registration no. 24316726 ECORYS Macro & Sector Policies T +31 (0)10 453 87 53 F +31 (0)10 452 36 60 DS/AN13968 Table of contents Preface 7 Summary 9 List of abbreviations and acronyms 19 1 Introduction 21 2 SNV 27 2.1 Description of the changes 27 2.2 Present situation 29 2.3 Current activities in three countries 30 3 Changes in structure and working methods at corporate level 35 3.1 Introduction 35 3.2 Development of the transformation components 35 3.2.1 Organisation of activities 36 3.2.2 Organisation of people 39 3.2.3 Organisation structure and processes 42 3.2.4 Organisation of funding 46 3.2.5 The role of SNV in the field 48 3.2.6 The role of SNV as knowledge organisation 50 3.2.7 Increase of efficiency 52 3.3 Assessment of the realisation of the transformation components 54 4 The field level implementation of capacity building 59 4.1 SNV intervention logic 59 4.2 Input 61 4.2.1 Client selection 62 4.2.2 Practice areas 70 4.2.3 Classification of inputs 73 4.2.4 Advisory staff and their competency 74 4.2.5 Assessment 76 4.3 Output 78 4.3.1 Assessment 85 4.4 Effectiveness (outcome and results) 86 4.4.1 Methodology used by SNV 86 4.4.2 Outcome: Capacities strengthened 88 4.4.3 Sustainability of capacities 90 DS/AN13968 4.4.4 Results: strengthened capacities lead to improved performance by clients 92 4.4.5 Assessment 95 4.5 Impact 95 4.5.1 Methodology used by SNV 95 4.5.2 Impact: improved performance by clients leads to reduced poverty and better governance 99 4.5.3 Assessment 100 5 Efficiency and learning capacity 103 5.1 Efficiency 103 5.2 Effectiveness and organisation 105 5.3 Learning Capacity and Practice areas 106 5.4 Assessment 111 6 Conclusions and points of attention for a next phase 115 6.1 Conclusions by topic 115 6.2 Points of attention for the next phase 120 Annex A List of contacts and documentation 123 Annex B Capacity Development Services 141 Preface This report contains the findings and conclusions of the evaluation of SNV (Netherlands Development Organisation). SNV, since late 2001 an independent organisation, receives core funding from the Ministry of Foreign Affairs. The purpose of the evaluation was to assess the progress in the transformation process as agreed upon between the Ministry of Foreign Affairs and SNV, to contribute to the accountability of the use of the funds and to report on the results achieved in the field. The evaluation assessed the realisation of the transformation process of SNV from an organisation that delivered technical assistance through projects to an organisation that supplies advisory services to meso level organisations. Furthermore, the effectiveness, efficiency of activities and learning capacity of the organisation were assessed. The evaluation covered the period 2002-2005. The evaluation was carried out during the period November 2005 to January 2006 and had to cope with a very tight time schedule. The evaluation included a study in the Netherlands, a survey among all SNV offices in the field and field visits to Bolivia, Kenya and Mali. The team consisted of consultants from ECORYS Netherlands BV and SEOR BV of the Erasmus University Rotterdam. The composition of the team is presented below. Willem Cornelissen Team leader Maaike Oosterbaan Project coordinator, Team Netherlands John Zevenbergen Team Netherlands Sarah Sijses Responsible for survey SNV portfolios Agnes Janszen Responsible for efficiency analysis Loet Mennes Senior advisor Karen Bakhuisen Field visit Kenya Kees van der Zanden Field visit Mali Graham Alders Local consultant Kenya Brahima Fomba Local consultant Mali Rafael Rojas L. Local consultant Bolivia The team would like to thank all SNV staff involved in the evaluation for providing information to the evaluation team and for the well prepared field visits in Bolivia, Kenya and Mali. Without this support from SNV, the evaluation could not have been carried out in such a short period. It should also be mentioned that the response to the survey to all SNV field offices was overwhelming. The team is also grateful for the cooperation of the many persons interviewed (see Annex A) in the Netherlands and in the three countries visited. Evaluation SNV 7 Summary Introduction The Netherlands Ministry of Foreign Affairs has assigned the evaluation of the implementation of the Subsidy Agreement between the Ministry and the Netherlands Development Organisation SNV over the period late 2002-2005 to ECORYS Netherlands in cooperation with SEOR of the Erasmus University Rotterdam. After approval of the methodological inception report on 28 th November 2005, the evaluative research took place during the month December 2005, and analysis and reporting during January 2006. The research has entailed a desk-study and interviews, a survey among all SNV portfolios 1 worldwide and field visits to the Santa Cruz portfolio in Bolivia, the North Rift portfolio in Kenya and the Koulikoro portfolio in Mali. The Terms of Reference formulated the following objective for the evaluation: To which extent is SNV capable to strengthen and promote the local capacity of meso- level organisations in developing countries? The strengthening of local capacity is defined as the process of capacity building (of an organisation) to reduce poverty and improve good governance. The evaluation has to focus on both approach and organisation of SNV and the achieved results, as well as the coherence between the two. The Terms of Reference formulated three main questions: 1. In which manner has SNV realized changes in its structure and working methods since becoming autonomous? 2. Did the advisory services of SNV contribute to capacity building and did this contribute to sustainable poverty reduction and good governance? 3. How efficient and effective are the advisory services of SNV and what is the learning capacity of SNV? Since 2002, the relation between SNV and the Ministry of Foreign Affairs has been settled in a Subsidy Agreement that provides funding to SNV for an amount of € 500 million over a period of five years. This represents well over 90 percent of all SNV costs. SNV has acquired a separate position in the Dutch development co-operation as a foundation which is almost fully dependent on funding, but is autonomous for its operations, namely offering advisory services free of charge in more than 30 countries. To SNV the period under review was characterised by a multi-dimensional change process comprising: 1 SNV operates at sub-national level and operates from sub-national offices. The total of advisory activities managed by and implemented through such an office is called a ‘portfolio’. Worldwide there are 100-110 portfolio teams active. Evaluation SNV 9 • Disentanglement from the Ministry of Foreign Affairs (2001) and transformation into an autonomous foundation; • A new intervention strategy (2001) leading to new services and activities; • These new services required both a change in the required capacities of staff and a change in the organisational structure of SNV. The objective and research questions mentioned above imply that this evaluation neither analysed the full range of the relations between the Ministry of Foreign Affairs and SNV, nor the policy implications thereof, such as the complementarities of SNV services vis-à- vis other cooperation activities either realized by or subsidized by the Ministry of Foreign Affairs. The evaluation has been mainly based on the statistical and financial information provided by SNV. This information has not been checked on veracity. The findings of three field mission to Bolivia, Kenya and Mali are used as case studies and cannot be hold representative for the entire organisation. However, the survey among portfolios covered almost the entire population of SNV portfolios and can be taken as statistically representative. Due to the restrictions in research time available, the effectiveness of the advisory services in the field has been studied on the basis of secondary information (existing studies and information) and interviews. Findings The findings are structured along the lines of the three main research questions as mentioned in the Terms of Reference. 1. In which ways has SNV realized changes in its structure and working methods since becoming autonomous? The transformation has been evaluated on the basis of the ten characteristics formulated in the Subsidy Agreement. SNV aims at complying with these characteristics by the end of the period of the Subsidy Agreement (2006). These changes can be distinguished in those in working methods and those in structure: a. changes in working methods SNV has completed a multi-faceted change process transforming its services from the supply of long-term technical assistance by individual experts to host organisations at micro level to the provision of advisory services in teams to (mainly) organisations operating at meso level. In 2004, between 64 percent (Asia) and 88 percent (East and Southern Africa) of the clients were categorised as ‘meso level clients’. In its working methods, SNV has completed the change process towards the provision of advisory and consultancy services (by teams). In the case of still existing narrowly marked out services, the client can still be attended by a single advisor, but not on a long- term assignment anymore.