Evaluation of ACACIA III: The Acacia Approach and Its Most Significant Outcomes 2006-2009 Facilitated by Ricardo Wilson-Grau, with support from Jennifer Vincent February 2010 Evaluation of ACACIA III Table of Contents CHAPTER 1: EXECUTIVE SUMMARY............................................................................. 2 INTRODUCTION ....................................................................................................................... 2 PHASE 1 – THE ACACIA APPROACH TO NETWORKING ............................................................ 2 Conclusions ........................................................................................................................ 3 PHASE 2 – THE EFFECTIVENESS OF THE ACACIA RESEARCH NETWORKS ................................. 4 Conclusions ........................................................................................................................ 4 CHAPTER 2: INTRODUCTION ........................................................................................... 6 CHAPTER 3: PURPOSE AND PRIMARY INTENDED USES AND USERS ................. 8 METHODOLOGY ...................................................................................................................... 9 EVALUATION QUESTIONS AND PROCESS ................................................................................ 10 CHAPTER 4: PHASE 1 – ACACIA’S APPROACH TO NETWORKING .................... 11 INTRODUCTION ..................................................................................................................... 11 METHODOLOGICAL CHALLENGES.......................................................................................... 11 FINDINGS – THE ACACIA APPROACH IS EFFECTIVE ............................................................... 12 Acacia’s goals .................................................................................................................. 12 Acacia’s operating principles .......................................................................................... 14 Acacia management process ............................................................................................ 18 DISCUSSION BY THE ACACIA TEAM ....................................................................................... 21 IN CONCLUSION ..................................................................................................................... 21 CHAPTER 5: PHASE 2 – ACACIA RESEARCH NETWORK OUTCOMES 2006-2009 .................................................................................................................................................. 23 INTRODUCTION ..................................................................................................................... 23 METHODOLOGICAL CHALLENGES.......................................................................................... 24 METHODOLOGY: PROCESS OF HARVESTING OUTCOMES ........................................................ 27 FINDINGS: REPRESENTATIVE OUTCOMES ACHIEVED SINCE 2006 ........................................... 27 SUMMARY OF OUTCOMES ..................................................................................................... 29 CHAPTER 6: PHASE 2 – ACACIA TEAM INTERPRETATION AND CONCLUSIONS REGARDING OUTCOMES .................................................................. 36 INTRODUCTION ..................................................................................................................... 36 OBJECTIVE #1: FOSTERING ON-GOING, ROBUST POLICY DIALOGUE ....................................... 36 OBJECTIVE #2: CATALYZING THRIVING REGIONAL RESEARCH NETWORKS ............................ 38 OBJECTIVE #3: INCREASING RESEARCH CAPACITY IN ICT4D ................................................ 42 OBJECTIVE #4: STIMULATING SOCIAL AND TECHNICAL INNOVATION IN ICTS ....................... 45 OBJECTIVE #5: CONTRIBUTING TO A FORMAL BODY OF KNOWLEDGE IN ICT4D ................... 52 OBJECTIVE #6: APPLYING MEANINGFUL GENDER ANALYSIS: ................................................ 55 META-CONCLUSIONS ............................................................................................................ 60 CHAPTER 7: GENERAL CONCLUSIONS AND POINTS FOR DISCUSSION .......... 61 CONCLUSIONS ON THE ACACIA APPROACH TO NETWORKING ............................................... 61 CONCLUSIONS ON THE EFFECTIVENESS OF ACACIA ............................................................... 61 RECOMMENDED POINTS FOR DISCUSSION .............................................................................. 62 CHAPTER 8: ANNEXES...................................................................................................... 64 Ricardo Wilson-Grau, February 2010 1 Evaluation of ACACIA III Chapter 1: Executive Summary Introduction This evaluation was designed to be an internal, formative evaluation of the Acacia approach of networking 2006-2009. Its dual primary purpose was a) to serve the Acacia team and its primary intended users in their decision-making to improve the Acacia approach to network programming and b) to generate knowledge that could be used to enhance the effectiveness and efficiency of the 13 Acacia networks. In addition to the uses that the findings would serve, this evaluation had two important process uses: First, the considerable participatory nature of the evaluation was intended to enhance the shared understanding amongst the Acacia team and the network co-ordinators, as well as network members, of the Acacia networking approach and the outcomes being achieved by the networks. Second, in the process of answering the evaluation questions, this report aimed to develop an outcomes monitoring modality that could be used in future to support and to reinforce both the application of the Acacia approach and the interventions designed for the supported network projects. In this participatory, experimental and developmental evaluation with a heavy learning focus, my involvement as an external evaluator aimed to facilitate the participatory process and ensure checks, balances, and objectivity. The evaluation employed an experimental mix of participative and formative evaluation modalities. We recognised that the experimental methodology meant that the evaluation would be an iterative process in which learning would lead to adaptation of the evaluation process itself. Consequently, the evaluation design was provisional and the evaluation questions and how to answer the questions emerged during the evaluation process. Phase 1 – The Acacia Approach to Networking The evaluation was carried out in two phases, each with a core evaluation question. Phase 1: To what extent is the Acacia networking approach for programming appropriate and relevant to the needs of African researchers, including women, and research institutions to develop research capacity in ICT4D? We carried out the first phase between September 2008 and May 2009 through an internet survey. The eight Acacia staff and fifteen leaders from eleven networks considered that the Acacia Networking Approach was fulfilling its purpose. They affirmed the relevance of Acacia‟s nine goals, nine operating principles and the ten management processes. There were, however, a few areas of relative weakness in the application of the principles: a) building research management capacity, b) building research institutions‟ capacity to manage the research and c) leveraging of other research funding opportunities and building sustainability for the research networks. This resounding affirmation of the Approach was accompanied by a number of suggestions for improvement. During the May 2009 Acacia team meeting in Dakar, the team discussed the findings about the Acacia Approach to Networking. These points were made: Acacia never envisaged long-term sustainability for the network projects themselves Thus, it was probably a mistake for the Acacia team to have agreed that “leveraging Ricardo Wilson-Grau, February 2010 2 Evaluation of ACACIA III of other research funding opportunities and building sustainability for the research networks” was one of Acacia‟s operating principles. The Acacia team recognised the tensions that exist between creating networks for building the research capacity amongst peers and the assumed capacity that exists in the host institutions. This would have required administrative support, which in many instances Acacia did provide – financial and administrative as well as legal assistance with subcontracts, for instance. Nonetheless, for many Acacia team members as well as the host institutions, managing multi-country networks was new and difficult and what Acacia could have developed was a toolkit to provide support. Another tension is between a) building the research capacity of researchers or b) building the administrative or research director capacity. In addition to these issues, the team considered the potential to explore further what might be done to overcome weaknesses and build on strengths. In the light, however, of the overall strength of the Approach and the desire to understand the achievements of the research networks in time for the Acacia Learning Forum in October, the team decided to put the Approach evaluation aside and undertake the second phase – an outcomes evaluation. Conclusions To what extent is the Acacia networking approach for programming appropriate and relevant to the needs of African researchers, including women, and research institutions to develop research capacity in