Smart Tools for Evaluating the Performance of Information Products and Services
Total Page:16
File Type:pdf, Size:1020Kb
Smart tools for evaluating the performance of information products and services Proceedings of a KIT/CTA/IICD/LEAP-IMPACT Technical Consultation, Amsterdam, The Netherlands, November 2002 CTA Working Document Number 8029 CTA’s working document series consists of material that, in view of its immediate relevance and practical utility to specific readerships, the Centre wishes to make available without the delays inherent in the formal publication process. These working documents have not yet undergone technical editing by CTA and should be cited accordingly. Comments on matters of substance are welcome, and should be addressed directly to CTA. Published and printed by: Technical Centre for Agricultural and Rural Cooperation (CTA) Compiled by: Karen Batjes-Sinclair Contents Summary................................................................................................................................ 1 Opening speeches .................................................................................................................9 Modupe Akande Research Professor, Obafemi Owolowo University.................................................11 Hans van Hartevelt, Head, KIT-ILS..........................................................................................................13 Sarah Cummings, KIT..................................................................................................................................16 Keynote addresses ...............................................................................................................21 Jan Donner, President, KIT ........................................................................................................................21 Carl B. Greenidge, Director, CTA..............................................................................................................25 Part 1 Toolkit papers ...........................................................................................................35 1.1 An evaluation framework for evaluating the performance and impact of information projects, products and services ........................................................................................... 37 Ibrahim Khadar, CTA ..................................................................................................................................37 1.2 Indicators: between hard measures and soft judgements ............................................. 46 Andreas Springer-Heinze, GTZ..................................................................................................................46 1.3 Production, validation, strategy and status of the toolkit .............................................. 53 Karen Batjes (Technical editor) and Sarah Cummings (KIT)................................................................53 Part 2 Working Group discussions...................................................................................... 63 2.1 Discussions on the toolkit.............................................................................................. 65 Part 3 CTA Resource book.................................................................................................. 75 The proposed CTA book on: Assessing the impact of information on development Kay Sayce , Publishing consultant ..............................................................................................................77 Part 4 Future of LEAP IMPACT: A review of allied initiatives .......................................... 83 4.1 LEAP: Lessons learned, recommendations for next time ............................................. 85 Allison Hewlitt, Bellanet...............................................................................................................................85 4.2 Support for online collaboration and communities at Bellanet..................................... 88 Katherine Morrow, Bellanet International Secretariat.............................................................................88 4.3 An introduction to LEAP IMPACT: A collaborative approach to developing evaluation practice in the information sector...................................................................... 92 Sarah Cummings, KIT..................................................................................................................................92 4.4 The FAO resource kit ...................................................................................................100 Byron Mook, ISNAR................................................................................................................................. 100 Part 5 Future direction of the toolkit, resource book and LEAP IMPACT .......................103 5.1 Future direction of the toolkit, resource book and LEAP IMPACT ............................105 Annexes .............................................................................................................................. 111 After action review (AAR).....................................................................................................112 Workshop programme ..........................................................................................................114 List of participants ................................................................................................................132 Acronyms.............................................................................................................................125 iii Smart tools for evaluating the performance of information products and services Summary Background Information practitioners increasingly face the challenge of having to manage and evaluate the projects in which they are involved, without having the necessary training and tools to do so. The complexity of issues involved makes that task even more daunting. Recognising this, participants of the CTA/LEAP IMPACT technical consultation on Assessing the performance and impact of agricultural information products and services, in Bonn (October 2001), recommended that further work should be done in the field to develop 'smart practices' which would facilitate the work of practitioners. Although the technical consultation in Bonn provided excellent material to further develop practical and cost-effective methods for evaluating agricultural information services, the focus tended to be concerned with measuring 'benefits' / impacts that the project has brought to farmers, agricultural marketing agencies and or scientists; as well as defining roles and relationships between output, outcome and impact. Further, a commitment was made to publish a resource book concerned with evaluation and impact assessment, which is currently underway. Less attention, however, was paid to the management and performance aspects of products and services, while issues of relevancy, efficiency, effectiveness and sustainability were only touched on. Consequent on this, in November 2002, KIT, CTA, and IICD organised a workshop on Smart tools for evaluating the performance of information products and services at KIT in Amsterdam, the Netherlands. Other LEAP IMPACT partners were also involved in the initiative, many of whom represented various organisations in African, Caribbean and the Pacific (ACP) states, Canada and Europe. The workshop aimed to facilitate the development of a toolkit for managers of information projects as well as provide a complement to the publication 'Evaluating information: A letter to the project manager' (Mook, 2001 – CTA Working Document 8025). In addition to this, it aimed to encourage practitioners to share information and experiences, leading to improvements in the tools and methods used to assess information products and services. The expected outputs were to: • Clarify key concepts and terminologies used in the management and performance of agricultural information products and services. • Validate the draft tools that will support information managers in their efforts to manage and evaluate the performance of specific types of information activities. • Develop a strategy for future development and refinement of the tools and publication of the toolkit after the workshop. • Select teams to develop and refine the additional tools and to draw up a timetable for development of the drafts. • Form a small Editorial Committee to further validate and refine the toolkit. 1 Smart tools for evaluating the performance of information products and services • Promote the LEAP IMPACT approach to a wider audience of information and development practitioners in the Public Seminar. • Draw conclusions about the future approach of the LEAP IMPACT community of practice in the Open Meeting. • Provide a forum for discussion of CTA’s impact resource book. The four-day workshop had three main elements: • The first two days were taken up discussing the toolkit, the direction it should take and amending/ writing the tools. • On the third day, there was an Open Session where specially invited guests spoke on issues concerning LEAP IMPACT. This allowed the participants to discuss the issues facing the community. The afternoon session culminated in an Open Meeting, which offered the three organisations (KIT, CTA, IICD) the opportunity to come together to discuss and debate general issues on the evaluation of information with NGOs and the public. • On the fourth day, a group of experts (drawn mostly from the participants) came together to work on the CTA resource book, while the participants