Shaping International Evaluation A 30‐Year Journey Edited by Gary Anderson Montreal and Ottawa CANADA UNIVERSALIA MANAGEMENT GROUP 5252 de Maisonneuve Blvd. W., Suite 310 Montreal, Quebec H4A 3S5 CANADA www.universalia.com Copyright © 2010 by Universalia Management Group. All rights reserved. No part of this publication may be reproduced, stored in a retrieval system or transmitted in any form or by any means – electronic, mechanical, photocopying or otherwise – without the prior permission of Universalia Management Group. Canadian Cataloguing in Publication Data Main entry under title: Shaping International Evaluation: A 30‐Year Journey ISBN 0‐9739343‐2‐8 Copy editor: Tracy Wallis Cover photo: Marie‐Hélène Adrien Printed in Canada This publication is printed on recycled paper. CONTENTS SECTION I Introduction Chapter 1: Shaping International Evaluation 3 SECTION II Cross‐Sectoral Trends Influencing International Evaluation 1980‐2010 Chapter 2: Something Called Capacity: The Evolution of an Idea in International Development 19 Chapter 3: Results‐Based Management 45 Chapter 4: Evaluating Partnerships in the Not‐for‐Profit Sector 75 Chapter 5: The Evolution of Institutional and Organizational Assessment 113 SECTION III Evaluation Trends in Specific Sectors and Regions Chapter 6: Evaluating Agricultural Systems 159 Chapter 7: Changing Perceptions of the Environment and Environmental Evaluation 205 Chapter 8: Evaluating ICT: A New Dress for Old Questions 239 Shaping International Evaluation v Chapter 9: Evaluating Gender Equality: Universalia Experiences and Perspectives 265 Chapter 10: How M&E is ‘Perceived’ and ‘Applied’ in the MENA Region 295 SECTION IV Conclusion Chapter 11: Conclusion 317 Contributors 329 Index 335 vi Shaping International Evaluation SECTION I Introduction CHAPTER ONE Shaping International Evaluation Gary Anderson, Marie‐Hélène Adrien, and Charles Lusthaus Universalia began in 1980, in an era when Microsoft and Apple were less than five years old. There were no cell phones or fax machines, and the worldwide web had not yet been conceived. It was a time when secretaries typed evaluation reports, though computers were beginning to be introduced for word processing as well as their traditional role in statistical calculation. Universalia consisted of Charles Lusthaus and Gary Anderson, the two founders, Gerry Cooney, our first professional employee, and a receptionist / secretary / bookkeeper. Few organizations of any kind can boast that the first three paid professionals are still engaged with the organization 30 years later – perhaps not even Microsoft and Apple, though they have strengths that Universalia has yet to attain! Evaluation has evolved considerably since Universalia was founded 30 years ago. Then, evaluation was just beginning to organize as a discipline. The Canadian Evaluation Society was not formed until 1981, and the American Evaluation Association came into being only in 1986. Universalia is proud to have been part of the development of the evalua‐ tion field in general, and the international development Shaping International Evaluation 3 One | Shaping International Evaluation evaluation field more particularly. To celebrate our 30th Anniversary, we invited a number of influential develop‐ ment writers and evaluators to join our celebration by contributing a chapter to this book. These contributions have allowed us to celebrate our 30th anniversary in the Univer‐ salia spirit of “theory and practice”, and to share our thoughts along with our exceedingly busy colleagues. We feel very honoured and privileged to have their original contributions and we thank them all! Introduction to the Book The book has two main sections. The first deals with major trends that have influenced international evaluation, and begins with Peter Morgan’s thoughts on capacity develop‐ ment, a dominant theme in development agencies, but as Peter points out, a hollow theme in international agree‐ ments, and one whose fuzziness creates great issues for the evaluator. Ken Stephenson follows with his analysis of Results‐Based Management, another major influence on international evaluation. Born in the same timeframe as Universalia was created, RBM supported the rising tide of conservatism espoused at the time by Margaret Thatcher in the UK and Ronald Reagan in the USA. However, like the context in which it arose, RBM has limitations in practice. The following chapter deals with another strong interna‐ tional development influence, “partnerships”. Charles Lusthaus, Katherine Garven, and Silvia Grandi have recently conducted substantive work in this area, so Chapter 4 enables them to share what they have learned. Finally, in response to the limitations of a “project” mentality, and in recognition of the importance of developing organizations and institutions, in Chapter 5, Katrina Rojas and Charles 4 Shaping International Evaluation Shaping International Evaluation | One Lusthaus consider how such structures are evaluated. This is a lasting contribution of Universalia’s work over more than two decades, and provides a logical conclusion to the first main section. The second substantive section considers the evolution of evaluation within specific sectors. We admit that the choice of sectors was largely arbitrary, but has worked out well. From the days of the Colombo Plan, agriculture has been seen as key to international development, so we begin the second section with Ronald Mackay and Doug Horton’s chapter on evaluation related to agricultural systems. It is perhaps fair to say that these authors have long been ahead of thinking in the field they evaluate, so they are ideal candidates to share their insights. Nowadays, environment is an important theme in development circles, and we are pleased to have Ramon Pérez Gil share his insights on the evolution of environment as a dominant international force as well as trends in evaluating environmental impact. Two additional “sectors” are included, Information and Commu‐ nication Technology (ICT) and Gender Mainstreaming. Ricardo Gomez and Shaun Pather look at ICT, a new, dynamic, and exceedingly important theme in our modern world, while Katherine Garven, Katrina Rojas, Anette Wenderoth, and Elisabetta Micaro consider trends in gender mainstreaming and the evaluation response. We conclude the second section with Doha Abdelhamid’s insights on development in general, and particularly developing evaluation capacities in the Middle East and North Africa. That contribution nicely melds the concepts of capacity, organizational development, and the dominant themes of the book. Shaping International Evaluation 5 One | Shaping International Evaluation Reflections on What is Ahead In considering the theme of capacity development, Peter Morgan introduces the notion that development agencies latch on to fads and adopt a passionate pursuit of such slogans, which in themselves are abstract and very difficult to define. While definitions are challenging enough for native speakers of English, understanding the meaning of capacity development may be an insurmountable problem for speakers of other languages. This, of course, introduces even more challenges for the evaluator. If people aren’t sure what it is, and cannot define it, it is next to impossible for the evaluator to ascertain whether an organization has it or not! Furthermore, if “capacity” is really “capability”, then it represents a potential for performance rather than current performance. Thus, the evaluator rapidly crosses into the sphere of predictive science – a domain fraught with risks and uncertainties. Morgan also points out the demands of increasingly com‐ plex development problems such as climate change for systems thinking and solutions. The linear forms of log‐ frames and cause‐and‐effect thinking simply do not fit with the need to understand complex, loosely coupled webs of interactions within equally complex enabling environments. As well, there may be a huge gulf between Western donor views of capacity and what people in developing countries value and understand. The enhancement of community engagement through elder’s circles or increasing the reach of traditional medicine healers may be of great importance to villagers, but neither are understood nor valued by devel‐ opment agencies. The indigenous systems of rewards and cultural patterns governing how those systems work are 6 Shaping International Evaluation Shaping International Evaluation | One essential. Evaluators seldom consider these informal systems. No matter how we approach capacity development, evalua‐ tion has yet to accommodate the hard to measure human attributes that are the real determinants of success or failure – such attributes as drive, will, persistence, leadership, adaptability, and inspiration. Educators often marvel at the success of a student who on the things measured by school examinations was at best average, but who goes on in life with determination and passion that enables him or her to become a major leader and contributor to great enterprise. Trying to emulate science, evaluators sometimes try to control “extraneous” variables, or ignore them in their analyses; however, when these are the primary contributors to success or failure, such an approach is folly. In organiza‐ tional performance terms, it is what Universalia has termed “motivation” in the organizational performance framework. That has always been a significant contributor to perform‐ ance, and it has always been and remains the most difficult aspect to assess. In international