Evaluating Democracy Support Methods and Experiences Evaluating Democracy Support Methods and Experiences Editor: Peter Burnell Contributors: Harry Blair Héctor Chayer Sandra Elena Hanne Lund Madsen Natalia Mirimanova Patrick D. Molutsi Margaret J. Sarles Fredrik Uggla Michael Wodzicki © International Institute for Democracy and Electoral Assistance and Swedish International Development Cooperation Agency 2007 International IDEA publications are independent of specific national or political interests. Views expressed in this publication do not necessarily represent the views of International IDEA, its Board or its Council members. This publication has been co-financed by the Swedish International Development Cooperation Agency, Sida. Sida does not necessarily share the views expressed in this material. Responsibility for its contents rests entirely with the authors. Applications for permission to reproduce or translate all or any part of this publication should be made to: International IDEA SE - 103 34 Stockholm Sweden Sida SE - 105 25 Stockholm Sweden International IDEA encourages dissemination of its work and will promptly respond to requests for permission to reproduce or translate its publications. Cover graphic design by: Kristina Schollin-Borg Cover photo: Bengt Olof Olsson/Bildhuset/SCANPIX Printed by: Bulls Graphics Sweden ISBN: 978-91-85724-13-0 Preface Democracy support has grown dramatically in the past two decades, and so has interest in the methods and techniques of evaluating democracy support. It is often asserted that evaluation of democracy support differs from the evaluation of other areas of development cooperation. In particular, it has been noted that the former field faces problems that relate to the diverse conceptions and definitions of democracy and democratization; the complex nature of democratization processes; and the difficulty of attributing changes at the national political level to individual projects. Such difficulties form the theoretical setting for the chapters of the present volume. This book is based on the proceedings of a workshop on Methods and Experiences of Evaluating Democracy Support, organized by the International Institute for Democracy and Electoral Assistance (International IDEA) and the Swedish International Development Cooperation Agency (Sida), and held in April 006. The main aim of the workshop was to explore ways in which existing methods and techniques of evaluating democracy support deal with challenges of causality and attribution. Clearly, this book is nowhere near providing answers to these questions, nor is it intended to. Rather, IDEA and Sida seek to share the main deliberations of the workshop, to stimulate further debates on the subject of evaluating democracy support and the challenges it faces, and—most importantly—to contribute to any new conceptualizations of methods and techniques for evaluating democracy support. The workshop also aimed to bring together three different communities, using International IDEA’s convening power and Sida’s expertise—the community of evaluators, the community of democracy programme designers and planners, and the community of implementers and practitioners. It produced a rich debate and a meeting place for very different perspectives. We take this opportunity to thank Professor Peter Burnell for the excellent work he has done in editing this book, as well as contributing the introductory chapter, which sets the publication in context. We thank Eve Johansson, whose professional input improved the readability of this publication tremendously. We also thank Keboitse Machangana, Advisor for Democracy Analysis and Assessment at International IDEA, and Fredrik Uggla of the Department for Evaluation and Audit at Sida for working tirelessly to bring this book to fruition. Vidar Helgesen Eva Lithman International IDEA Sida Stockholm August 007 Contents Acronyms and abbreviations ........................................................................6 Chapter 1 ..........................................................................................................1 Methods and experiences of evaluating democracy support: a moving frontier Peter Burnell What is evaluation? ................................................................................................ 16 Why evaluate? ........................................................................................................ 17 Evaluation and participation ............................................................................ 18 Evaluation and avoiding failure ....................................................................... 20 What can be evaluated? .......................................................................................... 22 In the evaluators’ sights ................................................................................... 23 Results-based and programme theory evaluations ............................................ 26 Evaluation: lessons of experience ............................................................................ 27 Measuring democratic progress ........................................................................ 27 Quantitative and qualitative methods ............................................................... 29 Assigning consequences .................................................................................... 31 A look forward ....................................................................................................... 32 Introducing the chapters .................................................................................. 32 What happens after evaluation? .............................................................................. 37 Evaluation in perspective ........................................................................................ 42 Chapter 2 ..........................................................................................................7 Evaluating the impact and effectiveness of USAID’s democracy and governance programmes Margaret J. Sarles Introduction to the Strategic and Operational Research Agenda (SORA) ............... 48 The rationale for SORA ......................................................................................... 49 Earlier efforts: the Centre for Development Information and Evaluation and SORA, Stage 1 ................................................................................................ 53 Methodological findings ................................................................................... 54 Substantive findings ......................................................................................... 55 SORA, Stage ....................................................................................................... 56 Setting up a Democracy Database .......................................................................... 57 A worldwide quantitative study of USAID’s democracy impact .............................. 58 Democracy surveys as evaluation tools ................................................................... 61 Expert interviews: ‘Voices from the Field’ ............................................................... 6 SORA, Stage : the National Academy of Sciences and the future .......................... 6 6 Figure 2.1: USAID-managed democracy and governance programmes .....................50 Notes ..................................................................................................................67 Chapter 3 ...................................................................................................... 71 Programme theory evaluation and democracy promotion: reviewing a sample of Sida-supported projects Fredrik Uggla Introduction ......................................................................................................71 Focusing on programme theory ..........................................................................7 Evaluating programme theory ......................................................................7 Discerning programme theory ......................................................................7 The model of analysis ...................................................................................76 The countries studied .........................................................................................78 Comparing programme theories .........................................................................79 The actor chain ............................................................................................80 Mechanisms .................................................................................................8 Actors and mechanisms combined ...............................................................86 Lack of assumptions .....................................................................................87 General findings .................................................................................................89 Assumptions and arguments .........................................................................89 How are we to use the results? ............................................................................90 Conclusion.........................................................................................................91 Table 3.1: Programme theory model of analysis: a hypothetical example .............................77 Table 3.2: Number of projects involving different types