SYSTEMS CONCEPTS IN EVALUATION An Expert Anthology EDITORS Bob Williams Iraj Imam INTRODUCTION FROM THE SERIES EDITOR It’s a pleasure to welcome readers to the first volume in a new series sponsored by the American Evaluation Association. The aim of the series is to make high quality work in evaluation available at a modest price that will make it easier for people in the United States and other countries to become aware of and keep up with this relatively new and fast-developing field. The Monograph Series was conceived of and will mainly consist of relatively brief single-author works, but will deviate from that model when the occasion arises. As it happened, an unusual opportunity made it possible to inaugurate the series with this very timely and well-staffed anthology. The series is overseen by the Publications Committee of the AEA, and supported by the Board of AEA, and was made possible as well as much improved by them, along with considerable help from the professional staff of the Association—most notably its Executive Director, Susan Kistler. This particular work also benefited greatly from its instigation and funding by the W. K. Kellogg Foundation, whose director of evaluation, Teri Behrens, was very helpful throughout; and from the extremely valuable comments of the two distinguished reviewers for AEA, Elliot Stern and William Trochim. The book content was designed by Rose Miller; it is being published for AEA through EdgePress of Inverness, California, whose indispensable Jeri Jacobson manages the logistics. Hard copies of this publication can be obtained from EdgePress PO Box 69 Point Reyes CA 94956, or Amazon (http://www.amazon.com) for $US18 soft cover and $36 hard cover. Your reactions to this book, and suggestions about other possible titles for the series would of course be much appreciated; after all, evaluators should practice what they preach! Michael Scriven [email protected] © 2006 American Evaluation Association ISBN 978-0-918528-22-3 (Paperback) 978-0-918528-21-6 (Hardback) CONTENTS ACKNOWLEDGEMENTS 1 INTRODUCTION 3 Systems Thinking for Evaluation 11 Gerald Midgley A Systemic Evaluation of an Agricultural Development: A Focus on the Worldview Challenge 35 Richard Bawden System Dynamics-based Computer Simulations and Evaluation 47 Daniel D Burke A Cybernetic Evaluation of Organizational Information Systems 61 Dale Fitch, Ph.D. Soft Systems in a Hardening World: Evaluating Urban Regeneration 75 Kate Attenborough Using Dialectic Soft Systems Methodology as an Ongoing Self-evaluation Process for a Singapore Railway Service Provider 89 Dr Boon Hou Tay & Mr Bobby, Kee Pong Lim Evaluation Based on Critical Systems Heuristics 101 Martin Reynolds Human Systems Dynamics: Complexity-based Approach to a Complex Evaluation 123 Glenda H Eoyang, Ph.D. Evaluating Farm and Food Systems in the US 141 Kenneth A Meter Systemic Evaluation in the Field of Regional Development 161 Richard Hummelbrunner Evaluation in Complex Governance Arenas: the Potential of Large System Action Research 181 Danny Burns Evolutionary and Behavioral Characteristics of Systems 197 Jay Forrest CONCLUDING COMMENTS 211 BIOGRAPHIES 215 ACKNOWLEDGEMENTS Producing this volume has been rather like running a medium-sized business. Indeed it would be a good systems case study on its own. We have tried to keep a collaborative ethos right through, so consequently the list of acknowledgements is extensive. Our thanks to the following folk and no doubt others we have forgotten; you know who you are. Robin Miller had the original idea. Michael Scriven and the American Evaluation Association provided the vehicle. Craig Russon made the link between the AEA and the Kellogg Foundation. Teri Behrens and the W K Kellogg Foundation supplied the grants that made the endeavour possible. The staff of the Center for Applied Local Research (CAL Research), especially Carol Coley, provided constant support. The authors went way beyond the call of duty many times, and so did those whose potential contributions we didn’t have space to include. The team of reviewers from all over the world looked at their allocated papers at least twice; Meenakshi Sankar, Leslie Goodyear, Hassan Qudrat-Ullah, Lee Mizell, Jacob Silver, Olaronke Ladipo, Samruddhi Thaker, Thomas Chapel, Gene Lyle, Shannon Lee, Chris High, Graham Smith, Bridget Roberts, Alison Amos, Bill Harris, Bob Dick, Cheyanne Church, Eileen Franco, Greg Yelland, John Sherman, John Smith, Mel Tremper, Mike Lieber, Lindsey Poole, Rainer Loidl-Keil, Robin Miller, Aisha Shah, Tracy Henderson, Tony Berkley, Melissa Weenink, and Yee Sue Lin. Our apologies if we have missed someone. CAL Research with help from Andie Williams of San Francisco Bay Area Evaluators (SFBAE) organized a two-day meeting between the authors, editors, and evaluators Beverley Parsons, Susan Hanson, Amy LaGoy, Tony Berkeley, and John Gargani. Ruhel Boparai took the notes. San Francisco Bay Area Evaluators (SFBAE) also hosted a dinner that allowed a whole bunch of SFBAE members to contribute to this publication. Bill Trochim and Elliot Stern reviewed the whole draft and raised the standard even higher. Derek Cabrera, Patricia Rogers, Doug Fraser, Tessie Catsambas, and Michael Patton provided vital comments and support. Contributors to the Eval-Syst discussion group kept the endeavor grounded. Rose Miller did the design and layout 1 INTRODUCTION Iraj Imam, Amy LaGoy, Bob Williams Here is a short history of an idea. We are somewhere in the 960s. The idea is an approach to describing and assessing a situation in a way that can inform appropriate choices for action and improvement. The idea gains political support and the grudging acceptance of the academic world. Some academics criticise it for lacking an overarching theory, but reluctantly allow it into their departments and schools, partly because it is popular and lucrative. Over time a network of consultants cluster around the idea, responding to the demands of clients who become interested in it. The idea becomes fashionable in many parts of the world. It moves from being an idea to a field of inquiry to, some would argue, a trans-disciplinelike statistics. But the idea is applied differently in each part of the world. There are arguments and splits. There are debilitating debates around about what is “real” and what is “perceived,” and the use of qualitative and quantitative data. These are partly resolved by those who argue the wise use of both. But the bruises persist. There are also debates over the role of the emerging field in empowering people, especially the disenfranchised. Over time the focus on a single stakeholder view of the situation is expanded to allow multiple stakeholder views. This emerging field is beset with communication difficulties and misunderstandings; the field has become so diverse that despite many attempts it defies a single definition or description. Is this a description of the evaluation field or the systems field? Theanswer is both. An intriguing aspect of the systems and evaluation fields is that they share many experiences, concepts, goals, even attitudes, yet know relatively little about each other. What each understands about the other is often crude and partial. Ask an evaluator about the systems field and you will probably hear aboutmodels, interconnections, and “holism,” but little more. Ask a systems practitioner about evaluation and you will hear about measurement, targets, and outcomes, and little else. These perceptions reflect only a small part of the two fields’ ngera of activity. They also fail to get to the core of what either field is about. Despite drawing on some of the same philosophical, sociological, and scientific developments in the latter 20th century, the two fields have operated virtually independently since their inceptions. In recent years, however, some systems practitioners have begun applying systems thinking to evaluation work. With contributions from this volume’s authors, as well as Tony Berkley, Derek Cabrera, Tessie Catsambas, Susan Hanson, Beverly Parsons, and Bill Trochim. USING SYSTEMS CONCEPTS IN EVALUATION And today there is growing interest among evaluators in what the systems field can offer them. During a lively session at the 2005 Joint American Evaluation Association and Canadian Evaluation Society conference, participants described what was exciting about the use of systems concepts in evaluation: • Makes you think differently. • Offers more effective ways of dealing with complexity and complex situations. • Links the local and global, across silos, sectors and disciplines. • Provides tools to work with different opinions of stakeholders. • Develops new ways of understanding situations. • Pays attention to coalitions. • Pays attention to properties that emerge unexpectedly. • Puts multiple projects and topics into comparable forms. • Acknowledges the richness and interdependence of real life. • Helps identify leverage points; the differences that make a difference to a program and signal where best to intervene. • Allows for measuring or accounting for dynamic changes in a program or system. • Provides practical guidelines for using theory-of-change techniques. • Recognizes the evolutionary nature of programs. The evaluators at that session clearly understood that systems concepts had something to offer their evaluation work. To help deepen that understanding, this publication addresses three questions: . What key systems concepts do evaluators need to know? 2. How can evaluation benefit from using systems concepts?