ear! the Celebrating Our 10th Y evaluation exchange A PERIODICAL ON EMERGING STRATEGIES IN EVALUATING CHILD AND FAMILY SERVICES HARVARD FAMILY RESEARCH PROJECT ■ HARVARD GRADUATE SCHOOL OF EDUCATION ■ VOL. IX NO. 4 ■ WINTER 2003/2004 from the d i r e c t o r ’ s d e s k ith this issue, The Evaluation practices we may need to rethink and, in addressing the impor- Exchange kicks off its tenth year tance of learning from our progress and success, introduce a W of publication. In our very first theme that emerges several times in this issue. issue, I said that the success of this resource Several articles offer additional thoughts about recent devel- would depend on your willingness to be an opments. Michael Scriven offers his perspective on the status of active participant. A decade later, I want to the evaluation profession and discipline. Other articles present thank you for heeding that call in ways nominations for the “best of the worst” evaluation practices, that have exceeded our expectations. emerging links between program evaluation and organization Thank you to the development, and some surprising find- Heather Weiss hundreds of authors ings about changes in university-based who have generously evaluation training. shared their experiences and thoughts, to in this issue Building on these and our own reflec- the thousands of subscribers who have tions, this issue also introduces topics that read and applied the content, and to the Reflecting on the Past and future issues will address in more depth. generous funders that have supported the Future of Evaluation While our basic format and approach will production and free distribution of The remain the same, we have included articles Evaluation Exchange. Finally, I want to that herald our commitment to covering thank my staff for remaining committed Theory & Practice themes we think require more attention in to The Evaluation Exchange and to grow- Experts recap the last decade the evaluation arena—diversity, interna- ing its value. in evaluation 2 tional evaluation, technology, and evalua- Rarely does a nonprofit publish a peri- Ask the Expert tion of the arts. Upcoming issues will fea- odical that sustains beyond its inaugural Michael Scriven on ture and spur dialogue about these topics issues. We are extremely proud that The evaluation as a discipline 7 and others, including program theory, Evaluation Exchange has evolved into a Promising Practices mixed methods, democratic approaches to nationally known resource, with a grow- Evaluators’ worst practices 8 evaluation, advocacy and activism, ac- ing and diverse audience of more than countability, and systems change, which Efforts to link evaluators 13,000 evaluators, practitioners, policy- worldwide 12 was the topic of our first issue 10 years makers, and funders. ago and remains a significant evaluation Narrative methods and While we want to celebrate this tenth- organization development 14 challenge today. year milestone, we know that we are oper- Like the evaluation field itself, much ating in a dynamic environment with ever- Questions & Answers has changed about The Evaluation Ex- changing demands and appetites for new A conversation with change. But our success still depends on Ricardo Millett 10 ideas presented in new ways. We can’t stop the participation of our readers. If you to celebrate too long—we need to be re- Beyond Basic Training have ideas about other topics you would flecting constantly on our field and practice Evaluation training trends 13 like to see featured, please don’t hesitate to so we can continuously improve our work. Book Review share them with us. We continue to wel- Accordingly, we have dedicated this is- The Success Case Method 16 come your feedback and contributions. sue to sharing some of the lessons that will inform our agenda in the future. We begin Spotlight in our Theory & Practice section with a se- Evaluation to inform learning technology policy 17 ries of reflections by renowned experts on what the past decade has meant for evalu- Evaluations to Watch Heather B. Weiss, Ed.D. ation. These essays point to areas where Evaluating a performing and Founder & Director we have not come far enough, identify visual arts intervention 18 Harvard Family Research Project New Resources From HFRP 19 > theory practice the evaluation & exchange Where We’ve Been and Where We’re Going: Experts Reflect and Look Ahead In this special edition of Theory & Practice, six evaluation experts share their thoughts on how the field has progressed (or regressed) in the last 10 years and con- sider what the next steps should be. rticles that appear in Theory & Practice occupy an important position and Founder & Director mission in The Evaluation Exchange. Tied directly to each issue’s theme, they Heather B. Weiss, Ed.D. Alead off the issue and provide a forum for the introduction of compelling new ideas in evaluation with an eye toward their practical application. Articles iden- Managing Editor tify trends and define topics that deserve closer scrutiny or warrant wider dissemina- Julia Coffman tion, and inspire evaluators and practitioners to work on their conceptual and meth- odological refinement. HFRP Contributors An examination of the topics covered in Theory & Practice over the last decade Heather B. Weiss, Ed.D. reads like a chronicle of many of the major trends and challenges the evaluation pro- Julia Coffman fession has grappled with and advanced within that same timeframe—systems change, Tezeta Tulloch the rise of results-based accountability in the mid-1990s, advances in mixed methods, learning organizations, the proliferation of complex initiatives, the challenges of evalu- Publications/Communications ating communications and policy change efforts, and the democratization of practices Manager in evaluation methodology, among many others. Stacey Miller As this issue kicks off the tenth year of publication for The Evaluation Exchange, Harvard Family Research Project is devoting this section to a discussion of some of the Publications Assistant trends—good and bad—that have impacted the theory and practice of evaluation over Tezeta Tulloch the last 10 years. ©2004 President & We asked six experts to reflect on their areas of expertise in evaluation and respond Fellows of Harvard College to two questions: (1) Looking through the lens of your unique expertise in evaluation, how is evaluation different today from what it was 10 years ago? and (2) In light of Published by your response, how should evaluators or evaluation adapt to be better prepared for Harvard Family Research Project Harvard Graduate School of Education the future? All rights reserved. This periodical On Theory-Based Evaluation: may not be reproduced in whole or in part without written permission Winning Friends and Influencing People from the publisher. Carol Hirschon Weiss Professor, Harvard University Graduate School of Education, Cambridge, Massachusetts Harvard Family Research Project gratefully acknowledges the support One of the amazing things that has happened to evaluation is that it has pervaded the of the Annie E. Casey Foundation, program world. Just about every organization that funds, runs, or develops programs W. K. Kellogg Foundation, and now calls for evaluation. This is true locally, nationally, and internationally; it is almost John S. and James L. Knight Foundation. The contents of this as true of foundations and voluntary organizations as it is of government agencies. publication are solely the responsibility The press for evaluation apparently arises from the current demand for accountability. of Harvard Family Research Project Programs are increasingly called on to justify their existence, their expenditure of and do not necessarily reflect funds, and their achievement of objectives. Behind the calls for accountability is an the view of our funders. awareness of the gap between almost unlimited social need and limited resources. The Evaluation Exchange accepts An exciting development in the last few years has been the emergence of evaluations query letters for proposed articles. based explicitly on the theory underlying the program. For some time there have been See our query guidelines on our website exhortations to base evaluations on program theory. I wrote a section in my 1972 at www.gse.harvard.edu/hfrp/eval/ submission.html textbook urging evaluators to use the program’s assumptions as the framework for evaluation.1 A number of other people have written about program theory as well, in- To request a free subscription to cluding Chen,2 Rossi (with Chen),3 Bickman,4 and Lipsey.5 The Evaluation Exchange, email us at [email protected] or 1 Weiss, C. H. (1972). Evaluation research: Methods of assessing program effectiveness. Englewood Cliffs, call 617-496-4304. NJ: Prentice-Hall. 2 Chen, H. T. (1990). Issues in constructing program theory. New Directions for Program Evaluation, 47, 7–18. Harvard Family Research Project 2 The Evaluation Exchange IX 4 > theory& practice For a while, nothing seemed to happen; evaluators went A second reason has to do with complex programs where about their business in accustomed ways—or in new ways that randomized assignment is impossible. In these cases, evaluators had nothing to do with program theory. In 1997 I wrote an ar- want some way to try to attribute causality. They want to be ticle, “How Can Theory-Based Evaluation Make Greater Head- able to say, “The program caused these outcomes.” Without way?”6 Now theory-based evaluation seems to have blossomed randomized assignment, causal statements are suspect. But if forth. A number of empirical studies have recently been pub- evaluators can show that the program moved along its expected lished with the words “theory-based” or “theory-driven” in the sequence of steps, and that participants responded in expected titles (e.g., Crew and Anderson,7 Donaldson and Gooler8). The ways at each step of the process, then they can claim a reason- evaluators hold the program up against the explicit claims and able approximation of causal explanation.