DOCSLIB.ORG

Re-Examining the Implications of Systems Thinking for Evaluation

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
Re-Examining the Implications of Systems Thinking for Evaluation RE-EXAMINING THE IMPLICATIONS OF SYSTEMS THINKING FOR EVALUATION BY EMILY F. GATES DISSERTATION Submitted in partial fulfillment of the requirements for the degree of Doctor of Philosophy in Educational Psychology in the Graduate College of the University of Illinois at Urbana-Champaign, 2016 Urbana, Illinois Doctoral Committee: Professor Jennifer C. Greene, Chair Professor Thomas A. Schwandt, Director of Research Professor Lizanne DeStefano Professor Martin Reynolds, Open University ABSTRACT Over the last twenty years, many prominent evaluators have been borrowing and using ideas, theories, and methods from the systems and complexity fields with little research on the implications of this trend for the evaluation field. This thesis examines this borrowing to identify over-arching implications for evaluation theory and practice. The first paper reviews inter- disciplinary literature on systems thinking and complexity science with regards to evaluating social policies and programs and identifies major implications for how evaluators theorize evaluation practice. The second paper reports on an analysis of eight cases of evaluation practice that use systems and complexity ideas and techniques and presents findings regarding how evaluators conceive of and practice evaluation. The third paper advances an argument for how evaluators can use critical systems heuristics to surface, reflect on, and make explicit the values that influence and should influence an evaluation. Collectively, these papers support the potential of borrowing from the systems and complexity fields to expand on and re-define evaluation theory and practice. ii ACKNOWLEDGMENTS This dissertation would not have been possible without the mentorship of Dr. Thomas Schwandt and Dr. Jennifer Greene. I am thankful to Tom for overseeing this dissertation, sharing his wisdom, providing generous editorial comments, and continuously challenging and expanding my thinking. I am thankful to Jennifer for her early and ongoing encouragement of this research, for being a supportive advisor throughout the years, and for continuously inspiring me to be a more thoughtful and compassionate scholar and evaluator. I would also like to thank Dr. Lizanne DeStefano for fostering my development as an evaluator, Dr. Martin Reynolds for paving a way in critical systems and evaluation, and Bob Williams for sharing his brilliant enthusiasm for evaluation and systems thinking – and to all five for investing significant time and energy in this project as committee members. I want to express my gratitude to the eight evaluators I interviewed for their time, insights, and leadership in the field. Additionally, I want to acknowledge the University of Illinois Graduate College for providing funding to complete this dissertation. I could not have accomplished this project without the support of my colleagues, friends, and family. My thanks to my QUERIES, I-STEM, WIDER, and ISSS Ph.D. colleagues for providing motivation, feedback, and a sense of community, to Carolina (for friendship from day one), Ayesha (for nudging me to come to UIUC and stick with it), Gabi (for sharing the ups and downs), Sarai (for reminding me to smile), Megan-Brette (for reminding me of the reasons we do this), Lorna (for providing a space to unwind), Priya (for your refreshing visits), Molly (for your intellectual companionship), Lisa (for giving me candles to light the way), and to Mbikayi (for listening to my ramblings and providing constant encouragement). Last but not least, I want to thank my parents, Gerry and Cindy, for their unconditional emotional and financial support and Tyler, Austin, and especially Carlye for being there in more ways than I can name. iii TABLE OF CONTENTS INTRODUCTION ........................................................................................................................... 1 REFERENCES .................................................................................................................... 7 PAPER 1: IMPLICATIONS OF SYSTEMS THINKING AND COMPLEXITY SCIENCE FOR EVALUATION .................................................................................................................... 10 REFERENCES .................................................................................................................. 39 PAPER 2: BORROWING KNOWLEDGE FROM THE SYSTEMS AND COMPLEXITY FIELDS: IMPLICATIONS FOR EVALUATION PRACTICE ................................................... 47 REFERENCES .................................................................................................................. 94 PAPER 3: VALUING IN EVALUATION WITH CRITICAL SYSTEMS HEURISTICS ....... 102 REFERENCES ................................................................................................................ 135 CONCLUSION ........................................................................................................................... 141 REFERENCES ............................................................................................................................ 144 APPENDICES ............................................................................................................................. 145 Appendix A: Interview Guide ......................................................................................... 145 Appendix B: Case Materials ............................................................................................ 147 iv INTRODUCTION Throughout the development of the evaluation field, evaluators have borrowed knowledge from other fields and disciplines. Consider, for example, the intersection of anthropology and evaluation to better understand the contexts of evaluands (Fetterman, 1984; Butler & Copeland-Carson, 2005); contributions of organizational theories of learning to strengthen evaluation use (Russ-eft & Preskill, 2009); political philosophy for envisioning the roles of evaluation in democracy (House & Howe, 1999); and econometric methods for assessing resource use and value (Yates, 2012). In the last fifteen years, many prominent evaluators have been borrowing ways of thinking (i.e. ideas, concepts, theories) and techniques (i.e. methods, tools, approaches) from the systems and complexity fields--an interdisciplinary area of inquiry including operational research, systems science, complexity science, and cybernetics that broadly focuses on the study of systems and the design of interventions to address problematic situations (Eoyang, 1999; Sanderson, 2000; Barnes et al, 2003; Williams and Imam, 2007; Callaghan, 2008; Rogers, 2008; Morell, 2010; Forss, Marra, & Schwartz, 2011; Patton, 2011; Westhorp, 2012; Byrne, 2013). One significant motivation for this borrowing arises from criticisms of longstanding tendencies to define the objects of evaluation by pre-determined objectives and rather linear and mechanistic processes and to study effects primarily as intended changes in individuals. Many evaluators as well as evaluation commissioners are arguing, to the contrary, that initiatives that address persistent, interconnected issues (often called wicked problems) operate within continually changing circumstances influenced by many factors at multiple scales and amidst differing perspectives, values, and interests. For the evaluation field to be relevant and useful to both stakeholders and to the wider public affected by such initiatives, a turn to the systems and complexity fields is needed (Stern, Saunders, & Stame, 2015). 1 However, at present, the significance and implications of this turn are not yet clear (Mowles, 2014; Stern et. al, 2015). The literature includes examinations of various theories and ideas for evaluation (Sanderson, 2000; Barnes et. al, 2003; Davies, 2004; Rogers, 2008; Callaghan, 2008; Hummelbrunner, 2011; Byrne, 2013; Marra, 2015); new ways to conduct evaluations using systems and complexity concepts (Morell, 2010; Patton, 2011; Forss, Marra, and Stern, 2011; Levin-Rozalis, 2014); cases of evaluations that used systems and complexity thinking and techniques (Williams & Imam, 2007; Patton, McKegg, and Weipeihana, 2015; Fredericks, Deegan, & Carman, 2008; Dyehouse, Bennett, Harbor, & Childress, 2009; Morell, Hilscher, Magura, & Ford, 2010; Durland & Fredericks, 2005; Reynolds & Williams, 2011); and literature reviews (Mowles, 2014; Walton, 2014). Additionally, there has been considerable research in neighboring fields, including public health, social services, community psychology, and international aid and development, that addresses the implications of systems and complexity thinking and techniques for designing, implementing, and evaluating policies and programs (e.g., Milstein, 2008; Ramalingham, 2013; Foster-Fishman, Yang, & Nowell, 2007). Much of this research has focused on ‘hard’ (i.e., quantitative, realist) traditions and largely overlooked ‘soft’ (i.e. qualitative, dialogic) and ‘critical’ (i.e., mixed methods, pragmatic) traditions within the systems and complexity fields. There is a need for broad reviews of this research to identify both current implications for evaluation theory and practice and future directions and to explore potential contributions of lesser-examined traditions. 2 Overview of the Three Papers The three papers in this thesis conceptually and empirically examine the implications of borrowing from the systems and complexity fields for evaluation theory and practice. The first paper reviews recent literature on systems thinking and complexity science in evaluation and related
Load more

Recommended publications
  • Soft Systems Methodology
    Soft Systems Methodology An Introduction Computer Science/Roskilde University, Spring 1994 Jesper Simonsen Soft Systems Methodology......................................................................................1 Background...........................................................................................................1 SSM 1981 ...............................................................................................................2 Stage 1 and 2..............................................................................................3 Stage 3........................................................................................................4 Stage 4........................................................................................................5 Stage 5........................................................................................................6 Stage 6 and 7..............................................................................................7 SSM 1990 ...............................................................................................................7 The Stream of Logic-Based Enquiry........................................................9 The Stream of Cultural Enquiry ............................................................11 SSM and the Construction of IS ........................................................................12 Table of Key Concepts and Techniques/Guidelines.......................................14 The Philosophy of SSM......................................................................................14
    [Show full text]
  • Systems Thinking Is the Process of Understanding How Things Influence One Another Within a Whole
    Systems Thinking Systems thinking is the process of understanding how things influence one another within a whole. In nature, systems thinking examples include ecosystems in which various elements such as air, water, movement, plants, and animals work together to survive or perish. In organizations, systems consist of people, structures, and processes that work together to make an organization healthy or unhealthy. Systems Thinking has been defined as an approach to problem solving, by viewing "problems" as parts of an overall system, rather than reacting to specific part, outcomes or events and potentially contributing to further development of unintended consequences. Systems thinking is not one thing but a set of habits or practices [1] within a framework that is based on the belief that the component parts of a system can best be understood in the context of relationships with each other and with other systems, rather than in isolation. Systems thinking focuses on cyclical rather than linear cause and effect. In science systems, it is argued that the only way to fully understand why a problem or element occurs and persists is to understand the parts in relation to the whole. [2] Standing in contrast to Descartes's scientific reductionism and philosophical analysis , it proposes to view systems in a holistic manner. Consistent with systems philosophy , systems thinking concerns an understanding of a system by examining the linkages and 1 / 8 Systems Thinking interactions between the elements that compose the entirety of the system. Science systems thinking attempts to illustrate that events are separated by distance and time and that small catalytic events can cause large changes in complex systems .
    [Show full text]
  • 'Maintaining Systems-Of-Systems Fit-For-Purpose'
    SW Hinsley FFP_thesis_iss_2_2_1.doc ‘Maintaining Systems-of-Systems Fit-For-Purpose’ A technique exploiting Material, Energy and Information Source, Sink and Bearer Analysis by Steven W. Hinsley A Doctoral Thesis Submitted as partial fulfilment of the requirements for the award of Ph.D. of Loughborough University Submitted September 2016 Revised with Corrections April 2017 © by Steven W. Hinsley 2017 Wolfson School of Mechanical, Electrical and Manufacturing Engineering Engineering Systems-of-Systems Group SW Hinsley FFP_thesis_iss_2_2_1.doc Wolfson School of Mechanical, Electrical and Manufacturing Engineering Engineering Systems-of-Systems Group SW Hinsley FFP_thesis_iss_2_2_1.doc Wolfson School of Mechanical, Electrical and Manufacturing Engineering Engineering Systems-of-Systems Group SW Hinsley Abstract Abstract Across many domains, systems suppliers are challenged by the complexity of their systems and the speed at which their systems must be changed in order to meet the needs of customers or the societies which the systems support. Stakeholder needs are ever more complex: appearing, disappearing, changing and interacting faster than solutions able to address them can be instantiated. Similarly, the systems themselves continually change as a result of both external and internal influences, such as damage, changing environment, upgrades, reconfiguration, replacement, etc. In the event of situations unforeseen at design time, personnel (for example maintainers or operators) close to the point of employment may have to modify systems in response to the evolving situation, and to do this in a timely manner so that the system and/or System-of- Systems (SoS: a set of systems that have to interoperate) can achieve their aims. This research was motivated by the problem of designing-in re-configurability to the constituent systems of a SoS to enable the SoS and its systems to effectively and efficiently counter the effects of unforeseen events that adversely affect fitness-for–purpose whilst operational.
    [Show full text]
  • Systems Thinking in the Forest Service: a Framework to Guide Practical Application for Social-Ecological Management in the Enterprise Program
    Portland State University PDXScholar Dissertations and Theses Dissertations and Theses 10-27-2016 Systems Thinking in the Forest Service: A Framework to Guide Practical Application for Social-Ecological Management in the Enterprise Program Megan Kathleen Kmon Portland State University Let us know how access to this document benefits ouy . Follow this and additional works at: http://pdxscholar.library.pdx.edu/open_access_etds Part of the Forest Management Commons, and the Management Information Systems Commons Recommended Citation Kmon, Megan Kathleen, "Systems Thinking in the Forest Service: A Framework to Guide Practical Application for Social-Ecological Management in the Enterprise Program" (2016). Dissertations and Theses. Paper 3312. 10.15760/etd.3292 This Thesis is brought to you for free and open access. It has been accepted for inclusion in Dissertations and Theses by an authorized administrator of PDXScholar. For more information, please contact [email protected]. Systems Thinking in the Forest Service: A Framework to Guide Practical Application for Social-Ecological Management in the Enterprise Program by Megan Kathleen Kmon A thesis submitted in partial fulfillment of the requirements for the degree of Master of Science in Systems Science Thesis Committee: Wayne Wakeland, Chair Martin Reynolds Joe Fusion Portland State University 2016 Abstract The U.S. Forest Service (USFS) Enterprise Program (EP), which provides fee-for- service consulting services to the USFS, is interested in integrating systems thinking into its service offerings. Despite there being several excellent sources on the range and diversity of systems thinking, no single framework exists that thoroughly yet concisely outlines what systems thinking is along with its deep history, theoretical tenets, and soft and hard approaches.
    [Show full text]
  • Systems Thinking in the Forest Service: a Framework to Guide Practical Application for Social-Ecological Management in the Enterprise Program
    Portland State University PDXScholar Dissertations and Theses Dissertations and Theses Fall 10-27-2016 Systems Thinking in the Forest Service: a Framework to Guide Practical Application for Social-Ecological Management in the Enterprise Program Megan Kathleen Kmon Portland State University Follow this and additional works at: https://pdxscholar.library.pdx.edu/open_access_etds Part of the Forest Management Commons Let us know how access to this document benefits ou.y Recommended Citation Kmon, Megan Kathleen, "Systems Thinking in the Forest Service: a Framework to Guide Practical Application for Social-Ecological Management in the Enterprise Program" (2016). Dissertations and Theses. Paper 3312. https://doi.org/10.15760/etd.3292 This Thesis is brought to you for free and open access. It has been accepted for inclusion in Dissertations and Theses by an authorized administrator of PDXScholar. Please contact us if we can make this document more accessible: [email protected]. Systems Thinking in the Forest Service: A Framework to Guide Practical Application for Social-Ecological Management in the Enterprise Program by Megan Kathleen Kmon A thesis submitted in partial fulfillment of the requirements for the degree of Master of Science in Systems Science Thesis Committee: Wayne Wakeland, Chair Martin Reynolds Joe Fusion Portland State University 2016 Abstract The U.S. Forest Service (USFS) Enterprise Program (EP), which provides fee-for- service consulting services to the USFS, is interested in integrating systems thinking into its service offerings. Despite there being several excellent sources on the range and diversity of systems thinking, no single framework exists that thoroughly yet concisely outlines what systems thinking is along with its deep history, theoretical tenets, and soft and hard approaches.
    [Show full text]
  • Kent Academic Repository Full Text Document (Pdf)
    Kent Academic Repository Full text document (pdf) Citation for published version Mingers, John (2011) Soft OR Comes of Age - But Not Everywhere! Omega, 39 (6). pp. 729-741. ISSN 0305-0483. DOI https://doi.org/10.1016/j.omega.2011.01.005 Link to record in KAR http://kar.kent.ac.uk/28986/ Document Version Author's Accepted Manuscript Copyright & reuse Content in the Kent Academic Repository is made available for research purposes. Unless otherwise stated all content is protected by copyright and in the absence of an open licence (eg Creative Commons), permissions for further reuse of content should be sought from the publisher, author or other copyright holder. Versions of research The version in the Kent Academic Repository may differ from the final published version. Users are advised to check http://kar.kent.ac.uk for the status of the paper. Users should always cite the published version of record. Enquiries For any further enquiries regarding the licence status of this document, please contact: [email protected] If you believe this document infringes copyright then please contact the KAR admin team with the take-down information provided at http://kar.kent.ac.uk/contact.html Soft OR Comes of Age – But Not Everywhere! John Mingers Kent Business School University of Kent Canterbury Kent CT2 7PE Email: j.mingers@kent,ac,uk Phone: 01227 824008 Abstract Over the last forty years new methods and methodologies have been developed to deal with wicked problems or “messes”. They are structured and rigorous but non-mathematical. Prime examples are: soft systems methodology (SSM), cognitive mapping/SODA and the strategic choice approach (SCA).
    [Show full text]
  • Table of Contents
    Table of Contents Sponsors and Affiliated Organizations ............................................... Inside front cover Welcome Message, Professor Ockie Bosch, ISSS President ..................................... 3 Conference Schedule .................................................................................................. 5 List of Abstracts ......................................................................................................... 21 Plenary Abstracts ...................................................................................................... 31 Presentation/Paper Abstracts .................................................................................... 39 Poster Abstracts ........................................................................................................ 99 Workshop Abstracts ................................................................................................ 103 Author Index ........................................................................................................... 113 Keywords Index ....................................................................................................... 117 Sponsors Information .............................................................................................. 126 Conference Centre Map and Local Maps ................................................................ 128 ISBN: 978-1-906740-15-3 Copyright ISSS 2017 1 2 Welcome to ISSS 2017 Vienna is internationally recognized as the cradle of Systems Science
    [Show full text]
  • An Abstract of the Thesis Of
    AN ABSTRACT OF THE THESIS OF Seth Taylor for the degree of Master of Science in Industrial Engineering presented on August 23, 2019. Title: A Method for Measuring Systems Thinking Learning Abstract approved: _____________________________________________________ Javier Calvo-Amodio The myriad of problems facing the world today are increasingly complex, dynamic, and transcend multiple domains necessitating the need for an equally complex and transdisciplinary approach to solve these problems. Systems thinking promises to provide the skills necessary for all citizens of the world, not just the experts, to handle these types of problems. The challenge is fostering the awareness and understanding of systems thinking necessary to cultivate a systems-literate society. Systems literacy is a promising and ongoing effort to establish a common systems language among all people, which requires establishing both the concepts and the path necessary to reach systems literacy. Systems thinking is founded on a set of four underlying concepts, or skills, that every systems thinker uses (distinctions, systems, relationships, and perspectives). The systems thinking learning path follows a process comprised of three levels – sensibility (awareness of systems), literacy (knowledge of systems), and capability (understanding of systems) – repeated across multiple phases. Recent educational curriculum has been developed to directly and indirectly teach these concepts to initial learners, or non-experts. However, no method to measure whether these initial learners are learning the underlying systems thinking concepts according to this learning process has been attempted. Hence, this research defines and measures the initial systems thinking learning process for non-experts. An experiment was conducted with 97 middle and high school students from the Science and Math Investigative Learning Experiences (SMILE) Program at Oregon State University to measure initial learning using the four systems thinking concepts across the three systems thinking learning levels.
    [Show full text]
  • Soft Systems Methodology: a Thirty Year Retrospectivea
    Systems Research and Behavioral Science Syst. Res. 17, S11–S58 (2000) Ž Research Paper Soft Systems Methodology: A Thirty Year Retrospectivea Peter Checkland* 25 Pinewood Avenue, Bolton-le-Sands, Carnforth, Lancashire, LA5 8AR, UK INTRODUCTION But the fact that general systems theory (GST) has failed in its application does not mean that Although the history of thought reveals a number systems thinking itself has failed. It has in fact of holistic thinkers — Aristotle, Marx, Husserl flourished in several different ways which were among them — it was only in the 1950s that any not anticipated in 1954. There has been devel- version of holistic thinking became insti- opment of systems ideas as such, development of tutionalized. The kind of holistic thinking which the use of systems ideas in particular subject then came to the fore, and was the concern of a areas, and combinations of the two. The devel- newly created organization, was that which opment in the 1970s by Maturana and Varela makes explicit use of the concept of ‘system’, and (1980) of the concept of a system whose elements today it is ‘systems thinking’ in its various forms generate the system itself provided a way of cap- which would be taken to be the very paradigm turing the essence of an autonomous living sys- of thinking holistically. In 1954, as recounted in tem without resorting to use of an observer’s Chapter 3 of Systems Thinking, Systems Practice, notions of ‘purpose’, ‘goal’, ‘information pro- only one kind of systems thinking was on the cessing’ or ‘function’. (This contrasts with the the- table: the development of a mathematically ory in Miller’s Living Systems (1978), which expressed general theory of systems.
    [Show full text]
  • SYSTEMS THINKING a Dissertation, by Derek Anthony Cabrera, May 2006
    SYSTEMS THINKING A Dissertation Presented to the Faculty of the Graduate School of Cornell University in Partial Fulfillment of the Requirements for the Degree of Doctor of Philosophy by Derek Anthony Cabrera May 2006 © 2006 Derek Anthony Cabrera SYSTEMS THINKING Derek Anthony Cabrera, Ph.D. Cornell University 2006 This research set out to clarify the construct of systems thinking and to define it as a conceptual framework apart from systems science, systems theory, systems methods, and other perceived synonyms. Greater clarity in the systems thinking construct will assist any one of the many current implementation efforts in which systems thinking is being applied in both scientific disciplines and practical fields. One case of this is the application of systems thinking in public health. The challenges associated with this effort are generalizeable to any of the other fields in which systems thinking is being applied. The ambiguities of the systems thinking construct are central to the challenges people face in understanding and implementing systems thinking. This exploratory empirical research used structured conceptualization methodology, which mixes qualitative methods with multivariate statistical methods, to investigate the challenges of implementing systems thinking in an applied context. The analysis shows that: (1) the literature reveals that significant ambiguities exist about what constitutes systems thinking and that practitioners are adopting these ambiguities, (2) the methodological review reveals that there are a disproportionate
    [Show full text]
  • SYSTEMS CONCEPTS in EVALUATION an Expert Anthology
    SYSTEMS CONCEPTS IN EVALUATION An Expert Anthology EDITORS Bob Williams Iraj Imam INTRODUCTION FROM THE SERIES EDITOR It’s a pleasure to welcome readers to the first volume in a new series sponsored by the American Evaluation Association. The aim of the series is to make high quality work in evaluation available at a modest price that will make it easier for people in the United States and other countries to become aware of and keep up with this relatively new and fast-developing field. The Monograph Series was conceived of and will mainly consist of relatively brief single-author works, but will deviate from that model when the occasion arises. As it happened, an unusual opportunity made it possible to inaugurate the series with this very timely and well-staffed anthology. The series is overseen by the Publications Committee of the AEA, and supported by the Board of AEA, and was made possible as well as much improved by them, along with considerable help from the professional staff of the Association—most notably its Executive Director, Susan Kistler. This particular work also benefited greatly from its instigation and funding by the W. K. Kellogg Foundation, whose director of evaluation, Teri Behrens, was very helpful throughout; and from the extremely valuable comments of the two distinguished reviewers for AEA, Elliot Stern and William Trochim. The book content was designed by Rose Miller; it is being published for AEA through EdgePress of Inverness, California, whose indispensable Jeri Jacobson manages the logistics. Hard copies of this publication can be obtained from EdgePress PO Box 69 Point Reyes CA 94956, or Amazon (http://www.amazon.com) for $US18 soft cover and $36 hard cover.
    [Show full text]
  • A Conceptual Framework for Complex Systems at the Crossroads of Food, Environment, Health, and Innovation
    sustainability Article A Conceptual Framework for Complex Systems at the Crossroads of Food, Environment, Health, and Innovation Margherita Paola Poto Faculty of Law, UiT The Arctic University of Norway, 9010 Tromsø, Norway; [email protected]; Tel.: +47-77623145 Received: 26 October 2020; Accepted: 19 November 2020; Published: 20 November 2020 Abstract: Climate-smart solutions and practices have the potential to contribute to the UN Sustainable Development Goals (SDGs) of achieving zero hunger; ensuring healthy lives and promoting the wellbeing of humans, animals, and plants; reducing ocean overharvesting and overfishing; tackling climate change; driving economic growth; and promoting innovation. Achieving these goals will require searching for, defining, and adopting the most effective and suitable scientific approach for studying synergies between often-opposing socioeconomic and environmental priorities. Developing a critical conceptual framework as a reading key for the SDGs’ interactions (theory building) and exploring the possibilities of upscaling successful climate-smart practices, with the case study offered by the SECURE project (theory testing) are the two methods adopted to answer the research hypothesis on the validity and scope of conceptual frameworks for complex systems. The paper concludes with a call for further testing of tools, approaches, and methods to enable dynamic systems thinking to inform upscaling efforts, while recognizing the transdisciplinary nature and complexity of the study of low-trophic marine resources
    [Show full text]