DOCSLIB.ORG

Using Theory of Change to Design and Evaluate Public Health Interventions: a Systematic Review Erica Breuer1*, Lucy Lee2, Mary De Silva2 and Crick Lund1

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
Using Theory of Change to Design and Evaluate Public Health Interventions: a Systematic Review Erica Breuer1*, Lucy Lee2, Mary De Silva2 and Crick Lund1 Breuer et al. Implementation Science (2016) 11:63 DOI 10.1186/s13012-016-0422-6 SYSTEMATIC REVIEW Open Access Using theory of change to design and evaluate public health interventions: a systematic review Erica Breuer1*, Lucy Lee2, Mary De Silva2 and Crick Lund1 Abstract Background: Despite the increasing popularity of the theory of change (ToC) approach, little is known about the extent to which ToC has been used in the design and evaluation of public health interventions. This review aims to determine how ToCs have been developed and used in the development and evaluation of public health interventions globally. Methods: We searched for papers reporting the use of “theory of change” in the development or evaluation of public health interventions in databases of peer-reviewed journal articles such as Scopus, Pubmed, PsychInfo, grey literature databases, Google and websites of development funders. We included papers of any date, language or study design. Both abstracts and full text papers were double screened. Data were extracted and narratively and quantitatively summarised. Results: A total of 62 papers were included in the review. Forty-nine (79 %) described the development of ToC, 18 (29 %) described the use of ToC in the development of the intervention and 49 (79 %) described the use of ToC in the evaluation of the intervention. Although a large number of papers were included in the review, their descriptions of the ToC development and use in intervention design and evaluation lacked detail. Conclusions: The use of the ToC approach is widespread in the public health literature. Clear reporting of the ToC process and outputs is important to strengthen the body of literature on practical application of ToC in order to develop our understanding of the benefits and advantages of using ToC. We also propose a checklist for reporting on the use of ToC to ensure transparent reporting and recommend that our checklist is used and refined by authors reporting the ToC approach. Keywords: Theory of change, Intervention development, Evaluation, Programme theory, Public health, Systematic review Background controlled trials (RCTs) [1]. Some researchers propose Most public health interventions are inherently complex, that understanding the public health intervention’s with multiple interacting components, delivered at mul- underlying theory of change (ToC) and its related uncer- tiple levels. This complexity makes them difficult to tainties may improve the evaluation of complex health evaluate using traditional experimental designs. Public interventions [1–3]. health interventions often rely on ongoing quality im- Theory-driven evaluation is a collection of evaluation provement based on the implementation experience. methods which emphasise the importance of under- Therefore, they may not reach the level of stability re- standing how and why a programme works in order to quired to conduct evaluations such as randomised evaluate it [4, 5]. By programme, we mean a set of orga- nised activities or interventions supported by resources * Correspondence: [email protected] designed to achieve a specific result [6]. The theories are 1Alan J. Flisher Centre for Public Mental Health, Department of Psychiatry and Mental Health, University of Cape Town, 46 Sawkins Road, Rondebosch, first made explicit and then used to see how the 7700 Cape Town, South Africa programme theory results in the intended outcomes [4]. Full list of author information is available at the end of the article © 2016 Breuer et al. Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated. Breuer et al. Implementation Science (2016) 11:63 Page 2 of 17 There are several overlapping types of theory-driven comes from a perspective of scientific realism and fo- evaluation approaches including logic models, logical cuses predominantly on the interaction between the frameworks, outcomes hierarchies, realist evaluation, context, mechanisms and outcomes of the programme. and ToC [4, 5]. Usually used post hoc, evaluators seek to uncover the ToC was developed by Weiss and others [7] within underlying programme theories. These theories are the tradition of theory-driven evaluation. Although often more abstract than the theories developed definitions of ToC vary, we define it as an approach through ToC or logic models [18]. The development of which describes how a programme brings about spe- ToC has been influenced by Freirean thinking on how cific long-term outcomes through a logical sequence to create social change by empowering individuals [19]. of intermediate outcomes [8]. The ToC is often devel- Despite some fundamental differences in their theoret- oped using a backward mapping approach which ical underpinnings, many of these approaches are used starts with the long-term outcome and then maps the interchangeably or together [18, 20]. required process of changeandtheshort-and ToC has been used widely in the development medium-term outcomes required to achieve this [9]. sector for programme development and evaluation by fun- During this process, the assumptions about what ders such as the UK’s Department for International needs to be in place for the ToC to occur are made Development, Comic Relief, Grand Challenges Canada explicit as well as the contextual factors which influ- and the Gates Foundation [3, 19]. However, there has ence the ToC. Additional elements of a ToC can in- been no global systematic review to our knowledge clude beneficiaries, research evidence supporting the on the use of ToC for the design and evaluation of ToC, actors in the context, sphere of influence, stra- public health interventions. Coryn et al. [4] conducted tegic choices and interventions, timelines and indica- a review of theory-driven evaluation more broadly. tors [8]. These elements are usually presented in a They found 45 examples of theory-driven evaluation diagram and/or narrative summary [8]. in the peer-reviewed literature between 1990 and The ToC is usually developed in consultation with 2009. These evaluations included education, crime stakeholders through workshops or interviews although and safety and transportation interventions. Roughly the participation of stakeholders can vary substantially in half (21/45) were evaluations of health interventions practice [10]. For example, some ToCs are developed [4]. A rapid analysis of the included papers in prepar- through a series of workshops and meetings with a wide ation for this review indicated that only three of these range of stakeholders including service users [11, 12] used ToC. whereas others are developed by evaluators and funders The lack of a systematic review means that there is using programme documentation [13, 14]. The resulting no clear idea of how the ToC has been used and re- ToCs can be used as a framework for programme devel- ported in the peer-reviewed and grey literature in rela- opment and evaluation [8]. The ToC approach is tion to public health interventions. Given the increasing method neutral and as such does not prescribe specific popularity of the ToC approach, understanding how it is types of evaluation methods such as qualitative inter- has been used and described previously allows future views or RCTs [15]. users of the approach to learn from the work of others ToC is distinct from sociological or psychological the- and build upon it. It also helps to move towards a more ories which describe why change occurs although these consistent way of using the ToC approach. may be used to inform the ToC [3]. For example, Bauer In this review, we sought to review both peer-reviewed used an ecological model of community organising to and grey literature to determine how ToCs have been inform a ToC for a capacity and advocacy initiative for developed and used in the development and evaluation residents to impact on public health policy and training of public health interventions globally. Specifically, we of public health professionals [16]. sought to answer the following questions: ToC differs from other theory-driven approaches to evaluation despite similar origins. For example, al- (1)How are ToCs for public health interventions though logic models outline the inputs, processes, developed and refined? outputs and outcomes of a programme in a similar (2)How is the ToC approach used in the manner to ToC, they can be rigid and do not make ex- (a) development of an intervention; plicit the causal pathways through which change hap- (b) implementation of the intervention; pens in the way that ToC does [3]. Similarly, although (c) development of indicators for measurement; logframes were initially developed to summarise discus- (d) evaluation of the intervention, including sions with stakeholders, funder-driven formats have statistical approaches; and largely reduced logframes to a results-based manage- (e) conceptualisation/evaluation of the influence of ment tool [17]. Realist evaluation, on the other hand, context. Breuer et al. Implementation Science (2016) 11:63 Page 3 of 17 Methods As the focus of this review was on public health inter- The authors developed a protocol for this review which ventions, we excluded papers in which the long-term was agreed prior to the commencement of the study. outcome of the ToC was a change within an individual This is available in Web Additional file 1. rather than change in the population. For example, a ToC describing how cognitive behavioural therapy may Inclusion and exclusion criteria impact on an individual’s cognitive processes and behav- The inclusion and exclusion criteria are listed in Table 1. iour would be a change within the individual.
Load more

Recommended publications
  • Theory of Change Primer
    Theory of Change Primer A STAP Advisory Document November 2020 LEAD STAP AUTHOR: Mark Stafford Smith STAP CONTRIBUTORS: Rosie Cooney, Graciela Metternicht, Blake Ratner, Jamidu Katima, Saleem Ali, and Rosina Bierbaum STAP SECRETARIAT CONTRIBUTORS: Christopher Whaley and Guadalupe Durón SUGGESTED CITATION: Stafford Smith, M. 2020. Theory of Change Primer, A STAP Advisory Document. Scientific and Technical Advisory Panel to the Global Environment Facility. Washington, D.C. COPYRIGHT: This work is shared under a Creative Commons Attribution-Non Commercial-No Derivative Works License. ABOUT STAP: The Scientific and Technical Advisory Panel (STAP) provides independent scientific and technical advice to the GEF on its strategies, programs and projects. https://stapgef.org ABOUT GEF: The Global Environment Facility (GEF) was established on the eve of the 1992 Rio Earth Summit to help tackle our planet’s most pressing environmental problems. Since then, the GEF has provided close to $20.5 billion in grants and mobilized an additional $112 billion in co-financing for more than 4,800 projects in 170 countries. Through its Small Grants Programme, the GEF has provided support to nearly 24,000 civil society and community initiatives in 133 countries. http://www.thegef.org COPY EDITOR: Emily Youers DESIGN AND LAYOUT: Phoenix Design Aid A/S, Denmark COVER PHOTOS: Ron Ramtang, Marius Dobilas, Marion Smith – Byers, Lucy Brown, Tati Nova photo Mexico, Jimmy Tran, Uwe Bergwitz, by VLADGRIN Theory of Change Primer A STAP Advisory Document November 2020 CONTENTS
    [Show full text]
  • Ids Working Paper
    IDS WORKING PAPER Volume 2017 No 499 Theories of Change for Promoting Empowerment and Accountability in Fragile and Conflict-Affected Settings Duncan Green October 2017 This paper is published in association with the Action for Empowerment and Accountability Research Programme (A4EA). Action for Empowerment and Accountability Research Programme In a world shaped by rapid change, the Action for Empowerment and Accountability Research programme focuses on fragile, conflict and violence affected settings to ask how social and political action for empowerment and accountability emerges in these contexts, what pathways it takes, and what impacts it has. A4EA is implemented by a consortium consisting of: the Institute of Development Studies (IDS), the Accountability Research Center (ARC), the Collective for Social Science Research (CSSR), the Institute of Development and Economic Alternatives (IDEAS), Itad, Oxfam GB, and the Partnership for African Social and Governance Research (PASGR). Research focuses on five countries: Egypt, Mozambique, Myanmar, Nigeria, and Pakistan. A4EA is funded by UK aid from the UK government. The views expressed in this publication do not necessarily reflect the official policies of our funder. Theories of Change for Promoting Empowerment and Accountability in Fragile and Conflict-Affected Settings Duncan Green IDS Working Paper 499 © Institute of Development Studies 2017 ISSN: 2040-0209 ISBN: 978-1-78118-405-9 A catalogue record for this publication is available from the British Library. This is an Open Access paper distributed under the terms of the Creative Commons Attribution Non Commercial 4.0 International licence, which permits downloading and sharing provided the original authors and source are credited – but the work is not used for commercial purposes.
    [Show full text]
  • Theory of Change
    THEORY OF CHANGE UNDAF CAMPANION GUIDANCE 1 ! UNDAF COMPANION GUIDANCE: THEORY OF CHANGE TABLE OF CONTENTS 1. INTRODUCTION .................................................................................................................................. 3 ................................................................................................................................. 2. THE CONCEPT 4 What is a theory of change? ............................................................................................................................. 4 Purpose: Why use a theory of change? .............................................................................................................. 4 Methodology: How to develop a theory of change? .............................................................................................. 4 Key principles for developing a theory of change ................................................................................................ 5 Key steps for developing a theory of change ...................................................................................................... 5 3. MOVING FROM THEORY TO PRACTICE ................................................................................. 6 Step 1: Focus ................................................................................................................................................ 7 Step 2: Change analysis .................................................................................................................................
    [Show full text]
  • Methodological Brief No.2: Theory of Change
    Methodological Briefs Impact Evaluation No. 2 Theory of Change Patricia Rogers UNICEF OFFICE OF RESEARCH The Office of Research is UNICEF’s dedicated research arm. Its prime objectives are to improve international understanding of issues relating to children’s rights and to help facilitate full implementation of the Convention on the Rights of the Child across the world. The Office of Research aims to set out a comprehensive framework for research and knowledge within the organization, in support of UNICEF’s global programmes and policies, and works with partners to make policies for children evidence-based. Publications produced by the Office are contributions to a global debate on children and child rights issues and include a wide range of opinions. The views expressed are those of the authors and/or editors and are published in order to stimulate further dialogue on impact evaluation methods. They do not necessarily reflect the policies or views of UNICEF. OFFICE OF RESEARCH METHODOLOGICAL BRIEFS UNICEF Office of Research Methodological Briefs are intended to share contemporary research practice, methods, designs, and recommendations from renowned researchers and evaluators. The primary audience is UNICEF staff who conduct, commission or interpret research and evaluation findings to make decisions about programming, policy and advocacy. This brief has undergone an internal peer review. The text has not been edited to official publication standards and UNICEF accepts no responsibility for errors. Extracts from this publication may be freely reproduced with due acknowledgement. Requests to utilize larger portions or the full publication should be addressed to the Communication Unit at [email protected] To consult and download the Methodological Briefs, please visit http://www.unicef-irc.org/KM/IE/ For readers wishing to cite this document we suggest the following form: Rogers, P.
    [Show full text]
  • Oxfam Strategic Plan, 2013-2019 Cover Photo: Caroline Gluck the POWER of PEOPLE AGAINST POVERTY 3 OXFAM STRATEGIC PLAN, 2013 – 2019
    Oxfam Strategic Plan, 2013-2019 Cover photo: Caroline Gluck THE POWER OF PEOPLE AGAINST POVERTY 3 OXFAM STRATEGIC PLAN, 2013 – 2019 Contents 1. foreword Foreword 5 Oxfam’s Vision 6 The power of people against poverty 7 Oxfam’s guiding principles 10 2. SIX GOALS TO CHANGE OUR WORLD Introduction 13 Goal 1: The Right to be Heard – People claiming their right to a better life 14 Goal 2: Advancing gender justice 15 Goal 3: Saving lives, now and in the future 16 Goal 4: Sustainable food 17 Goal 5: Fair sharing of natural resources 18 Goal 6: Financing for development and universal essential services 19 3. six goals to change the way we work Introduction 22 Operational Goal 1: Creating a worldwide influencing network 23 Operational Goal 2: Program quality, monitoring, evaluation and learning (MEL) 24 Operational Goal 3: Strengthening accountability 25 Operational Goal 4: Investing in people 26 Operational Goal 5: Cost effectiveness 27 Operational Goal 6: Income strategy 28 4. benchmarks and budget Measuring the impact of our work for people in poverty 30 Funding the ambition for poverty reduction in this Strategic Plan 30 Glossary of abbreviations 31 1. foreword Photo: Rachel Corner THE POWER OF PEOPLE AGAINST POVERTY 5 OXFAM STRATEGIC PLAN, 2013 – 2019 Oxfam has a vision: nine billion people will live equitably There are plenty of reasons for hope. Rejection of the ‘old’ and free from the injustice of poverty, on a planet that has economic growth paradigm opens crucial opportunities the natural resources to sustain them. for new thinking and approaches that favour a just and sustainable future.
    [Show full text]
  • Learning from the Women's Empowerment Index
    ‘ Learning from the Women’s Empowerment Index: Case Studies on the Income and Time Domains Columbia University Workshop Team 2015-2016 Lisa Bacon Nicole Figot Krista Jorstad Aly Marczynski Julia Nethero Albane Sautory Table of Contents Acknowledgements ........................................................................................................ 4 Acronyms ....................................................................................................................... 5 List of Figures ................................................................................................................ 6 List of Boxes ................................................................................................................... 6 Executive Summary ...................................................................................................... 7 CHAPTER 1 Project Framework and Background ...................................................... 9 Introduction ................................................................................................................. 10 The Hunger Project: An Organizational Overview ................................................. 11 The Hunger Project’s Programs ................................................................................ 12 The Hunger Project’s Monitoring & Evaluation System .......................................... 12 The Women’s Empowerment Index .......................................................................... 13 Research Framework and Background
    [Show full text]
  • Social Change: Mechanisms and Metaphors
    Social Change: Mechanisms and Metaphors Kieran Healy∗ August 1998 ∗Department of Sociology, 2–N–2 Green Hall, Princeton University, Princeton N.J. 08544. Email at [email protected]. Please do not cite or distribute without permis- sion. Thanks to Miguel Centeno, Sara Curran, Paul DiMaggio, Patricia Fernandez–Kelly, Frank Karel, Lynn Robinson, Brian Steensland, Steve Tepper, Bruce Western and Bob Wuthnow for suggestions, discussion and comments. Research for this paper was funded by The Robert Wood Johnson Foundation. 1 2 Kieran Healy Contents Introduction 3 The Varying Fortunes of Social Change 4 The Legacy of the Nineteenth Century . 4 Social Change as Modernization . 5 The Analysis of Change 8 Prediction . 8 Metaphors of change . 11 Organic development . 11 Ecological competition and selection . 11 Diffusion and contagion . 12 Path dependence and hysteresis . 13 Complexity and self–organization . 14 Mechanisms and causes . 15 Sources of Social Change 16 Demography . 16 Technology . 17 Economic change . 20 Planned change . 21 Organizations . 23 Institutions . 25 Culture . 28 Conclusion 31 Appendix: An Annotated Bibliography 34 Demography and social change . 34 Technology and change . 38 Economic and political change . 41 Planned change and community change . 49 Organizational change . 54 Gender and social change . 62 Institutions and change . 66 Culture and social change . 69 Complexity theory . 76 Social theory and social change . 80 Social Change: mechanisms and metaphors 3 Introduction Good social science should be able to explain how and why things change. Questions about change can be posed directly, as when we wonder whether religious observance is declining in America, or try to explain why large corporations first appeared at the end of the nineteenth century.
    [Show full text]
  • Theory of Change Primer
    GEF/STAP/C.57/Inf.04 December 11, 2019 57th GEF Council Meeting December 17 - 19, 2019 Washington, D.C. THEORY OF CHANGE PRIMER Theory of Change Primer A STAP document December 2019 Theory of Change primer A STAP document Foreword This document provides a synthesis of guidance specifically aimed at carrying out Theory of Change processes in a GEF context, as part of a growing suite of STAP documents designed to support the design of interventions within GEF’s goal to apply leading practices to deliver transformational change. Theory of Change interacts with many other elements of project and program design, which this document is not intended to replicate, so these interactions are only discussed briefly in terms of their implications for ToC, and are often cross-referenced to other STAP, GEF or external sources for more details. In drawing up this primer, STAP spoke with diverse practitioners in the GEF family, whose inputs were greatly appreciated; a wide range of sources, on-line and in the peer-reviewed literature, were also reviewed, more details about which can be found in the companion Supplement to this Primer (a short literature review and annotated bibliography)1. As one correspondent commented, “ToC is challenging as it requires you to deconstruct your assumptions and the mini-outcomes towards goal achievement in a much more detailed way. It takes a lot of practice to be able to use the tool/process in a comprehensive and intelligent way.” This is very true, but as The Annie E. Casey Foundation says bluntly, “Communities have too much at stake to engage in work without a clearly defined purpose.” This primer aims to structure the complexity of ToC so that significant improvement in outcomes can be achieved for as little pain as possible.
    [Show full text]
  • Examples of Theories of Change Collated and Annotated by Isabel Vogel (Consultant) and Zoe Stephenson, DFID EVD, July 2012
    Appendix 3: Examples of Theories of Change Collated and annotated by Isabel Vogel (consultant) and Zoe Stephenson, DFID EVD, July 2012 Contents Examples of Theories of Change ...................................................................................................... 1 1. Introduction ................................................................................................................................... 2 1.1 Types of theory of change ....................................................................................................... 2 1.2 DFID Evaluation Department Draft Checklist for theories of change (July 2012) ..................... 3 2. DFID Country and Sector Theories of Change .............................................................................. 4 2.1 DFID Programme: Theory of Change for Improving the Quality of General Education in Ethiopia ......................................................................................................................................... 4 2.2 DFID Programme: Theory of Change for India Poorest States Inclusive Growth Programme (PSIG) ......................................................................................................................................... 11 2.3 DFID Programme Theory of Change: Roads in East DRC .................................................... 17 3. Generic causal pathways for policy areas ................................................................................... 29 3.1 DFID How To Note: Theory of Change for interventions
    [Show full text]
  • Supporting Young People to Make Change Happen a Review of Theories of Change
    A joint working paper by ActKnowledge and Oxfam SUPPORTING YOUNG PEOPLE TO MAKE CHANGE HAPPEN A REVIEW OF THEORIES OF CHANGE SYNTHESIS REPORT FEBRUARY 2013 Written by Eoin Collins and Heléne Clark Supported By TABLE OF CONTENTS 1.Introduction ...................................................................................................................... 1 2. Understanding of and Approach to the Brief ................................................................... 1 2.1 Theory of Change: Defining the Concept ..................................................................... 1 2.2 Framing the Research ................................................................................................. 2 3. Theories of Change Underpinning Support for Youth to Create Change ........................... 3 3.1 Young People Participate in Political, Policy and Other Decision-Making Structures .... 6 3.2 Young People are Civically Engaged and Active in their Communities ......................... 8 3.3 Young People Engage Specifically on Positive Change through Youth-led Initiatives or Effective Inter-generational Partnerships........................................................................ 10 3.4 Young People have Leadership Skills to Work at Local and Global Levels .................. 16 4. Conclusions .................................................................................................................... 19 Bibliography ......................................................................................................................
    [Show full text]
  • The Use of Theories of Change in Media and Governance Programmes
    The Use of Theories of Change in Media and Governance Programmes Final Report to SDC 10 November 2019 By Christoph Spurk and Dr. Carmen Koch IAM Institute of Applied Media Studies Winterthur The Use of Theories of Change in Media and Governance Programmes Final Report to SDC © Swiss Agency for Development and Cooperation SDC 2019 All rights reserved. Reproduction, copy, transmission, or translation of any part of this publication may be made only under the following conditions: • with the prior permission of the publisher; or • with a licence from the Copyright Licensing Agency Ltd., 90 Tottenham Court Road, London W1P 9HE, UK, or • from another national licensing agency; or • under the terms set out below. This publication is copyright, but may be reproduced by any method without fee for teaching or nonprofit purposes, but not for resale. Formal permission is required for all such uses, but normally will be granted immediately. For copying in any other circumstances, or for re-use in other publications, or for translation or adaptation, prior written permission must be obtained from the publisher and a fee may be payable. Contacts SDC Network on Democratisation, Decentralisation and Local Governance Swiss Agency for Development and Cooperation (SDC) Democratisation, Decentralisation and Local Governance Tel: +41 (0)58 462 12 01 E-mail: [email protected] www.shareweb.ch/site/DDLGN Institute of Applied Media Studies (IAM) Christoph Spurk and Carmen Koch Institute of Applied Media Studies (IAM) Zurich University of Applied Sciences (ZHAW) Theaterstrasse 15c CH 8401 Winterthur Switzerland Email: [email protected]; [email protected] www.zhaw.ch/linguistik/iam/institute-of-applied-media-studies 1 Content I Introduction ................................................................................................................
    [Show full text]
  • Theory of Change Paper, UNICEF Strategic Plan, 2018–2021 Realizing the Rights of Every Child, Especially the Most Disadvantaged
    UNICEF/2017/EB/11 Distr.: General 18 July 2017 Original: English For information United Nations Children’s Fund Executive Board Second regular session 2017 12-15 September 2017 Item 4 of the provisional agenda* Theory of Change Paper, UNICEF Strategic Plan, 2018–2021 Realizing the rights of every child, especially the most disadvantaged Contents I. Theory of Change overview ........................................................................................................... 3 A. Introduction: theories of change and the UNICEF Strategic Plan, 2018-2021 ....................... 3 B. Overview of the Theory of Change ........................................................................................ 3 C. Goal Areas (outcomes) ........................................................................................................... 6 D. Links to the Sustainable Development Goals and Convention on the Rights of the Child ..... 7 E. Outputs ................................................................................................................................... 8 F. Risks and assumptions .......................................................................................................... 11 G. Implementation strategies (HOWs) ...................................................................................... 15 H. Organizational performance enablers ................................................................................... 18 II. Goal Area 1 theory of change ...................................................................................................
    [Show full text]