Breuer et al. Implementation Science (2016) 11:63 DOI 10.1186/s13012-016-0422-6 SYSTEMATIC REVIEW Open Access Using theory of change to design and evaluate public health interventions: a systematic review Erica Breuer1*, Lucy Lee2, Mary De Silva2 and Crick Lund1 Abstract Background: Despite the increasing popularity of the theory of change (ToC) approach, little is known about the extent to which ToC has been used in the design and evaluation of public health interventions. This review aims to determine how ToCs have been developed and used in the development and evaluation of public health interventions globally. Methods: We searched for papers reporting the use of “theory of change” in the development or evaluation of public health interventions in databases of peer-reviewed journal articles such as Scopus, Pubmed, PsychInfo, grey literature databases, Google and websites of development funders. We included papers of any date, language or study design. Both abstracts and full text papers were double screened. Data were extracted and narratively and quantitatively summarised. Results: A total of 62 papers were included in the review. Forty-nine (79 %) described the development of ToC, 18 (29 %) described the use of ToC in the development of the intervention and 49 (79 %) described the use of ToC in the evaluation of the intervention. Although a large number of papers were included in the review, their descriptions of the ToC development and use in intervention design and evaluation lacked detail. Conclusions: The use of the ToC approach is widespread in the public health literature. Clear reporting of the ToC process and outputs is important to strengthen the body of literature on practical application of ToC in order to develop our understanding of the benefits and advantages of using ToC. We also propose a checklist for reporting on the use of ToC to ensure transparent reporting and recommend that our checklist is used and refined by authors reporting the ToC approach. Keywords: Theory of change, Intervention development, Evaluation, Programme theory, Public health, Systematic review Background controlled trials (RCTs) [1]. Some researchers propose Most public health interventions are inherently complex, that understanding the public health intervention’s with multiple interacting components, delivered at mul- underlying theory of change (ToC) and its related uncer- tiple levels. This complexity makes them difficult to tainties may improve the evaluation of complex health evaluate using traditional experimental designs. Public interventions [1–3]. health interventions often rely on ongoing quality im- Theory-driven evaluation is a collection of evaluation provement based on the implementation experience. methods which emphasise the importance of under- Therefore, they may not reach the level of stability re- standing how and why a programme works in order to quired to conduct evaluations such as randomised evaluate it [4, 5]. By programme, we mean a set of orga- nised activities or interventions supported by resources * Correspondence: [email protected] designed to achieve a specific result [6]. The theories are 1Alan J. Flisher Centre for Public Mental Health, Department of Psychiatry and Mental Health, University of Cape Town, 46 Sawkins Road, Rondebosch, first made explicit and then used to see how the 7700 Cape Town, South Africa programme theory results in the intended outcomes [4]. Full list of author information is available at the end of the article © 2016 Breuer et al. Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated. Breuer et al. Implementation Science (2016) 11:63 Page 2 of 17 There are several overlapping types of theory-driven comes from a perspective of scientific realism and fo- evaluation approaches including logic models, logical cuses predominantly on the interaction between the frameworks, outcomes hierarchies, realist evaluation, context, mechanisms and outcomes of the programme. and ToC [4, 5]. Usually used post hoc, evaluators seek to uncover the ToC was developed by Weiss and others [7] within underlying programme theories. These theories are the tradition of theory-driven evaluation. Although often more abstract than the theories developed definitions of ToC vary, we define it as an approach through ToC or logic models [18]. The development of which describes how a programme brings about spe- ToC has been influenced by Freirean thinking on how cific long-term outcomes through a logical sequence to create social change by empowering individuals [19]. of intermediate outcomes [8]. The ToC is often devel- Despite some fundamental differences in their theoret- oped using a backward mapping approach which ical underpinnings, many of these approaches are used starts with the long-term outcome and then maps the interchangeably or together [18, 20]. required process of changeandtheshort-and ToC has been used widely in the development medium-term outcomes required to achieve this [9]. sector for programme development and evaluation by fun- During this process, the assumptions about what ders such as the UK’s Department for International needs to be in place for the ToC to occur are made Development, Comic Relief, Grand Challenges Canada explicit as well as the contextual factors which influ- and the Gates Foundation [3, 19]. However, there has ence the ToC. Additional elements of a ToC can in- been no global systematic review to our knowledge clude beneficiaries, research evidence supporting the on the use of ToC for the design and evaluation of ToC, actors in the context, sphere of influence, stra- public health interventions. Coryn et al. [4] conducted tegic choices and interventions, timelines and indica- a review of theory-driven evaluation more broadly. tors [8]. These elements are usually presented in a They found 45 examples of theory-driven evaluation diagram and/or narrative summary [8]. in the peer-reviewed literature between 1990 and The ToC is usually developed in consultation with 2009. These evaluations included education, crime stakeholders through workshops or interviews although and safety and transportation interventions. Roughly the participation of stakeholders can vary substantially in half (21/45) were evaluations of health interventions practice [10]. For example, some ToCs are developed [4]. A rapid analysis of the included papers in prepar- through a series of workshops and meetings with a wide ation for this review indicated that only three of these range of stakeholders including service users [11, 12] used ToC. whereas others are developed by evaluators and funders The lack of a systematic review means that there is using programme documentation [13, 14]. The resulting no clear idea of how the ToC has been used and re- ToCs can be used as a framework for programme devel- ported in the peer-reviewed and grey literature in rela- opment and evaluation [8]. The ToC approach is tion to public health interventions. Given the increasing method neutral and as such does not prescribe specific popularity of the ToC approach, understanding how it is types of evaluation methods such as qualitative inter- has been used and described previously allows future views or RCTs [15]. users of the approach to learn from the work of others ToC is distinct from sociological or psychological the- and build upon it. It also helps to move towards a more ories which describe why change occurs although these consistent way of using the ToC approach. may be used to inform the ToC [3]. For example, Bauer In this review, we sought to review both peer-reviewed used an ecological model of community organising to and grey literature to determine how ToCs have been inform a ToC for a capacity and advocacy initiative for developed and used in the development and evaluation residents to impact on public health policy and training of public health interventions globally. Specifically, we of public health professionals [16]. sought to answer the following questions: ToC differs from other theory-driven approaches to evaluation despite similar origins. For example, al- (1)How are ToCs for public health interventions though logic models outline the inputs, processes, developed and refined? outputs and outcomes of a programme in a similar (2)How is the ToC approach used in the manner to ToC, they can be rigid and do not make ex- (a) development of an intervention; plicit the causal pathways through which change hap- (b) implementation of the intervention; pens in the way that ToC does [3]. Similarly, although (c) development of indicators for measurement; logframes were initially developed to summarise discus- (d) evaluation of the intervention, including sions with stakeholders, funder-driven formats have statistical approaches; and largely reduced logframes to a results-based manage- (e) conceptualisation/evaluation of the influence of ment tool [17]. Realist evaluation, on the other hand, context. Breuer et al. Implementation Science (2016) 11:63 Page 3 of 17 Methods As the focus of this review was on public health inter- The authors developed a protocol for this review which ventions, we excluded papers in which the long-term was agreed prior to the commencement of the study. outcome of the ToC was a change within an individual This is available in Web Additional file 1. rather than change in the population. For example, a ToC describing how cognitive behavioural therapy may Inclusion and exclusion criteria impact on an individual’s cognitive processes and behav- The inclusion and exclusion criteria are listed in Table 1. iour would be a change within the individual.