Contents Measuring implementation strength Literature review draft report 2012 Acknowledgements Dr Joanna Schellenberg, Principal Investigator, IDEAS project, London School of Hygiene & Tropical Medicine. Funded by the Bill & Melinda Gates foundation Review supervised by Dr Bilal Iqbal Avan Review prepared by Dr Natalia Bobrova Copy edited by Kate Sabot Coordination of publication by Agnes Becker Copyright: London School of Hygiene & Tropical Medicine ISBN - 978 0 902657 90 9 w: ideas.lshtm.ac.uk Contents Contents Acknowledgements ................................................................................................................................. 2 Acronyms and abbreviations .................................................................................................................. 4 List of tables ............................................................................................................................................ 5 List of figures ........................................................................................................................................... 5 Introduction ............................................................................................................................................ 6 Methods .................................................................................................................................................. 8 Results and discussion .......................................................................................................................... 10 Summary and conclusions .................................................................................................................... 16 References ............................................................................................................................................ 17 Appendix 1. Data extraction and glossary ............................................................................................ 23 Appendix 2. Review tools ...................................................................................................................... 78 Appendix . “tudies desiptios ......................................................................................................... 81 Appedi . “tudies tools saples. ..................................................................................................... 89 ideas.lshtm.ac.uk Measuring implementation strength 3 Acronyms and abbreviations Acronyms and abbreviations ACIC Assessment of Chronic Illness Care ACT Assertive Community Treatment AIDS Acquired Immune Deficiency Syndrome BSC Balanced Score Card CCC Community Care Centres CCM Chronic Care Model CQI Continuous Quality Improvement DACT Dartmouth Assertive Community Treatment HIV Human Immunodeficiency Virus IDEAS Informed Decisions for Actions IPS Individual Placement Support IRT Implementation Rating Instrument ISD Integrated Delivery System MHI Medical Home Index NACO National AIDS Control Organization NGOs Non-Governmental Organisations PDC Project Developmental Continuity PRISMA Providing Innovative Service Models and Assessment QI Quality Improvement QRF Quarterly Report Form RED Reaching Every District approach SAT System Assessment Tool SE Supported Employment TB Tuberculosis UNAIDS Joint United Nations Programme on HIV/AIDS UNICEF United Nations Children's Fund WHO World Health Organization ideas.lshtm.ac.uk Measuring implementation strength 4 List of tables and figures List of tables Table 1: Included study characteristics ................................................................................................. 11 Table 2. Data extraction for measurements of implementation. ......................................................... 23 Table 3. Glossary of implementation strength. .................................................................................... 75 Table 4. Search strategy ........................................................................................................................ 78 Table 5. Main websites used in grey literature search. ........................................................................ 79 Table 6. List of experts contacted. ........................................................................................................ 79 Table 7. Data extraction form for the tools measuring implementation strength. .............................. 80 Table 8. An example of judgemental rating scales for the subcomponent: Development of a diagnostic and evaluative system and individualised programme for children. .................................. 89 Table 9. Sample from the field-based implementation rating scale..................................................... 90 Table 10. Essential components of the physician-based nutrition programme. .................................. 91 Table 11. Indices of implementation of Medicaid expansions, their components and items within components, and their percentage contribution to the overall index of strength of implementation. .............................................................................................................................................................. 92 Table 12. Programme criteria for fidelity to assertive community care. .............................................. 94 Table 13.Sample of structure and content of the ACT Scale. ............................................................... 95 Table 14. Formula for homeless programme implementation scales. ................................................. 96 Table 15.Domains, their attributes and definitions for evaluation of implementation fidelity of the system-of-care model. .......................................................................................................................... 97 Table 16. Program elements included in the early head start implementation rating scales. ............. 99 Table 17. Medical home index domains and themes. ........................................................................ 100 Table 18. Example of the scale for the Care Coordination domain. ................................................... 100 Table 19. Scale to evaluate large-scale organisational change efforts. .............................................. 101 Table 20.List of indicators used to rate the implementation of the ISD system. ............................... 102 Table 21. Statements or action points to measure the implementation rate of the TIPPA project . 103 Table 22.Implementation intensity components. .............................................................................. 104 Table 23. Nurse manager survey of ten steps implementation. ........................................................ 105 Table 24. Seven dimension of health service systems from System Assessment Tool. ..................... 109 Table 25. Balanced score card indicators and domains. ..................................................................... 110 Table 26. The strategy components. .................................................................................................. 111 Table 27. RED Implementation components, indicators, indicator weighting scheme and level needed to achieve high implementation score indicator Individual indicator Individual ............................... 112 Table 28. An example of oe iditos desiptio fo suppotie supeisio opoets fo WHO guide: Implementing the Reaching Every District approach: A guide for district health management teams ............................................................................................................................ 114 Table 29.Sample of the summary scores for the process evaluation of community care centres..... 115 List of figures Figure 1. Monitoring and evaluation of health systems strengthening. (WHO, 2010)........................... 7 Figure 2. Search Results .......................................................................................................................... 9 ideas.lshtm.ac.uk Measuring implementation strength 5 Introduction Introduction Measuring implementation strength (sometimes referred to as implementation intensity) is an important programme evaluation process which helps to understand why some programmes are successful and some fail, attribute outcomes to a programme, and anticipate outcomes of future programmes (Bryce et al., 2011, Dane and Schneider, 1998, Durlak, 1998b, Dusenbury et al., 2003, Hall et al., 1994, Proctor et al., 2011, Victora et al., 2011). Implementation data can also help in evaluating and improving progress toward specific outcomes and intervention strategies (Damschroder and Hagedorn, 2011, Durlak, 1998b, McGraw et al., 2000, Proctor et al., 2011, Mowbray et al., 2003). In developing a new approach, the Distit Ealuatio Platfo, to ealuate large-scale effectiveness for proven interventions at a national level, Victora et al., emphasise the ipotae of easuig ipleetatio stegth as isuffiiet ipleetatio is a oo reason for absence of ipat (Victora et al., 2011). Nevertheless, despite the importance of evaluating implementation strength of complex, multidimensional interventions, scientific evidence devoted to this issue is limited, especially in low income countries. The common evaluation framework (Figure 1. Monitoring and evaluation of health systems strengthening. (WHO, 2010)), initially developed to measure health system performance and recently used in measuring implementation strength of community case