© Tanya Holden/DFID © Tanya Holden/DFID UNDER WHAT CONDITIONS DO INSPECTION, MONITORING AND ASSESSMENT IMPROVE SYSTEM EFFICIENCY, SERVICE DELIVERY AND LEARNING OUTCOMES FOR THE POOREST AND MOST MARGINALISED? A REALIST SYNTHESIS OF SCHOOL ACCOUNTABILITY IN LOW- AND MIDDLE-INCOME COUNTRIES SYSTEMATIC REVIEW [DECEMBER 2016] 1 The authors of this report are Dr David Eddy-Spicer, University of Virginia Dr Melanie Ehren, UCL Institute of Education, University College London Dr Mukdarut Bangpan, UCL Institute of Education, University College London Dr Meena Khatwa, UCL Institute of Education, University College London Frank Perrone, University of Virginia Contact David Eddy-Spicer, Associate Professor Curry School of Education, University of Virginia [email protected] Funding This is an independent report commissioned and funded by the Research and Evidence Division in the Department for International Development. This material has been funded by UK aid from the UK Government, however, the views expressed do not necessarily reflect the UK Government’s official policies. Acknowledgments The review team gratefully acknowledges the financial support of the UK Department for International Development, and guidance from DfID staff, including Louise Banham, Ed Barnett, Laura Rivkin and Jessica Vince. At the Institute of Education, we appreciate the substantial support and input from the EPPI-Centre, especially Kelly Dickson and Sandy Oliver. We are grateful for administrative support from Chris To and Claire Phillips, London Centre for Leadership and Learning. Finally, we are indebted to Gill Westhorp of Community Matters, Australia, for her suggestions and comments on drafts of the protocol. Conflicts of interests None Citation Eddy-Spicer D, Ehren M, Bangpan M, Khatwa M, Perrone F (2016) Under what conditions do inspection, monitoring and assessment improve system efficiency, service delivery and learning outcomes for the poorest and most marginalised? A realist synthesis of school accountability in low- and middle-income countries. London: EPPI-Centre, Social Science Research Unit, UCL Institute of Education, University College London. Picture Photo by Tanya Holden/DFID using licence https://creativecommons.org/licenses/by-nc/2.0/ © Copyright Authors of the systematic reviews on the EPPI-Centre website (http://eppi.ioe.ac.uk/) hold the copyright for the text of their reviews. The EPPI-Centre owns the copyright for all material on the website it has developed, including the contents of the databases, manuals, and key wording and data-extraction systems. The centre and authors give permission for users of the site to display and print the contents of the site for their own non- commercial use, provided that the materials are not modified, copyright and other proprietary notices contained in the materials are retained, and the source of the material is cited clearly, following the citation details provided. Otherwise, users are not permitted to duplicate, reproduce, republish, distribute, or store material from this website without written permission. 2 GUIDANCE ON TYPE OF REVIEW AND REPORT STRUCTURE METHODOLOGICAL APPROACH This report presents the findings from a systematic review using realist methodology. The review uses transparent methods to identify, critically appraise and synthesise studies to inform the development of theories about how school accountability policies operate locally to improve school systems and children’s learning outcomes in low- and middle-income countries (LMICs). The theories take the form of configurations of ‘contexts’, ‘mechanisms’ and ‘outcomes’ (CMO), which offer explanatory accounts of the processes through which school accountability policies work in local contexts to achieve school-level outcomes. The narrative supporting each CMO configuration provides a theoretically-informed and empirically grounded explanation for the proposed configurations, presenting a transparent argument based on the synthesised literature. As a type of systematic review, the purpose of a realist review is not only to explore whether a particular intervention does or does not work but to explain why certain outcomes arise through elaboration of the connections amongst contexts, mechanisms, and outcomes grounded in the literature. This review follows the publication standards for realist reviews put forward by the RAMESES (Realist and Meta-narrative Evidence Syntheses: Evolving Standards) project (Wong et al., 2013). REPORT STRUCTURE As this is the technical report of a realist review, using systematic methods, some sections of the report are necessarily detailed. Without compromising the transparency that is expected of a systematic review, we have structured this report to help those who are more concerned with the findings than the methods. Therefore, the report is organised in three sections: 1. Systematic review summary: An eight-page executive summary of the key findings of the review. 2. Main technical report: This contains the background and methods to the reviews (Chapters 1-2, an overview of the studies included in the review (Chapter 3) and the reviews findings (Chapter 4-6). The review findings outlining the CMO syntheses in full are presented in standalone chapters for each school accountability policy area: Assessment, Monitoring and Inspection. The technical report concludes with a summary of the CMO configurations, in addition to the strengths, limitations and implications of the review. 3. Appendices: The appendices contain additional details about the reviews search strategy, coding tools used, and further details about the studies including how the CMO’s configurations were generated. 1 Summary SYSTEMATIC REVIEW SUMMARY This systematic review explores how school accountability policies operate locally to improve school systems and children’s learning outcomes in low- and middle-income countries (LMICs). These policies include: • Assessment: student examinations used to monitor the quality of the education system, some of which (high-stakes examinations) also carry direct consequences for performance for schools, school teachers and individual students. • Monitoring: the system-level processes designed to collect, compare and report school-level information about the composition, organisation and function of schools. • Inspection: formal site visits to schools by education authorities to observe classroom and management activities. Overall, findings suggest that: • Desirable school-level outcomes were associated with coherent support for meeting performance expectations and for translating information about performance into the everyday practices of teaching and learning. • Undesirable school-level outcomes were associated with insufficient consideration of school leaders’ and teachers’ capacities to engage productively with accountability activities, whether in interpreting exam results, in making use of Educational Management and Information System (EMIS) information or in conducting school self-evaluations as part of inspection. OUTLINE OF EVIDENCE Overall, the findings from the three elements of school accountability suggest that: • Assessment may improve the quality of teaching and learning when the following mechanisms are triggered by specific conditions prevalent in the local school context: o Trust in the pedagogical authority of the assessment approaches is triggered by system- and school- level support for teaching tied to assessment approaches. o Teachers’ close attention to results in ways that improve teaching follows from customised guidance around interpreting results. o Incentives prompt teachers’ desire for reward and improvements in teaching quality when incentives are focused on individual (not collective) performance and are perceived as high-value. o Parental oversight of quality of teaching and learning promotes student performance gains when individual student incentives are perceived by parents as being of high value. • Key barriers to assessment activities that aim to improve teaching and learning can include: o School staff fearing the consequences of poor performance. o Lack of individual teacher incentives. o Lack of training and support to use and interpret assessment results effectively by school staff. • Monitoring could lead to improvement in school management and performance when one or more of the following conditions are prevalent in the local school context: o Interpreting information: Sustained effects on school management and student attendance are seen when there is consistent and clear feedback about results that is accompanied by training to interpret the results across district, sub-district and school levels. o Accuracy of information: Timely and accurate reporting of school- and district-level information occurs when those at higher levels of the system place value on understanding system performance rather than rewarding positive results (‘reality testing’). 3 Summary o Local school development planning: This is likely to be effective when school leaders and teachers are given opportunities and the ability to learn from failure. o Acting on information: School management committees use information effectively to improve school conditions when parents develop capacity for interpreting results and pressure schools to improve teaching quality and learning. o Parental involvement: Service delivery and learning outcomes improve when parents participate in monitoring activities. • Inspection generally has a limited impact on systems and school-level