Evaluation of NIHR investment in Cochrane infrastructure and systematic reviews Committee members: Professor Jos Kleijnen (Chair), Dr Phil Alderson, Dr Jane Aubin, Professor John Cairns, Ms Sally Crowe, and Professor Paul Garner Report writer: Kate Misso 10 February 2017 1 TABLE OF CONTENTS LIST OF TABLES ............................................................................................................................ 5 LIST OF FIGURES ........................................................................................................................... 6 LIST OF ABBREVIATIONS ............................................................................................................... 7 EXECUTIVE SUMMARY ............................................................................................................... 10 1. The global landscape of systematic reviews ............................................................................. 10 2. The performance of NIHR funded Cochrane Review Groups (CRGs) ....................................... 11 3. Cochrane’s impact on key clinical and policy issues in the NHS ............................................... 13 4. The economic impact of systematic reviews ............................................................................ 14 5. Current and planned developments in Cochrane and stakeholders’ views ............................. 14 Conclusions ....................................................................................................................................... 16 Recommendations and considerations for NIHR Cochrane funding ................................................ 16 Recommendations to Cochrane ................................................................................................... 16 Recommendations to NIHR ........................................................................................................... 17 INTRODUCTION ......................................................................................................................... 19 CHAPTER 1 – SYSTEMATIC REVIEWS IN HEALTH: THE GLOBAL LANDSCAPE ................................... 23 Overview ................................................................................................................................... 23 Organisations funding or preparing systematic reviews in the UK................................................... 26 Organisations funding or preparing systematic reviews globally ...................................................... 30 Cochrane Review Groups .................................................................................................................. 31 Global landscape of systematic review production: Main points .................................................. 32 CHAPTER 2 - PERFORMANCE OF NIHR FUNDED CRGS AND COCHRANE UK ................................... 33 Overview ................................................................................................................................... 33 Outputs ............................................................................................................................................. 33 Publication of reviews by NIHR-funded Cochrane Review Groups .............................................. 33 Maintenance and updating workload ........................................................................................... 34 Most accessed Cochrane reviews ................................................................................................. 34 'Empty' reviews ............................................................................................................................. 35 Number of guidelines based on SRs.............................................................................................. 37 Quality of reviews produced by NIHR-funded Cochrane Review Groups ........................................ 39 Readability study ........................................................................................................................... 39 MECIR evaluation .......................................................................................................................... 39 Concordance between Cochrane and non-Cochrane meta-analyses ........................................... 43 Comparative Risk of Bias (RoB) analysis of Cochrane and non-Cochrane reviews ....................... 43 Impact factors ............................................................................................................................... 45 Activities ............................................................................................................................................ 48 2 Activity of NIHR-funded CRGs1 ...................................................................................................... 48 Editorial support and managerial processes ................................................................................. 50 Workshops and training ................................................................................................................ 52 Specialised registers maintained by NIHR-funded CRGs .............................................................. 52 Performance of NIHR-funded CRGS and Cochrane UK: Main points .............................................. 54 CHAPTER 3 - MEETING KEY CLINICAL AND POLICY QUESTIONS IN THE NHS .................................. 55 Overview ................................................................................................................................... 55 Current NHS plans ............................................................................................................................. 55 Better reviews: relevant, timely and high quality ............................................................................. 56 What impact has Cochrane had in meeting clinical and policy issues in the NHS? .......................... 56 Policies .......................................................................................................................................... 56 Reviews that change practice and change or save lives ............................................................... 57 Culture change: concept of scientific rigour and independence .................................................. 62 Communicating with relevant audiences ..................................................................................... 63 Methodology ................................................................................................................................. 65 Cochrane as a benchmark ............................................................................................................. 66 Meeting key clinical and policy issues in the NHS: Main points .................................................... 67 CHAPTER 4 – CASE STUDIES OF THE ECONOMIC IMPACT OF SELECTED SYSTEMATIC REVIEWS ...... 68 Overview ................................................................................................................................... 68 York CRD/CHE study .......................................................................................................................... 68 Committee reflections ...................................................................................................................... 71 Case studies of the economic impact of selected systematic reviews: Main points ........................ 74 CHAPTER 5 - COCHRANE STRATEGY AND THE NHS ...................................................................... 75 Overview ................................................................................................................................... 75 Report from Stakeholder interviews ................................................................................................ 75 Cochrane’s Strategic Plan ................................................................................................................. 76 Cochrane’s key goals of the 2020 Strategy ....................................................................................... 76 Meeting key clinical and policy issues in the NHS ............................................................................ 79 Publish when ready ....................................................................................................................... 82 Priority list of titles ........................................................................................................................ 82 Centralisation or decentralisation? Cochrane's proposed plans for Group level change: Review of the structure and function of Cochrane Groups ............................................................................... 83 Planned developments in reporting and monitoring processes between Cochrane and NIHR ....... 84 Current and planned developments in Cochrane: Main points ..................................................... 86 CHAPTER 6 - THE VALUE FOR MONEY OF NIHR COCHRANE INVESTMENT AND POINTS TO NOTE FOR THE FUTURE ............................................................................................................................... 87 3 Overview ................................................................................................................................... 87 Cochrane has been good value in the past, but could it have been better value? .........................