1 The London School of Economics and Political Science HIERARCHIES OF EVIDENCE IN EVIDENCE-BASED MEDICINE CHRISTOPHER J BLUNT A THESIS SUBMITTED TO THE DEPARTMENT OF PHILOSOPHY, LOGIC & SCIENTIFIC METHOD OF THE LONDON SCHOOL OF ECONOMICS FOR THE DEGREE OF DOCTOR OF PHILOSOPHY, LONDON, SEPTEMBER 2015 2 DECLARATION I certify that the thesis I have presented for examination for the PhD degree of the London School of Economics and Political Science is solely my own work other than where I have clearly indicated that it is the work of others (in which case the extent of any work carried out jointly by me and any other person is clearly identified in it). The copyright of this thesis rests with the author. Quotation from it is permitted, provided that full acknowledgement is made. This thesis may not be reproduced without my prior written consent. I warrant that this authorisation does not, to the best of my belief, infringe the rights of any third party. I declare that my thesis consists of 79,599 words. 3 ABSTRACT Hierarchies of evidence are an important and influential tool for appraising evidence in medicine. In recent years, hierarchies have been formally adopted by organizations including the Cochrane Collaboration [1], NICE [2,3], the WHO [4], the US Preventive Services Task Force [5], and the Australian NHMRC [6,7]. The development of such hierarchies has been regarded as a central part of Evidence-Based Medicine (e.g. [8-10]), a movement within healthcare which prioritises the use of epidemiological evidence such as that provided by Randomised Controlled Trials (RCTs). Philosophical work on the methodology of medicine has so far mostly focused on claims about the superiority of RCTs, and hence has largely neglected the questions of what hierarchies are, what assumptions they require, and how they affect clinical practice. This thesis shows that there is great variation in the hierarchies defended and in the interpretations they are, and can be, given. The interpretative assumptions made in using hierarchies are crucial to the content and defensibility of the underlying philosophical commitments concerning evidence and medical practice. Once this variation is been identified, it becomes clear that the little philosophical work that has been done so far affects only some hierarchies, under some interpretations. Modest interpretations offered by La Caze [11], conditional hierarchies like GRADE [12-14], and heuristic approaches such as that defended by Howick et al. [15,16] all survive previous philosophical criticism. This thesis extends previous criticisms by arguing that modest interpretations are so weak as to be unhelpful for clinical practice; that GRADE and similar conditional models omit clinically- relevant information, such as information about variation in treatments’ effects and the causes of different responses to therapy; and that heuristic approaches lack the necessary empirical support. The conclusion is that hierarchies in general embed untenable philosophical assumptions: principally that information about average treatment effects backed by high-quality evidence can justify strong recommendations, and that the impact of evidence from individual studies can and should be appraised in isolation. Hierarchies are a poor basis for the application of evidence in clinical practice. The Evidence-Based Medicine movement should move beyond them and explore alternative tools for appraising the overall evidence for therapeutic claims. 4 For Lynn Blunt (1955-2014) 5 There was but one question he left unasked, and it vibrated between his lines: if gross miscalculations of a person’s value could occur on a baseball field, before a live audience of thirty thousand, and a television audience of millions more, what did that say about the measurement of performance in other lines of work? If professional baseball players could be over- or under-valued, who couldn’t? Bad as they may have been, the statistics used to evaluate baseball players were probably far more accurate than anything used to measure the value of people who didn’t play baseball for a living. —Michael Lewis, Moneyball, p.72 [17] Real evidence is usually vague and unsatisfactory. It has to be examined—sifted. But here the whole thing is cut and dried. No, my friend, this evidence has been very cleverly manufactured—so cleverly that it has defeated its own ends. —Agatha Christie, The Mysterious Affair at Styles [18] 6 ACKNOWLEDGEMENTS I would like to thank John Worrall for supervising this thesis, for his continual support and feedback, and for giving the lectures which introduced me to Evidence-Based Medicine in the distant past of my undergraduate days. I also want to thank Roman Frigg for acting as secondary supervisor, and for encouraging me to study for the PhD and to do it at LSE. Huge thanks are also due to Luc Bovens for his support and for the opportunities to teach in the department, without which this thesis and my time at LSE would have been much the worse. Thanks are also due to Alex Voorhoeve, Foad Dizadji-Bahmani and Armin Schulz. I also want to thank Stephen John, Hasok Chang and Tim Lewens for their supervision whilst at Cambridge, and in particular Hasok for giving the first philosophy lecture I ever attended back in 2006. He is almost entirely accountable for my decision to study philosophy of science. My thanks also go to the Arts & Humanities Research Council and the Philosophy department for funding this work, and to Bill Bottriell for his extremely generous support as an undergraduate. Bill is one of the kindest and friendliest people I have been lucky enough to meet. Invaluable advice came from my friend and colleague Rosa Runhardt, as well as Catherine Greene and Nicolas Wuethrich. I was fortunate enough to teach a great many undergraduate and masters students over the last four years, and cannot stress enough how valuable that experience was. Unending gratitude goes to my father Robin Blunt and my mother, Lynn, who was probably the only person in the world who cared about this thesis more than me. I only wish she could read it. Finally, and most of all, thank you to my partner Kate Butler, without whose continual distractions and endless patience this thesis would either have been finished much sooner or never at all. 7 CONTENTS Chapter 1: Hierarchies of Evidence in Evidence-Based Medicine p.10 Chapter 2: The Myths of the Hierarchy of Evidence p.37 Chapter 3: On the Interpretation of Hierarchies p.85 Chapter 4: The Challenges to Hierarchies p.118 Chapter 5: Relative Rankings p.156 Chapter 6: Evidence and Evidence Bases: Challenges to GRADE and Levels of Evidence p.174 Chapter 7: Conclusions: Heuristics and EBM after Hierarchies p.233 Bibliography p.251 8 TABLE OF FIGURES Fig.1 Hierarchy of evidence from Users’ Guides to the Medical Literature (2002) by Guyatt p.19 et al. Fig.2 Hierarchy of evidence from AASM (1999) by Chesson et al. p.21 Fig.3 ‘4s’ hierarchy of pre-appraised sources (2001) by Haynes. p.21 Fig.4 SIGN hierarchy of evidence bases (2004) from NICE Guideline Development Manual. p.22 Fig.5 Hierarchy of methodologies (1981) by Sackett. p.42 Fig.6 Hierarchy of evidence bases (2006) used by NICE. p.43 Fig.7 Hierarchy of evidence bases (1999) by the Australian National Health & Medical p.44 Research Council. Fig.8 Diagram summarizing the stages in GRADE appraisal (2011) by Guyatt et al. p.45 Fig.9 GRADE hierarchy (2011) by Balshem et al. p.46 Fig.10 Simplified GRADE hierarchy from ATS (2006) by Schunemann et al. p.49 Fig.11 Table presenting criteria for levels of evidence quality in GRADE from Users’ Guides to p.50 the Medical Literature (2008) by Guyatt et al. Fig.12 Hierarchy of evidence strength (1998) by Edwards, Russell & Stott. p.59 Fig.13 Levels of evidence hierarchy from CHEST (1992) by Cook et al. p.61 Fig.14 Levels of evidence hierarchy from CEBM (2009) by Howick et al. p.62 Fig.15 Grades of recommendation table from CEBM (2009) by Howick et al. p.63 Fig.16 Hierarchy and grades of recommendation from GREG (2003) by Mason & Eccles. p.64 Fig.17 Grades of recommendation table from CHEST (1998) by Guyatt et al. p.65 Fig.18 Hierarchy of evidence (2003) by Evans. p.67 Fig.19 Hierarchy of evidence from How to read a paper (1997) by Greenhalgh. p.69 Fig.20 Hierarchy of evidence bases with catch-all category (2000) by van Tulder et al. p.73 9 Fig.21 Table showing GRADE reformulated as a categorical hierarchy of evidence bases. p.77 Fig.22 Hierarchy of evidence bases with conditions from CHEST (1989) by Sackett. p.78 Fig.23 Hierarchy of evidence with conditions from AAN (2012) by Elamin & Montori. p.79 Fig.24 Example of a Sources of Invalidity table (1963) by Campbell & Stanley. p.88 Fig.25 First hierarchy of evidence (1979) by the Canadian Task Force on the Periodic Health p.90 Examination. Fig.26 Hierarchy of evidence from CHEST (1986) by Sackett. p.92 Fig.27 Checklist for valid evidence from CHEST (1992) by Cook et al. p.94 Fig.28 First EBM hierarchy of evidence from Users’ Guides IX (1995) by Guyatt et al. p.96 Fig.29 Tickbox hierarchy of evidence (1997) by Muir Gray. p.96 Fig.30 Hierarchy of evidence for CEBM (2002) by Ball & Phillips, an update of Phillips et al.’s p.101 1998 hierarchy. Fig.31 Hierarchy of evidence depicted in two formats (2011) by Eaves. p.106 Fig.32 Pyramid of evidence (2008) by Ho, Peterson & Masoudi. p.107 Fig.33 Table reporting study methodology, comparison, patient population, and primary p.161 and secondary outcome measures for four studies of Tocilizumab for Rheumatoid arthritis.