Meta analyses – pros and cons Emily S Sena, PhD Centre for Clinical Brain Sciences, University of Edinburgh @camarades_ CAMARADES: Bringing evidence to translational medicine CAMARADES • Collaborative Approach to Meta-Analysis and Review of Animal Data from Experimental Studies • Look systematically across the modelling of a range of conditions • Data Repository – 30 Diseases – 40 Projects – 25,000 studies – from over 400,000 animals CAMARADES: Bringing evidence to translational medicine Why do we do meta-analysis of animal studies? • Animal models are generally performed to inform human health but when should you be convinced to move to the next step? • Systematic reviews & meta-analyses: – assess the quality and range of evidence – identify gaps in the field – quantify relative utility of outcome measures – inform power/sample size calculations – assess for publication bias – try to explain discrepancies between preclinical and clinical trial results – inform clinical trial design CAMARADES: Bringing evidence to translational medicine Systematic review and meta-analysis • Pros – What have we learnt about…….. • Translation? • Quality? • The 3Rs? • Cons – What are the limitations? • As good as the data that goes in? • Rapidly outdated • The Impact CAMARADES: Bringing evidence to translational medicine Systematic review and meta-analysis • Pros – What have we learnt about…….. • Translation? • Quality? • The 3Rs? • Cons – What are the limitations? • As good as the data that goes in? • Rapidly outdated • The Impact CAMARADES: Bringing evidence to translational medicine 1,026 interventions in experimental stroke In vitro and in vivo - 1026 Tested in vivo - 603 Effective in vivo - 374 Tested in clinical trial - 97 Effective in clinical trial - 1 O’Collins et al, 2006 CAMARADES: Bringing evidence to translational medicine Data from in vivo studies • There are huge amounts of often confusing data Hypothermia: a systematic • Systematic review can help search identified 222 to make sense of it experiments in 3353 animals • If you select extreme bits of the evidence you can “prove” either harm or substantial benefit Better • Investigating the sources behind this variation may be helpful in translation Worse Van der Worp et al Brain 2007 CAMARADES: Bringing evidence to translational medicine You can usually find what you’re looking for … • 12 graduate psychology students • 5 day experiment: rats in T maze with dark arm alternating at random, and the dark arm always reinforced • 2 groups – “Maze Bright” and “Maze dull” Group Day Day Day Day Day 1 2 3 4 5 “Maze 1.33 1.60 2.60 2.83 3.26 bright” “Maze 0.72 1.10 2.23 1.83 1.83 dull” Δ +0.60 +0.50 +0.37 +1.00 +1.43 Rosenthal and Fode (1963), Behav Sci 8, 183-9 CAMARADES: Bringing evidence to translational medicine Non-random sample • N=2671 • Most published in vivo research is at high risk of bias CAMARADES: Bringing evidence to translational medicine Improvement over time….. CAMARADES: Bringing evidence to translational medicine Internal Validity: Lessons from NXY-059 • Infarct Volume – 11 publications, 29 experiments, 408 animals – Improved outcome by 44% (35-53%) è Efficacy Randomisation Blinded conduct Blinded of experiment assessment of outcome Macleod et al, 2008 CAMARADES: Bringing evidence to translational medicine Random sample from PubMed CAMARADES: Bringing evidence to translational medicine Good ‘quality’ journals CAMARADES: Bringing evidence to translational medicine The ‘best’ institutions: RAE 1173 CAMARADES: Bringing evidence to translational medicine Reporting of randomisation across 3 datasets, 2009-10 CAMARADES: Bringing evidence to translational medicine Experimental Design: - choice of strain Van Drongelen et al. 2012 – vascular function during pregnancy CAMARADES: Bringing evidence to translational medicine Experimental Design: - indicative power calculations Mechanical induced outcomes Calculated Behavioural No. of Median Calculated Median Effect sample size Test experiments N power Size (IQRs) (power=0.8) von Frey 48 9 0.3 1.3 (0.9-1.8) 11 (electronic) von Frey 369 11 0.5 1.5 (0.8-2.4) 9 (filaments) Pin prick 12 11 0.6 1.6 (0.3-8.3) 8 Randall-Selitto 156 10 0.8 1.9 (1.0-3.8) 6 paw pressure CAMARADES: Bringing evidence to translational medicine Anti-emetic research - refine duration of experiments SuBgroup Effect size 5mg (24h) 0.00 [-0.15, 0.15] (P = 1.00) § ondansetron protects 50% of the patients 10mg (24h) treated with Cisplatin 0.00 [-0.23, 0.23] (P = 1.00) § ondansetron reduced the number of 10mg (6h) animal developing emesis 0.11 [-0.15, 0.38] (P = 0.40) 10mg (4h) § Efficacy was dependent on dosage and 0.48 [0.26, 0.70] (P < 0.0001) duration of the observation period § Supporting evidence to improve the model 10mg (2h) § Provides evidence supporting refinement 0.76 [0.31, 1.21] (P = 0.0010) of the model (4h instead of 24h) Total -2 0 2 0.33 [0.17, 0.48] (P < 0.0001) Favours Favours control ondansetron Percie du Sert N et al (2011) Cancer Chemother Pharmacol 67(3): 667-686. CAMARADES: Bringing evidence to translational medicine Systematic review and meta-analysis • Pros – What have we learnt about…….. • Translation? • Quality? • The 3Rs? • Cons – What are the limitations? • As good as the data that goes in? • Rapidly outdated • The Impact CAMARADES: Bringing evidence to translational medicine Correlation vs Causation I’m not saying Susan Boyle caused Americans to support gay marriage 70% Oppose legalisation of gay marriage 60% 57% 50% 50% 40% 43% 35% 30% 2009: SuBo wows Support legalisation the world with “I of gay marriage Dreamed a Dream” 20% 2000 2002 2004 2006 2008 2010 2012 2014 But Susan Boyle happened and then support of gay marriage increased CAMARADES: Bringing evidence to translational medicine Apples vs oranges Vs. • Some argue that where you have too much heterogeneity meta-analysis is not appropriate • What’s the purpose? • Animals vs patients • Collinearity • Limits what you can infer CAMARADES: Bringing evidence to translational medicine Only as good as the data that go in Not all outcomes and a priori analyses are reported • Publication bias – Neutral and negative studies – Time lag/remain unpublished – Less likely to be identified • p-hacking – Selective analysis – Selective outcome reporting CAMARADES: Bringing evidence to translational medicine 0.5 0.5 0.4 0.4 0.3 0.3 0.2 Precision 0.2 Precision 0.1 0.1 0.0 0.0 -0.1 -150 -100 -50 0 50 100 150 -10 -5 0 5 10 15 20 25 Effect Size Effect Size/Standard Error CAMARADES: Bringing evidence to translational medicine Overall efficacy was reduced from; 32% (95% CI 30 to 34%) to 26% (95% CI 24 to 28%) CAMARADES: Bringing evidence to translational medicine Publication bias in experimental stroke • Trim and Fill suggested 16% of experiments remain unpublished • Best estimate of magnitude of problem – overstatement of efficacy 31% • Only 2% publications reported no significant treatment effects CAMARADES: Bringing evidence to translational medicine Different patterns of publication bias in different fields outcome observed corrected Disease improvement 40% 30% Less models improvement Toxicology harm 0.32 0.56 More harm model Harm Benefit CAMARADES: Bringing evidence to translational medicine Rate of publication • There are more papers published in a day than most people could read in a month • In 2013, 4700 new publications were added to PubMed every working day Domain Number In vivo and in vitro 610 In vivo 350 Pharmacology 76 Neurosciences 52 • If a panelist for a neuroscience grant awarding body were to try to keep up to date, spending 30 minutes reading each paper, and did nothing else but read papers all year, they would get through 30% of the total CAMARADES: Bringing evidence to translational medicine Literature Search September 2012 Records identified through online databases (n= 65,156) Identification Unique records 31,342 duplicates (n= 33,184) removed Articles screened Screening (n= 33,184) Eligibility CAMARADES: Bringing evidence to translational medicine Literature Search September 2012 Records identified through online databases (n= 65,156) Identification Unique records 31,342 duplicates (n= 33,184) removed Articles screened Screening (n= 33,184) Included based on dual screening Eligibility (n= 4,568) CAMARADES: Bringing evidence to translational medicine Machine Learning & Text Mining • Machine learning – form of artificial intelligence by which computers learn without being explicitly programmed. Construction of algorithms which learn and make then make data-driven predictions or decisions. • Text mining - the process of deriving high-quality information from text. CAMARADES: Bringing evidence to translational medicine Accumulation of data - machine learning Original search up to Sept 2012 Search update Sept 2012 – Nov 2015 Records identified through online Records identified through online databases databases (n = 65,156) (n =21,362) 31,342 duplicates UniQue records 9482 duplicates UniQue records removed (n = 33,184) removed (n = 11,880) 5,772 excluded Articles screened Articles included based on machine through machine (n = 33,184) learning algorithm (n = 6,180) learning algorithm Included based on duplicate screening Included based on positive predictive (n = 4,568) value (n=3,461) All included studies of animal models of neuropathic pain (n = 8,029) CAMARADES: Bringing evidence to translational medicine • Huge production of unnecessary, misleading and conflicted reviews • often serve as easily publishable units or marketing tools • Suboptimal reviews can be harmful given the prestige