Meta Analyses – Pros and Cons

Meta analyses – pros and cons

Emily S Sena, PhD Centre for Clinical Brain Sciences, University of Edinburgh

@camarades_

CAMARADES: Bringing evidence to translational medicine CAMARADES

• Collaborative Approach to Meta-Analysis and Review of Animal Data from Experimental Studies • Look systematically across the modelling of a range of conditions • Data Repository – 30 Diseases – 40 Projects – 25,000 studies – from over 400,000 animals

CAMARADES: Bringing evidence to translational medicine Why do we do meta-analysis of animal studies?

• Animal models are generally performed to inform human health but when should you be convinced to move to the next step? • Systematic reviews & meta-analyses: – assess the quality and range of evidence – identify gaps in the field – quantify relative utility of outcome measures – inform power/sample size calculations – assess for publication bias – try to explain discrepancies between preclinical and clinical trial results – inform clinical trial design

CAMARADES: Bringing evidence to translational medicine Systematic review and meta-analysis

• Pros – What have we learnt about…….. • Translation? • Quality? • The 3Rs? • Cons – What are the limitations? • As good as the data that goes in? • Rapidly outdated • The Impact

CAMARADES: Bringing evidence to translational medicine Systematic review and meta-analysis

CAMARADES: Bringing evidence to translational medicine 1,026 interventions in experimental stroke

In vitro and in vivo - 1026

Tested in vivo - 603

Effective in vivo - 374

Tested in clinical trial - 97

Effective in clinical trial - 1

O’Collins et al, 2006 CAMARADES: Bringing evidence to translational medicine Data from in vivo studies

• There are huge amounts of often confusing data Hypothermia: a systematic • Systematic review can help search identified 222 to make sense of it experiments in 3353 animals • If you select extreme bits of the evidence you can “prove” either harm or

substantial benefit Better • Investigating the sources behind this variation may

be helpful in translation Worse

Van der Worp et al Brain 2007 CAMARADES: Bringing evidence to translational medicine You can usually find what you’re looking for …

• 12 graduate psychology students • 5 day experiment: rats in T maze with dark arm alternating at random, and the dark arm always reinforced • 2 groups – “Maze Bright” and “Maze dull”

Group Day Day Day Day Day 1 2 3 4 5 “Maze 1.33 1.60 2.60 2.83 3.26 bright” “Maze 0.72 1.10 2.23 1.83 1.83 dull” Δ +0.60 +0.50 +0.37 +1.00 +1.43

Rosenthal and Fode (1963), Behav Sci 8, 183-9 CAMARADES: Bringing evidence to translational medicine Non-random sample

• N=2671 • Most published in vivo research is at high risk of bias

CAMARADES: Bringing evidence to translational medicine Improvement over time…..

CAMARADES: Bringing evidence to translational medicine Internal Validity: Lessons from NXY-059

• Infarct Volume – 11 publications, 29 experiments, 408 animals – Improved outcome by 44% (35-53%) è Efficacy

Randomisation Blinded conduct Blinded of experiment assessment of outcome

Macleod et al, 2008 CAMARADES: Bringing evidence to translational medicine Random sample from PubMed

CAMARADES: Bringing evidence to translational medicine Good ‘quality’ journals

CAMARADES: Bringing evidence to translational medicine The ‘best’ institutions: RAE 1173

CAMARADES: Bringing evidence to translational medicine Reporting of randomisation across 3 datasets, 2009-10

CAMARADES: Bringing evidence to translational medicine Experimental Design: - choice of strain

Van Drongelen et al. 2012 – vascular function during pregnancy

CAMARADES: Bringing evidence to translational medicine Experimental Design: - indicative power calculations Mechanical induced outcomes Calculated Behavioural No. of Median Calculated Median Effect sample size Test experiments N power Size (IQRs) (power=0.8)

von Frey 48 9 0.3 1.3 (0.9-1.8) 11 (electronic) von Frey 369 11 0.5 1.5 (0.8-2.4) 9 (filaments)

Pin prick 12 11 0.6 1.6 (0.3-8.3) 8

Randall-Selitto 156 10 0.8 1.9 (1.0-3.8) 6 paw pressure

CAMARADES: Bringing evidence to translational medicine Anti-emetic research - refine duration of experiments Subgroup Effect size 5mg (24h) 0.00 [-0.15, 0.15] (P = 1.00) § Ondansetron protects 50% of the patients 10mg (24h) treated with Cisplatin 0.00 [-0.23, 0.23] (P = 1.00)

§ Ondansetron reduced the number of 10mg (6h) animal developing emesis 0.11 [-0.15, 0.38] (P = 0.40) 10mg (4h) § Efficacy was dependent on dosage and 0.48 [0.26, 0.70] (P < 0.0001) duration of the observation period § Supporting evidence to improve the model 10mg (2h) § Provides evidence supporting refinement 0.76 [0.31, 1.21] (P = 0.0010) of the model (4h instead of 24h)

Total -2 0 2 0.33 [0.17, 0.48] (P < 0.0001) Favours Favours control ondansetron

Percie du Sert N et al (2011) Cancer Chemother Pharmacol 67(3): 667-686.

CAMARADES: Bringing evidence to translational medicine Systematic review and meta-analysis

CAMARADES: Bringing evidence to translational medicine Correlation vs Causation

I’m not saying Susan Boyle caused Americans to support gay marriage

70% Oppose legalisation of gay marriage 60% 57% 50% 50%

40% 43% 35%

30% 2009: SuBo wows Support legalisation the world with “I of gay marriage Dreamed a Dream” 20% 2000 2002 2004 2006 2008 2010 2012 2014

But Susan Boyle happened and then support of gay marriage increased

CAMARADES: Bringing evidence to translational medicine Apples vs Oranges

Vs.

• Some argue that where you have too much heterogeneity meta-analysis is not appropriate • What’s the purpose? • Animals vs patients • Collinearity • Limits what you can infer

CAMARADES: Bringing evidence to translational medicine Only as good as the data that go in

Not all outcomes and a priori analyses are reported • Publication bias – Neutral and negative studies – Time lag/remain unpublished – Less likely to be identified • p-hacking – Selective analysis – Selective outcome reporting

CAMARADES: Bringing evidence to translational medicine 0.5 0.5

0.4 0.4

0.3 0.3

0.2

Precision 0.2 Precision

0.1

0.1 0.0

0.0 -0.1 -150 -100 -50 0 50 100 150 -10 -5 0 5 10 15 20 25 Effect Size Effect Size/Standard Error

CAMARADES: Bringing evidence to translational medicine Overall efficacy was reduced from; 32% (95% CI 30 to 34%) to 26% (95% CI 24 to 28%)

CAMARADES: Bringing evidence to translational medicine Publication bias in experimental stroke • Trim and Fill suggested 16% of experiments remain unpublished • Best estimate of magnitude of problem – Overstatement of efficacy 31% • Only 2% publications reported no significant treatment effects

CAMARADES: Bringing evidence to translational medicine Different patterns of publication bias in different fields

outcome observed corrected Disease improvement 40% 30% Less models improvement Toxicology harm 0.32 0.56 More harm model

Harm Benefit

CAMARADES: Bringing evidence to translational medicine Rate of publication

• There are more papers published in a day than most people could read in a month • In 2013, 4700 new publications were added to PubMed every working day Domain Number In vivo and in vitro 610 In vivo 350 Pharmacology 76 Neurosciences 52 • If a panelist for a neuroscience grant awarding body were to try to keep up to date, spending 30 minutes reading each paper, and did nothing else but read papers all year, they would get through 30% of the total

CAMARADES: Bringing evidence to translational medicine Literature Search September 2012 Records identified through online databases (n= 65,156) Identification Unique records 31,342 duplicates (n= 33,184) removed

Articles screened Screening (n= 33,184) Eligibility

Articles screened Screening (n= 33,184)

Included based on dual screening

Eligibility (n= 4,568)

CAMARADES: Bringing evidence to translational medicine Machine Learning & Text Mining

• Machine learning – form of artificial intelligence by which computers learn without being explicitly programmed. Construction of algorithms which learn and make then make data-driven predictions or decisions. • Text mining - the process of deriving high-quality information from text.

CAMARADES: Bringing evidence to translational medicine Accumulation of data - machine learning

Original search up to Sept 2012 Search update Sept 2012 – Nov 2015

Records identified through online Records identified through online databases databases (n = 65,156) (n =21,362)

31,342 duplicates Unique records 9482 duplicates Unique records removed (n = 33,184) removed (n = 11,880)

5,772 excluded Articles screened Articles included based on machine through machine (n = 33,184) learning algorithm (n = 6,180) learning algorithm

Included based on duplicate screening Included based on positive predictive (n = 4,568) value (n=3,461)

All included studies of animal models of neuropathic pain (n = 8,029)

CAMARADES: Bringing evidence to translational medicine • Huge production of unnecessary, misleading and conflicted reviews • Often serve as easily publishable units or marketing tools • Suboptimal reviews can be harmful given the prestige and influence these type of studies have acquired • They should be realigned to remove bias and vested interests • They should be integrated better with the primary production of evidence CAMARADES: Bringing evidence to translational medicine Research & Bias

• All research is susceptible to bias • Systematic reviews and meta-analyses – Developing field of research with a number of different approaches to its conduct and reporting – provide empirical evidence to spur the field to improve the rigor (& reduce bias) of preclinical research • Our worst bias is meta – being more aware of biases makes us more willing to assume that others’ biases, and not ours, are responsible for our disagreement

CAMARADES: Bringing evidence to translational medicine Systematic review and meta-analysis

CAMARADES: Bringing evidence to translational medicine Impact

• Research improvement – Changes to research practice – Development of tools to improve research quality – Important to assess whether interventions improve research quality?

CAMARADES: Bringing evidence to translational medicine • In the hierarchy of evidence randomised controlled trials are 80 now considered the gold standard 60 in clinical trial design. • It required empirical evidence to 40 revolutionise clinical research and

% papers 20 to convince trialists of the significant importance of methodological rigor 0 in both the conduct and reporting of Nonrandom Unblinded Blinded assignment randomisation randomisation their studies. CAMARADES: Bringing evidence to translational medicine IL-1RA in animal models of stroke

100

80 “The 2009 systematic review highlighted areas of weakness with respect the lack of reporting on certain aspects of experimental design. While we did not 70 necessarily agree with all recommendations and also felt that not-reported did not 60 mean not done we did take on board that future studies did need to more fully 50 report details of experimental design. This change is reflected in the positive

40 outcome of the follow-up 2016 systematic review”

30 --- Professor Stuart Allan, University of Manchester 20

0 Random allocation to groups Blinded induction of ischeamia Blinded assessment of Sample size calculation outcome

2009 2016 McCann SK, Cramond F, Macleod MR, Sena ES (2016). Systematic Review and Meta-Analysis of the Efficacy of Interleukin-1 Receptor Antagonist in Animal Models of Stroke: an Update. Translational stroke research 7(5): 395-406. CAMARADES: Bringing evidence to translational medicine Tools designed to help

• NC3Rs Experimental Design Assistant

https://www.nc3rs.org.uk/experimental-design-assistant-eda

CAMARADES: Bringing evidence to translational medicine Tools designed to help

• Disease specific guidelines

CAMARADES: Bringing evidence to translational medicine Tools designed to help

• The ARRIVE guidelines

§ Checklist of 20 items, containing key information necessary to describe a study comprehensively and transparently.

§ Consensus between: § Scientists § Statisticians § Journal editors § Research funders

§ Used to ensure transparent and comprehensive reporting

https://www.nc3rs.org.uk/arrive-guidelines

CAMARADES: Bringing evidence to translational medicine Randomised study

– Reporting of in vivo research published in a major journal – RCT of mandating authors to submit a completed ARRIVE checklist – Outcome: proportion of studies fully compliant with ARRIVE – Status: randomisation complete, outcome ascertainment in progress

CAMARADES: Bringing evidence to translational medicine Key messages

• Systematic review & meta-analysis have been very useful: – Most in vivo studies do not report simple measures to reduce bias – In vivo studies which do not report simple measures to avoid bias give larger estimates of treatment effects – Publication and selective outcome reporting biases are important and prevalent • You can only find these things out by studying large numbers of studies

CAMARADES: Bringing evidence to translational medicine But…. • Any experimental design can be subverted; what’s important is knowing how to recognise when this has happened • They do not replace hypothesis-testing experiments • Technological tools are required to ensure their continued relevance • Ensure they’re warranted, are of a high quality and that they fulfil a purpose

CAMARADES: Bringing evidence to translational medicine Thanks to......

CAMARADES: Bringing evidence to translational medicine Training in critical appraisal of publications in biomedicine

https://ecrf1.clinicaltrials.ed.ac.uk/iicarus/Training https://ecrf1.clinicaltrials.ed.ac.uk/npqip/Review/TrainingCover

CAMARADES: Bringing evidence to translational medicine • Easy to use platforms for systematic review (SyRF)

http://syrf.org.uk/

CAMARADES: Bringing evidence to translational medicine Observational study

– In May 2013, NPG changed instructions for authors to require full details of experimental design – Before and after study of compliance with those requirements in • (1) papers in NPG journals and • (2) similar papers published at same time in other journals – Status: Outcome assessment in progress

CAMARADES: Bringing evidence to translational medicine