What Is Meta-Analysis?

What is ...? For further titles in the series, visit: www.whatisseries.co.uk What is meta-analysis?

Robert Hodgson PhD Research Consultant, York Health Economics Consortium, University of York

G Meta-analysis is a set of statistical G Meta-analysis of trials provides more techniques for combining data from precise estimates of treatment effect, by independent studies to produce a single making use of all available data. estimate of effect. G Meta-analysis is often part of the systematic G Meta-analysis is often used review process, many systematic reviews within healthcare, but is also applied include one or more meta-analyses. in other disciplines including psychology and the social sciences. G The validity of any meta-analysis depends on the studies on which it is based. G Within healthcare, meta-analysis is often used to assess the clinical G Well-conducted meta-analyses aim for effectiveness of interventions; it does complete coverage of all relevant studies, this by combining data from two or look for the presence of heterogeneity more studies (usually randomised among studies, and explore the robustness of controlled trials). the main findings using sensitivity analysis.

EVIDENCE-BASED HEALTHCARE • theory • knowledge • practice

Systematic reviews difficulties that anyone trying to make sense and meta-analysis of effectiveness research might encounter. It is common in medical research to find that several clinical trials have attempted to Small effects and trends answer similar questions about clinical Because meta-analysis may combine data effectiveness; for example: Does the new from a number of studies, it potentially treatment confer significant benefits includes a large number of patients and compared with the conventional may be better placed to examine rare treatment? Often, it is the case that these events, such as safety issues and side effects individual studies will not be able to associated with interventions. The larger conclusively demonstrate whether a sample size achieved by combining a treatment confers significant clinical benefit. number of studies also allows investigators Meta-analysis allows the data from all to examine any differences in intervention relevant trials to be combined and can help effects between specific patient groups, to establish with greater certainty the something that is often not possible within relative clinical benefit of the treatment individual studies. options. A good example of this is a retrospective review of the evidence on the Inconsistency effectiveness of thrombolytic therapy for the The results of studies evaluating a particular prevention of myocardial infarction.1 The intervention can often appear to be study showed that, had meta-analysis been inconsistent or contradictory. Meta-analysis conducted at an early stage, after only a few provides a way of not only estimating trials had been completed, it would have the average likely effect, but also a way demonstrated the benefits of thrombolytic to quantify and explain inconsistencies. therapy. Instead, experts remained unaware This can be achieved through the of its benefits for many years and patients examination of subgroups, as described were not given an effective therapy. Meta- above, or through use of the technique of analyses are now a fundamental part of meta-regression (see below). evidence-based medicine. Overcoming bias When can you When meta-analysis is undertaken as part of do a meta-analysis? a rigorously conducted systematic review, it Meta-analysis can be used whenever there can also help to overcome bias. A well- is more than one study that has estimated conducted systematic review allows data the effect of an intervention or risk factor, from all relevant studies to be included in a and the studies are sufficiently similar in meta-analysis. A meta-analysis carried out as terms of the participants, interventions, part of such a review therefore allows for outcome measurements and settings, so unbiased synthesis of the empirical data. that it is reasonable to combine the results Unsystematic reviews are less likely to of these studies. Meta-analysis need not identify all available evidence and may be only be used to combine the results of influenced by the prior beliefs of the randomised controlled trials (RCTs), but can reviewer. Meta-analysis carried out as part of also be used to combine data from other an unsystematic review is therefore more types of studies, such as case-control and likely to be based on only a proportion of cohort studies to give a relative risk, or the available evidence and also more likely diagnostic test accuracy studies to provide to produce biased estimates of effect. summary estimates of the sensitivity and specificity of a new test compared with an Transparency existing test. As well as potentially reducing bias, meta- analyses conducted as part of a rigorous Benefits of meta-analyses systematic review have the further Meta-analysis offers a rational and helpful advantage of transparency. An important way of dealing with a number of practical aspect of the systematic review approach is

2 www.whatisseries.co.uk

that all the decisions that have been taken (such as funnel plots) and a greater throughout the process of achieving the understanding of the pitfalls of meta- final aggregate effect sizes are made clear. analysis.5 This allows readers to determine for themselves the reasonableness of the Meta-analysis and the decisions taken and their likely impact on systematic review process the final estimate of effect size. It also allows The main requirement for a worthwhile other researchers to replicate their results, meta-analysis is a well-conducted systematic which is an important part of the scientific review.6 Such a review should include an process. extensive search, which interrogates several electronic databases and other information Criticisms of meta-analysis sources to identify as many relevant studies A common criticism of meta-analysis is that as possible. The systematic review should it seeks to combine different kinds of studies also use explicit and objective criteria for the in the same analysis, thereby ignoring inclusion or rejection of studies.7 However important differences between studies. competent the meta-analysis, if the original Careful consideration is necessary when systematic review was partial, flawed or planning a meta-analysis to ensure that the otherwise unsystematic, then the meta- studies included in the analysis are analysis may provide a precise quantitative sufficiently similar so that that any resulting estimate that is simply wrong. The main summary estimate has a clinically requirement of systematic review is easier to meaningful interpretation; for example, it state than to execute: a complete, unbiased makes no sense to combine studies that collection of all the original studies of compare different interventions. It is, acceptable quality that examine the same however, worth emphasising that one of the therapeutic question. Among the many strengths of meta-analysis is its ability to checklists for the assessment of the quality answer broader questions and allow of systematic reviews, the PRISMA statement inconsistencies to be investigated. (Preferred Reporting Items for Systematic Therefore, combining dissimilar studies may Reviews and Meta-Analyses) is particularly be justified when attempting to explain recommended and widely adopted by variations in clinical effectiveness. journal editors.8 Another criticism levelled at meta-analysis is that the results of meta-analyses can Conducting meta-analyses appear to contradict those of subsequent Forest plots RCTs. A number of studies have, for The usual way of displaying data from a example, compared the results of meta- meta-analysis is by a pictorial representation analyses with subsequent findings from (often known as a Forest plot). An example large-scale, well-conducted, RCTs (so-called is shown in Figure 1.9 This displays the ‘mega trials’). The results of such findings from each individual study as a comparisons have, so far, been mixed – blob or square, with squares towards the left there has been good agreement in the side indicating that the new treatment is majority of cases but some discrepancies in better, whereas blobs on the right indicate others.2,3 For example, one such exercise led that the new treatment is less effective than to publication of a paper subtitled ‘Lessons the comparator. The size of the blob or from an “effective, safe, simple intervention” square is proportional to the precision of the that wasn’t’ (use of intravenous magnesium study (roughly speaking, the sample size). A after heart attacks).4 In many of these cases, horizontal line (usually the 95% confidence it emerged that the different analyses were interval) is drawn around each of the studies’ either asking different questions or differed blobs to represent the uncertainty of the in some other way. In some cases, these estimate of the treatment effect. The differences have been explained by flaws in aggregate effect size obtained by the original meta-analyses and have led to a combining all of the studies is usually number of methodological improvements displayed as a diamond.

www.whatisseries.co.uk 3

Figure 1. Presentation of the findings from a meta-analysis9

Beta-blocker deaths No (%) of deaths/patients Logrank Variance Ratio of crude death rates (99% CI) observed – observed – Study Beta-blocker Control expected expected beta-blocker:control Wilcox (oxprenolol) 14/157 (8.9) 10/158 (8.9) 2.0 5.6 Norris (propanolol) 21/226 (9.3) 24/228 (9.3) –1.4 10.2 Multicentre (propanolol) 15/100 (15.0) 12/95 (12.6) 1.2 5.8 Baber (propanolol) 28/355 (7.9) 27/365 (7.4) 0.9 12.7 Andersen (alprenolol) 61/238 (25.6) 64/242 (26.4) –1.0 23.2 Balcon (propanolol) 14/56 (25.0) 15/58 (25.9) –0.2 5.5 Barber (practolol) 47/221 (21.3) 53/228 (23.2) –2.2 19.5 Wilcox (propanolol) 36/259 (13.9) 19/129 (14.7) –0.7 10.5 CPRG (oxprenolol) 9/177 (5.1) 5/136 (3.6) 1.1 3.3 Multicentre (practolol) 102/1,533 (6.7) 127/1,520 (8.4) –13.0 53.0 Barber (propanolol) 10/52 (19.2) 12/47 (25.5) –1.6 4.3 BHAT (propanolol) 138/1,916 (7.2) 188/1,921 (9.8) –24.8 74.6 Multicentre (timolol) 98/945 (10.40 152/939 (16.2) –27.4 54.2 Hjalmarson (metoprolol) 40/698 (5.7) 62/697 (8.9) –11.0 23.7 Wilhelmsson (alprenolol) 7/114 (6.1) 14/116 (12.1) –3.4 4.8

Total* 640/7,047 (9.1) 784/6,879 (11.4) –81.6 310.7

0 0.5 1.0 1.5 2.0 Reduction 23.1% (standard error 5.0) p<0.0001 beta-blocker better beta-blocker worse Heterogeneity between 15 trials: 2 = 13.9; df = 14; p>0.1 χ Treatment effect p<0.0001 * 95% confidence interval as shown for the odds ratio

Calculating effect sizes outcomes are most commonly presented as A range of different clinical outcomes is odds or risk ratios. Odds ratios represent the used in clinical trials. It is important to ratio of odds of a particular event, such as a understand these to be able to conduct and heart attack or death occurring in the interpret the results of a meta-analysis. intervention group relative to the control There are three types of outcomes group. While risk ratios represent the ratio of commonly reported in clinical trials: binary the risk or probability of an event occurring outcomes (where each individual’s outcome in the intervention group relative to the is one of only two possible categorical control group. Effective measures for responses, such as dead or alive); continuous outcomes are most commonly continuous outcomes (where each present as a risk or mean difference, but may individual’s outcome is a measurement of a also be presented as a standardised mean numerical quantity, such as blood pressure difference where the mean difference is or visual acuity); and time-to-event divided by its standard deviation. This is outcomes (that analyse the time until an undertaken to standardise results on a event occurs, such as death or relapse). uniform scale where studies all assess the Binary outcomes are the most common same outcome (for example, anxiety), but type of outcome presented in medical use different measurement scales. For time- research. Measures of effect for binary to-event outcomes, the main effect measure

4 www.whatisseries.co.uk

Copyright © Hayward Medical Communications 2014. All rights reserved. No unauthorised reproduction or distribution. For reprints or permissions, contact [email protected] Copyright cautious one. of publication bias should always be a occur due to other factors, so the conclusion effect. Funnel plot asymmetry can also are probably smaller studies showing no analysis may have missed some trials. These asymmetric, this suggests that the meta- www.whatisseries.co.uk inverted funnel (Figure 2a). expected picture is one of a symmetrical chance variability than larger studies, the plot of effect size against sample size studies included in the meta-analysis in a chance). the findings could be affected by the play of some other measure of the extent to which funnel plot presence of publication bias is to examine a review findings can be investigated. publication bias so its potential impact on empirically assess the existence of meta-analysis is that it allows us to of a treatment benefit. An advantage of estimates of effect that are biased in favour Publication bias can lead to summary conclude the treatment is effective. less likely to be published than those that findings (that is, no benefit of treatment) are whereby clinical trials that obtain negative systematic review is publication bias, A key concern when conducting a Checking for publication bias the study). estimate (roughly equivalent to the size of study are weighted by the precision of the approach in which the estimates from each outcomes differ, they all use a similar used to combine different types of you cross a busy road. While the methods example, your hazard of death changes as risk and may change continuously; for hazard ratio is a measure of instantaneous the risk of an event in the control group. A time in the intervention group relative to (likelihood) of having an event at any given interpreted as the ratio of the risk is the hazard ratio. Hazard ratios are funnel plot asymmetry due to publication (p=0.01, p=0.05), can help to distinguish ‘milestones’ of statistical significance contour lines are associated with One simple way of assessing the likely Contour-enhanced funnel plots, in which © Hayward 14 As smaller studies have more . 12 16 Funnel plots display the Medical Communications 15 If the plot is 2014. 10,11 13 All (or rights reserved. other factors. apparent publication bias can be due to interpreting the results of these tests, as studies. Caution must also be used when the meta-analysis contains ten or more tests should therefore only be used where publication bias, even where it exists. These powerful and they will not detect of these methods is that they are not very tests of publication bias. A problem with all Peters test other tests, including the Begg test effects have not been published). Many (implying that small studies with small larger effect sizes than would be expected tests whether small studies tend to have widely used to test for publication bias. It been criticised as a result. asymmetry by eye and have sometimes plots rely on subjective assessment of use different doses or types of a particular in the type of patients they recruit or may some extent; for example, studies can differ inevitable that these will differstudies to results from different studies and it is Meta-analysis involves combining the Heterogeneity bias from other factors (Figure 2b). The Egger’s regression test been developed to test for publication bias. number of formal statistical methods have Figure 2. ‘missing’ studies are expected) effectiveness of injected cholera vaccines against placebo (the ellipse indicatesthe vaccine likely as opposed areas towhere placebo), b) Contour-funnel plot of a meta-analysiscompared investigating with a placebo the (a negative natural logarithm odds ratio [In(OR)]a) correspondsFunnel plot ofto a favouringmeta-analysis investigating the effectiveness of injected cholera vaccines In addition to these visual assessments, a No Precision (1/sc) unauthorised What is 0 4 6 8 2 a) 3– –1 –2 –3 ■ 0.1>p>0.05 ■ 13 , have also been proposed as Examples of funnel plots reproduction ■ 0.05>p>0.01 ■ meta-analysis? In(OR) 012 or 17 distribution. 12 ■ p<0.01 ●Studies is currently 15 18 For Funnel and reprints Precision (1/sc) 0 4 6 8 2 b) 3– –1 –2 –3 or permissions, Wha 15 In(OR) contact 012 t is ...? [email protected] 5 Reprinted from J Clin Epidemiol, Vol 61, Peters JL, Sutton AJ, Jones DR, Abrams KR, Rushton L, Contour-enhanced meta-analysis funnel plots help distinguish publication bias from other causes of asymmetry, Pages 991–996, Copyright (2008), with permission from Elsevier. What is ...? What is meta-analysis?

intervention. These differences are termed The presence or absence of heterogeneity heterogeneity. Differences in studies are influences the subsequent method of often classified into different types of analysis. If heterogeneity is absent, then the heterogeneity. Heterogeneity resulting from analysis employs what is termed fixed- differences in patient characteristics, effects modelling. This assumes that the size interventions investigated and outcomes is of the treatment effect is the same (fixed) known as clinical heterogeneity. across all studies and that the variation seen Differences in study design, including between studies is due only to the play of aspects such as blinding, are known as chance. Random-effects models assume methodological heterogeneity. These that the treatment effect really does vary differences between the studies included in between studies. Such models tend to a systematic review are important, because increase the width of the confidence differences in the patients or study design interval around any summary estimate, may influence both the size and direction of making it more difficult to obtain significant any treatment effect we observe. results. Where moderate or substantial It is essential when we are conducting a heterogeneity exists it may not be meta-analysis that proper consideration is inappropriate to calculate an overall given to possible heterogeneity, and that summary intervention effect estimate and any resulting summary statistic has we should seek to explain the reasons meaningful clinical interpretation. A key behind the heterogeneity. The technique of difficulty in conducting a meta-analysis meta-regression (see below) was introduced therefore lies in deciding which sets of because it provides one way to help explain studies to include, and which to leave out. heterogeneity. Clearly, to get a precise answer to a specific question, only studies that exactly match the Sensitivity analyses question should be included. Unfortunately, Decisions taken about selection, inclusion studies often differ substantially and it is and methods of analysis may affect the important get clinical input on whether main findings of a systematic review; these differences are likely to influence the because of this, it is usual for meta-analysts magnitude of any treatment effect. to carry out some sensitivity analysis. This Meta-analyses should test thoroughly for explores the ways that the main findings are the existence of heterogeneity. A test which changed according to the approach to was often used was Cochrane’s Q, a statistic combining studies. A good sensitivity based on the chi-squared test.19 analysis will explore, among other things, Unfortunately, this test has low power and the effect of excluding various categories of may sometimes fail to detect heterogeneity studies; for example, unpublished studies or when it is present. To try to overcome this, a those studies at a high risk of bias. It may second test, the I2 statistic, was developed.20 also examine how the results vary by the This test seems attractive because it scores mode of analysis (fixed effect versus random heterogeneity between 0% and 100%. effects) or by various subgroups (perhaps Caution must be taken to avoid defined by patient group, type of overinterpreting the I2 statistic, but, as rule intervention or setting). In a meta-analysis of thumb, less than 40% heterogeneity may without sensitivity analyses, the reader has be termed low and may well be to guess the likely impact of these unimportant, 30% to 60% represents important factors on the key findings. moderate heterogeneity, while 60% or more can be viewed as substantial heterogeneity.7 Advanced techniques The interpretation of the I2 statistic should, The range of meta-analysis techniques has however, make reference to the magnitude grown substantially over the past few years, and direction of any treatment effect with increasingly sophisticated methods observed, and the degree of clinical and being developed. A few of the most methodological heterogeneity between important new and more advanced studies.7 techniques are discussed below.

6 www.whatisseries.co.uk

Meta-regression effectiveness of treatment A compared with When heterogeneity is detected among a treatment C. Mixed-treatment comparison group of studies, it is important to is a special case of network meta-analysis investigate what may have caused it. Meta- and combines direct and indirect evidence regression is a technique that allows for a particular pairwise comparison. These researchers to explore which types of methods allow all of the information on patient-specific factors or study design relative treatment effects to be used and factors contribute to the heterogeneity. The permit a broader range of questions about simplest type of meta-regression uses the relative effectiveness of interventions to summary data from each trial, such as: the be answered. average effect size, the average disease severity at baseline and the average length Conclusion of follow-up. This approach is valuable, but it Meta-analysis is an important part of many has only limited ability to identify important systematic reviews, and a powerful tool that factors, such as those relating to the size of allows us to answer important questions the treatment effect, 21 and can only be about the effectiveness of interventions; applied when we have a relatively large however, if not used appropriately, meta- number of studies. Meta-regression can also analysis can generate erroneous answers. be applied, using individual patient data Care should be taken when using meta- (IPD) (see below), which is better able to analytical techniques to combine data from give answers to the important questions, primary studies. The quality of any meta- such as, ‘which types of patients are most analysis is also fundamentally limited by the likely to benefit from this treatment?’ quality of the underlying studies (the so- called GIGO principle of ‘garbage in, Individual patient data meta-analysis garbage out’). IPD meta-analysis involves obtaining the The field of meta-analysis has expanded raw data from each included study. These rapidly in recent years and continues to data are then combined into a single data develop, with new methodological set and analysed as if they were a single, advances as well as findings from empirical multicentre study. IPD meta-analysis is often research. Two recent books provide seen as a gold standard of systematic review excellent reviews of current knowledge and and meta-analysis, as it has a number of offer far more detail than can be presented advantages over standard meta-analysis in this brief introduction. See below for based on summary statistics. In particular, it further reading G allows consistent analysis to be applied to References all studies and allows greater scope for more 1. Antman EM, Lau J, Kupelnick B, Mosteller F, Chalmers TC. A detailed analyses, particularly the comparison of results of meta-analyses of randomized control trials and recommendations of clinical experts. Treatments for myocardial 22 investigation of sub-groups. While IPD infarction. JAMA 1992; 268: 240–248. 2. LeLorier J, Gregoire G, Benhaddad A, Lapierre J, Derderian F. meta-analysis has a number of advantages, Discrepancies between meta-analyses and subsequent large obtaining the original patient data from randomized, controlled trials. New Engl J Med 1997; 337: 536–542. 3. Cappelleri JC, Ioannidis JP, Schmid CH et al. Large trials vs meta- each of the trials can be challenging.22 analysis of smaller trials: how do their results compare? JAMA 1996; 276: 1332–1338. 4. Egger M, Smith GD. Misleading meta-analysis. BMJ 1995; 310: 752–754. Network meta-analysis and 5. Borenstein M, Hedges LV, Higgins JP, Rothstein HR. Introduction to mixed-treatment analysis meta-analysis. Chichester: John Wiley & Sons, 2009. 6. Bailar JC 3rd. The promise and problems of meta-analysis. N Engl J Network meta-analysis allows indirect Med 1997; 337: 559–561. 7. Higgins J, Green S. Cochrane Handbook for Systematic Reviews of evidence on relative treatment effects to be Interventions, Version 5.1.0. The Cochrane Collaboration, 2011. www.cochrane-handbook.org (last accessed 16 May 2014) used to obtain effects data on interventions 8. Moher D, Liberati A, Tetzlaff J, Altman DG, Group P. Preferred that have not been investigated in head-to- reporting items for systematic reviews and meta-analyses: the PRISMA statement. Int J Surg 2010; 8: 336–341. head trials. Imagine we have two trials, one 9. Lewis S, Clarke M. Forest plots: trying to see the wood and the trees. BMJ 2001; 322: 1479–1480. comparing treatment A with treatment B, 10. Thornton A, Lee P. Publication bias in meta-analysis: its causes and consequences. J Clin Epidemiol 2000; 53: 207–216. and another comparing treatment B with 11. Hopewell S, Loudon K, Clarke MJ, Oxman AD, Dickersin K. treatment C. Network meta-analysis would Publication bias in clinical trials due to statistical significance or direction of trial results. Cochrane Database Syst Rev 2009: allow us to estimate the relative MR000006.

www.whatisseries.co.uk 7

12. Egger M, Davey Smith G, Schneider M, Minder C. Bias in meta- correlation test for publication bias. Biometrics 1994; 50: 1088–1101. analysis detected by a simple, graphical test. BMJ 1997; 315: 19. Ioannidis JP, Patsopoulos NA, Evangelou E. Uncertainty in 629–634. heterogeneity estimates in meta-analyses. BMJ 2007; 335: 914–916. 13. Peters JL, Sutton AJ, Jones DR, Abrams KR, Rushton L. 20. Higgins JP, Thompson SG, Deeks JJ, Altman DG. Measuring Comparison of two methods to detect publication bias in meta- inconsistency in meta-analyses. BMJ 2003; 327: 557–560. analysis. JAMA 2006; 295: 676–680. 21. Schmid CH, Stark PC, Berlin JA, Landais P, Lau J. Meta-regression 14. Sterne JA, Egger M. Funnel plots for detecting bias in meta- detected associations between heterogeneous treatment effects analysis: guidelines on choice of axis. J Clin Epidemiol 2001; 54: and study-level, but not patient-level, factors. J Clin Epidemiol 2004; 1046–1055. 57: 683–697. 15. Peters JL, Sutton AJ, Jones DR, Abrams KR, Rushton L. Contour- 22. Riley RD, Lambert PC, Abo-Zaid G. Meta-analysis of individual enhanced meta-analysis funnel plots help distinguish publication bias participant data: rationale, conduct, and reporting. BMJ 2010; 340: from other causes of asymmetry. J Clin Epidemiol 2008; 61: 991–996. c221. 16. Lau J, Ioannidis JP, Terrin N, Schmid CH, Olkin I. The case of the misleading funnel plot. BMJ 2006; 333: 597–600. Further reading 17. Terrin N, Schmid CH, Lau J. In an empirical evaluation of the 1. Borenstein M, Hedges LV, Higgins JP, Rothstein HR. Introduction to funnel plot, researchers could not visually identify publication bias. J meta-analysis. Chichester: John Wiley & Sons, 2009. Clin Epidemiol 2005; 58: 894–901. 2. Egger M, Smith G, Altman D (eds). Systematic reviews in health care: 18. Begg CB, Mazumdar M. Operating characteristics of a rank Meta-analysis in context. Oxford: John Wiley & Sons, 2008.

First published 2001 Revised edition 2009 Revised edition 2014 This publication, along with the others in the series, is available online at www.whatisseries.co.uk The data, opinions and statements appearing in the article(s) herein are those of the contributor(s) concerned. Accordingly, the publisher, and their respective employees, officers and agents, accept no liability for the consequences of any such inaccurate or misleading data, opinion or statement. Published by Hayward Medical Communications, a division of Hayward Group Ltd, The Pines, Fordham Road, Newmarket CB8 7LG. Tel: 01638 723560. Fax: 01638 723561. email: [email protected] Copyright © 2014 Hayward Group Ltd. All rights reserved. For reprints, permissions or to discuss training packages, email: [email protected]

8 www.whatisseries.co.uk