Graphical Methods for Detecting Bias in Meta-Analysis
Total Page:16
File Type:pdf, Size:1020Kb
See discussions, stats, and author profiles for this publication at: https://www.researchgate.net/publication/13513488 Graphical methods for detecting bias in meta-analysis Article in Family medicine · October 1998 Source: PubMed CITATIONS READS 26 702 1 author: Robert Ferrer University of Texas Health Science Center at San Antonio 79 PUBLICATIONS 2,097 CITATIONS SEE PROFILE Some of the authors of this publication are also working on these related projects: Small Troubles, Adaptive Responses View project All content following this page was uploaded by Robert Ferrer on 06 January 2014. The user has requested enhancement of the downloaded file. Vol. 30, No. 8 579 Research Series Graphical Methods for Detecting Bias in Meta-analysis Robert L. Ferrer, MD, MPH The trustworthiness of meta-analysis, a set of techniques used to quantitatively combine results from different studies, has recently been questioned. Problems with meta-analysis stem from bias in selecting studies to include in a meta-analysis and from combining study results when it is inappro- priate to do so. Simple graphical techniques address these problems but are infrequently applied. Funnel plots display the relationship of effect size versus sample size and help determine whether there is likely to have been selection bias in including studies in the meta-analysis. The L’Abbé plot displays the outcomes in both the treatment and control groups of included studies and helps to decide whether the studies are too heterogeneous to appropriately combine into a single measure of effect. (Fam Med 1998;30(8):579-83.) Our faith in the answers provided by scientific in- multiple studies to see the patterns that clarify a line quiry rests on our confidence that its methods are of scientific inquiry. As a way of summarizing re- sound. If we lose confidence in a particular method, search, the systematic review is considered superior we may begin to doubt a whole series of previously to the traditional expert narrative review,2 which is established “truths.” Such skepticism1 has arisen about vulnerable to the biases of its authors. meta-analysis, the “study of studies” designed to com- Enthusiasm for meta-analysis is based on the fact bine the results of a number of different reports into that it provides a quantitative synthesis of all the rel- one report to create a single, more precise estimate of evant and methodologically acceptable evidence.3 an effect. This article will briefly outline why skepti- This enthusiasm is reflected in the logarithmic growth cism has developed about meta-analysis and will of published meta-analyses, from 16 in all of the 1970s present two graphical methods to evaluate the valid- to more than 500 in 1996.4 The Cochrane Collabora- ity of a meta-analysis. tion, a worldwide effort to produce systematic reviews First developed by social scientists as a way to in- to guide evidence-based practice, further exemplifies tegrate and summarize evidence from multiple stud- a strong faith in the validity of meta-analysis. ies, meta-analysis forms the statistical core of the sci- Not everyone is so positive about meta-analysis, ence of systematic review. The central idea behind however. Critics have argued that attempting to sum- systematic reviews is that no single study can pro- marize the results of different studies with a single vide a definitive answer to a medical question. Rather, measure leads to a substantial risk of error,5-7 and progressive understanding develops through a pro- empirical evidence has recently emerged to support cess of synthesizing and integrating the results from their doubts. Several studies have demonstrated dis- cordant results in 10%–34% of comparisons between meta-analyses and large randomized clinical trials (RCTs) on the same topic.8-10 Though there is no a From the Department of Family Practice, University of Texas Health priori basis for believing that RCTs are always cor- Science Center at San Antonio. rect when a discrepancy exists, the 50-year history of 580 September 1998 Family Medicine building knowledge with RCTs has weighed in their favor, and the validity of meta-analyses has become suspect. Table 1 Sources of Bias in Meta-analysis Bias in Meta-analysis The most important biases in meta-analysis arise Type of Bias Definition from two sources. These are 1) the choice of studies Publication bias20 Positive studies* are submitted for 11 publication more frequently and are more included in the meta-analysis and 2) how the results easily published than negative studies.** of those studies are combined to produce a summary effect estimate.7 Multiple publication21 Duplicate publication of study results leads to over-sampling of data from the Just as bias in selecting patients may flaw an epi- same research. demiological study, bias in selecting studies may flaw a meta-analysis. Selection bias in meta-analysis de- English-language bias22 Negative studies are less likely to be published in English-language journals rives from several potential problems with literature than positive studies. Thus, English- searches,11 including publication bias, citation bias, language literature searches fail to retrieve multiple publication, and English-language bias (see negative studies. Table 1 for definitions). Each of these biases increases Citation bias23 Positive studies are cited more frequently the likelihood that meta-analysis will include studies and thus are more easily identified than that demonstrate statistically significant differences negative studies. and fail to include those that do not. * Positive studies—studies that demonstrate statistically significant The other major bias occurs when individual stud- differences ies are combined despite significant heterogeneity in ** Negative studies—studies that do not demonstrate statistically significant differences their results. In such cases, summary measures are misleading and can mask impor- tant subgroup differences in out- come. For instance, variation in treatment effect by patient age Figure 1 would be hidden by an analysis that pooled the results of studies Symmetrical Funnel Plot done in young and old popula- tions. Or, the existence of a threshold dose would be ob- scured by an analysis that com- bined low-dose and high-dose treatment studies. Unfortunately, most meta- analyses are not presented in a way that permits readers to assess whether biases are present, even though there are simple graphi- cal methods that help to do so. These graphical methods include funnel plots and L’Abbé plots. The Funnel Plot The funnel plot is a simple vi- sual tool to examine whether a meta-analysis is based on a bi- ased sample of studies.12 The fun- nel plot is a scatterplot of effect size versus sample includes an unbiased sample of studies, including size, with each data point on the plot representing one studies that find both positive and negative results, study. Because large studies estimate effect size more the funnel is symmetrical. Figure 1 shows a schematic precisely than small studies, they tend to lie in a nar- representation of a symmetrical funnel plot. row band at the top of the scatterplot, while the smaller When the sample of studies is biased, however, the studies, with more variation in results, fan out over a funnel will be asymmetrical. This asymmetry is most larger area at the bottom, thus creating the visual im- commonly the result of smaller studies being biased pression of an inverted funnel. When a meta-analysis toward positive results and larger effect sizes, due to Research Series Vol. 30, No. 8 581 the biases listed in Table 1. Fig- ure 2 shows a funnel plot cre- Figure 2 ated from a meta-analysisl3 that found that calcium supplemen- Asymmetrical Funnel Plot: Meta-analysis of Calcium and Preeclampsia tation during pregnancy was as- sociated with a lower risk of preeclampsia. Marked asymme- try in the funnel plot suggests bi- ased inclusion of studies that showed beneficial effects from calcium. In fact, 15 months af- ter the publication of that meta- analysis, a large randomized trial14 found no benefit from cal- cium supplementation in pre- venting preeclampsia. In many cases, visually in- specting a funnel plot suffices to draw conclusions about possible bias. But, if uncertainty persists about its symmetry, statistical techniques are available to test the hypothesis that a funnel plot is symmetrical.12 Funnel plots may also be used to examine whether a line of in- Figure 3 quiry is converging on a more precise effect estimate over Funnel Plot: Study Result Versus Year of Publication time.15 In this case, effect esti- mate is plotted against year of publication. The schematic ex- ample in Figure 3 shows that the most recent studies have less variability in their results than older studies, suggesting that current research methods are providing more accurate results than older methods. The L’Abbé Plot The L’Abbé plot16 (pro- nounced lah-bay) is another type of scatterplot in which each data point represents one study. The event rate in the control group is plotted against the event rate in the treatment group (the event rate can be either the proportion or the rate at which the endpoint occurred in each group). If the treatment group has better outcomes than the control group, data points fall below the line of identity, which has a slope = 1. If the control has better outcomes than the treatment, the data points fall above the line of identity (Figure 4). The L’Abbé plot best addresses this question about a meta-analysis: how does the treatment effect relate 582 September 1998 Family Medicine to baseline risk? If the relation- ship is similar from study to Figure 4 study, the data points will form a consistent band on the L’Abbé Plot scatterplot, as in the schematic L’Abbé plot in Figure 5, and it is appropriate to calculate a summary effect measure.