Graphical Methods for Detecting Bias in Meta-Analysis

Home , Funnel plot

See discussions, stats, and author profiles for this publication at: https://www.researchgate.net/publication/13513488

Graphical methods for detecting bias in meta-analysis

Article in Family medicine · October 1998 Source: PubMed

CITATIONS READS 26 702

1 author:

Robert Ferrer University of Texas Health Science Center at San Antonio

79 PUBLICATIONS 2,097 CITATIONS

SEE PROFILE

Some of the authors of this publication are also working on these related projects:

Small Troubles, Adaptive Responses View project

All content following this page was uploaded by Robert Ferrer on 06 January 2014.

The user has requested enhancement of the downloaded file. Vol. 30, No. 8 579

Research Series

Graphical Methods for Detecting Bias in Meta-analysis

Robert L. Ferrer, MD, MPH

The trustworthiness of meta-analysis, a set of techniques used to quantitatively combine results from different studies, has recently been questioned. Problems with meta-analysis stem from bias in selecting studies to include in a meta-analysis and from combining study results when it is inappro- priate to do so. Simple graphical techniques address these problems but are infrequently applied. Funnel plots display the relationship of effect size versus sample size and help determine whether there is likely to have been selection bias in including studies in the meta-analysis. The L’Abbé plot displays the outcomes in both the treatment and control groups of included studies and helps to decide whether the studies are too heterogeneous to appropriately combine into a single measure of effect.

(Fam Med 1998;30(8):579-83.)

Our faith in the answers provided by scientific in- multiple studies to see the patterns that clarify a line quiry rests on our confidence that its methods are of scientific inquiry. As a way of summarizing re- sound. If we lose confidence in a particular method, search, the systematic review is considered superior we may begin to doubt a whole series of previously to the traditional expert narrative review,2 which is established “truths.” Such skepticism1 has arisen about vulnerable to the biases of its authors. meta-analysis, the “study of studies” designed to com- Enthusiasm for meta-analysis is based on the fact bine the results of a number of different reports into that it provides a quantitative synthesis of all the rel- one report to create a single, more precise estimate of evant and methodologically acceptable evidence.3 an effect. This article will briefly outline why skepti- This enthusiasm is reflected in the logarithmic growth cism has developed about meta-analysis and will of published meta-analyses, from 16 in all of the 1970s present two graphical methods to evaluate the valid- to more than 500 in 1996.4 The Cochrane Collabora- ity of a meta-analysis. tion, a worldwide effort to produce systematic reviews First developed by social scientists as a way to in- to guide evidence-based practice, further exemplifies tegrate and summarize evidence from multiple stud- a strong faith in the validity of meta-analysis. ies, meta-analysis forms the statistical core of the sci- Not everyone is so positive about meta-analysis, ence of systematic review. The central idea behind however. Critics have argued that attempting to sum- systematic reviews is that no single study can pro- marize the results of different studies with a single vide a definitive answer to a medical question. Rather, measure leads to a substantial risk of error,5-7 and progressive understanding develops through a pro- empirical evidence has recently emerged to support cess of synthesizing and integrating the results from their doubts. Several studies have demonstrated dis- cordant results in 10%Ð34% of comparisons between meta-analyses and large randomized clinical trials (RCTs) on the same topic.8-10 Though there is no a From the Department of Family Practice, University of Texas Health priori basis for believing that RCTs are always cor- Science Center at San Antonio. rect when a discrepancy exists, the 50-year history of 580 September 1998 Family Medicine building knowledge with RCTs has weighed in their favor, and the validity of meta-analyses has become suspect. Table 1 Sources of Bias in Meta-analysis Bias in Meta-analysis The most important biases in meta-analysis arise Type of Bias Definition from two sources. These are 1) the choice of studies Publication bias20 Positive studies* are submitted for 11 publication more frequently and are more included in the meta-analysis and 2) how the results easily published than negative studies.** of those studies are combined to produce a summary effect estimate.7 Multiple publication21 Duplicate publication of study results leads to over-sampling of data from the Just as bias in selecting patients may flaw an epi- same research. demiological study, bias in selecting studies may flaw a meta-analysis. Selection bias in meta-analysis de- English-language bias22 Negative studies are less likely to be published in English-language journals rives from several potential problems with literature than positive studies. Thus, English- searches,11 including publication bias, citation bias, language literature searches fail to retrieve multiple publication, and English-language bias (see negative studies. Table 1 for definitions). Each of these biases increases Citation bias23 Positive studies are cited more frequently the likelihood that meta-analysis will include studies and thus are more easily identified than that demonstrate statistically significant differences negative studies. and fail to include those that do not. * Positive studies—studies that demonstrate statistically significant The other major bias occurs when individual stud- differences ies are combined despite significant heterogeneity in ** Negative studies—studies that do not demonstrate statistically significant differences their results. In such cases, summary measures are misleading and can mask important subgroup differences in outcome. For instance, variation in treatment effect by patient age Figure 1 would be hidden by an analysis that pooled the results of studies Symmetrical Funnel Plot done in young and old popula- tions. Or, the existence of a threshold dose would be ob- scured by an analysis that combined low-dose and high-dose treatment studies. Unfortunately, most meta- analyses are not presented in a way that permits readers to assess whether biases are present, even though there are simple graphical methods that help to do so. These graphical methods include funnel plots and L’Abbé plots.

The Funnel Plot The funnel plot is a simple visual tool to examine whether a meta-analysis is based on a biased sample of studies.12 The funnel plot is a scatterplot of effect size versus sample includes an unbiased sample of studies, including size, with each data point on the plot representing one studies that find both positive and negative results, study. Because large studies estimate effect size more the funnel is symmetrical. Figure 1 shows a schematic precisely than small studies, they tend to lie in a nar- representation of a symmetrical funnel plot. row band at the top of the scatterplot, while the smaller When the sample of studies is biased, however, the studies, with more variation in results, fan out over a funnel will be asymmetrical. This asymmetry is most larger area at the bottom, thus creating the visual im- commonly the result of smaller studies being biased pression of an inverted funnel. When a meta-analysis toward positive results and larger effect sizes, due to Research Series Vol. 30, No. 8 581

the biases listed in Table 1. Fig- ure 2 shows a funnel plot cre- Figure 2 ated from a meta-analysisl3 that found that calcium supplemen- Asymmetrical Funnel Plot: Meta-analysis of Calcium and Preeclampsia tation during pregnancy was as- sociated with a lower risk of preeclampsia. Marked asymmetry in the funnel plot suggests biased inclusion of studies that showed beneficial effects from calcium. In fact, 15 months af- ter the publication of that meta- analysis, a large randomized trial14 found no benefit from calcium supplementation in pre- venting preeclampsia. In many cases, visually in- specting a funnel plot suffices to draw conclusions about possible bias. But, if uncertainty persists about its symmetry, statistical techniques are available to test the hypothesis that a funnel plot is symmetrical.12 Funnel plots may also be used to examine whether a line of in- Figure 3 quiry is converging on a more precise effect estimate over Funnel Plot: Study Result Versus Year of Publication time.15 In this case, effect estimate is plotted against year of publication. The schematic example in Figure 3 shows that the most recent studies have less variability in their results than older studies, suggesting that current research methods are providing more accurate results than older methods.

The L’Abbé Plot The L’Abbé plot16 (pro- nounced lah-bay) is another type of scatterplot in which each data point represents one study. The event rate in the control group is plotted against the event rate in the treatment group (the event rate can be either the proportion or the rate at which the endpoint occurred in each group). If the treatment group has better outcomes than the control group, data points fall below the line of identity, which has a slope = 1. If the control has better outcomes than the treatment, the data points fall above the line of identity (Figure 4). The L’Abbé plot best addresses this question about a meta-analysis: how does the treatment effect relate 582 September 1998 Family Medicine

to baseline risk? If the relationship is similar from study to Figure 4 study, the data points will form a consistent band on the L’Abbé Plot scatterplot, as in the schematic L’Abbé plot in Figure 5, and it is appropriate to calculate a summary effect measure. How- ever, if the plot demonstrates that treatment effect varies ac- cording to baseline risk, then a summary effect measure is misleading. For example, if the studies17 in Figure 6 were to be condensed into a summary effect measure, that summary would obscure the observation that there appears to be a threshold baseline risk of coronary heart disease to realize benefits from treatment. Neutral or harmful effects from treatment are evident at control group rates below 50 per 1,000 person/years, while beneficial ef- Figure 5 fects emerge when baseline risk exceeds 50 per 1,000 person/ L’Abbé Plot: Consistent Treatment Effect years. Such inconsistencies in the treatment effect’s relationship to the baseline event rate should raise doubts about the credibility of a single summary effect measure.

Conclusions Meta-analysis is a potentially valuable technique to synthe- size research data from multiple studies. Its methods are still being refined; published meta- analyses in medicine number slightly more than 2,000, com- pared with more than half a million RCTs.4 Thus, it should not be surprising that the results of RCTs are perceived as more trustworthy. Meta-analysis will evolve as new standards devel- op,l8 and techniques such as meta-regression reach beyond summary effect measures to examine subgroup differences and interaction effects.4,19 As this evolution occurs, diagnostic aids such as funnel plots and L’Abbé plots will permit clinicians and researchers to better evaluate the results of current meta-analyses. Authors of meta-analyses should provide these graphs as a standard part of their data presentation, and readers should expect to see them. Research Series Vol. 30, No. 8 583

Figure 6

L’Abbé Plot of Coronary Heart Disease Trials

Correspondence: Address correspondence to Dr Ferrer, University of 11. Egger M, Davey-Smith G. Bias in location and selection of studies. Texas Health Science Center at San Antonio, Department of Family Prac- BMJ 1998;316:61-6. tice, 7703 Floyd Curl Drive, San Antonio, TX 78284-7795. 210-358- 12. Egger M, Davey-Smith G, Schneider M, Minder C. Bias in meta- 3998. Fax: 210-220-3763. E-mail: [email protected]. analysis detected by a simple graphical test. BMJ 1997;315:629-34. 13. Bucher HC, Guyatt GH, Cook RJ, et al. Effect of calcium supplementation on pregnancy-induced hypertension and preeclampsia. JAMA 1996;275:1113-7. REFERENCES 14. Levine RJ, Hauth JC, Curet LB, et al. Trial of calcium to prevent preeclampsia. N Engl J Med 1997;337:69-76. 1. Bailar JC. The promise and problems of meta-analysis. N Engl J 15. Light R, Pillemer D. Summing up. Cambridge, Mass: Harvard Med 1997;337:559-60. University Press, 1984. 2. Cook DJ, Mulrow CD, Haynes RB. Systematic reviews: synthesis 16. L’Abbe KA, Detsky AS, O’Rourke K. Meta-analysis in clinical re- of best evidence for clinical decisions. Ann Intern Med 1997;126: search. Ann Intern Med 1987;107:224-33. 376-80. 17. Davey-Smith G, Song F, Sheldon TA. Cholesterol lowering and mor- 3. Egger M, Davey-Smith G. Meta-analysis: potentials and promise. tality: the importance of considering initial level of risk. BMJ 1993; BMJ 1997;315:1371-4. 306:1367-73. 4. Lau J, Ioannidis JPA, Schmid CH. Summing up evidence: one an- 18. Pogue J, Yusuf S. Overcoming the limitations of current meta- swer is not always enough. Lancet 1998;351:123-7. analysis of randomized controlled trials. Lancet 1998;351:47-52. 5. Thompson SG, Pocock SJ. Can meta-analyses be trusted? Lancet 19. Davey-Smith G, Egger M, Phillips AN. Meta-analysis: beyond the 1991;338:1127-30. grand mean? BMJ 1997;315:1610-4. 6. Shapiro S. Meta-analysis/schmeta-analysis. Am J Epidemiol 1994; 20. Dickersin K, Min YI, Meinert CL. Factors influencing publication 140:771-8. of research results: follow-up of applications submitted to two insti- 7. Eysenck HJ. Meta-analysis and its problems. BMJ 1994;309: tutional review boards. JAMA 1992;267:374-8. 789-92. 21. Huston P, Moher D. Redundancy, disaggregation, and the integrity 8. Villar J, Carroli G, Belizan JM. Predictive ability of meta-analyses of medical research. Lancet 1996;347:1024-6. of randomized controlled trials. Lancet 1995;345:772-6. 22. Egger M, Zellweger T, Antes G. Language bias in randomized con- 9. Cappelleri JC, Ioannidis JPA, Schmid CH, et al. Large trials versus trolled trials published in English and German. Lancet 1997;350:326-9. meta-analysis of smaller trials: how do their results compare? JAMA 23. Ravnskov U. Cholesterol-lowering trials in coronary heart disease: 1996;276:1332-8. frequency of citation and outcome. BMJ 1992;305:15-9. 10. LeLorier J, Gregoire G, Benhaddad A, Lapierre J, Derderian F. Discrepancies between meta-analyses and subsequent large randomized, controlled trials. N Engl J Med 1997;337:536-42.

View publication stats