Graphical Displays in Meta-Analysis.Pptx

Graphical Representation of Meta-analysis Findings

Emily E. Tanner-Smith Associate Editor, Campbell Methods Coordinating Group Research Assistant Professor, Vanderbilt University

Campbell Collaboration Colloquium Chicago, IL May 22nd, 2013

The Campbell Collaboration www.campbellcollaboration.org

Outline

• Introduction • Forest plots • Funnel plots • Bubble plots • Other graphs • Software resources • Summary

The Campbell Collaboration www.campbellcollaboration.org

1 Introduction

• Graphs are an essential tool for conveying the results of a meta-analysis to readers • But if poorly constructed, graphs can be misleading and/or confuse readers • Graphs should strive for accuracy, simplicity, clarity, and aesthetics • This workshop will provide an overview of expectations and guidelines for graphical displays of meta-analysis results in Campbell Collaboration reviews

The Campbell Collaboration www.campbellcollaboration.org

Introduction: Basic Graphing Principles

• Descriptive titles and/or captions • Use of legends (when appropriate) • Representative range of scale • Properly labeled axes • Inclusion of reference points on axes • Graphs should reflect the statistical precision of results • Explicit mention of any excluded data • Data in graphs should generally be available elsewhere in the review (except in very large reviews) • Aesthetics (line thickness, symbol size, symbol types, parsimony)

The Campbell Collaboration www.campbellcollaboration.org

2 FOREST PLOTS

The Campbell Collaboration www.campbellcollaboration.org

Forest plots

• The “workhorse” graph in meta-analysis • Display effect size estimates and confidence intervals for each study included in the meta-analysis • Effect size estimates typically shown with blocks proportionate to the weight assigned to a given study – Functions to draw the eye toward studies with larger sample size/larger weights, and away from smaller studies with wider confidence intervals

The Campbell Collaboration www.campbellcollaboration.org

3 Forest plots

• Estimated mean effect size with confidence interval shown at the bottom, typically with a diamond • In random effects meta-analyses, prediction intervals can be used to display dispersion in the estimated effect • Studies should be ordered in a meaningful way – Effect size magnitude – Study weight (precision) – Chronological order – Other meaningful study characteristic

The Campbell Collaboration www.campbellcollaboration.org

Forest plots

Source: Regehr, C., Alaggia, R., Dennis, J., Pitts, A., & Saini, M. (2013). Interventions to reduce distress in adult victims of sexual violence and rape. Campbell Systematic Reviews, 3. doi:10.4073/csr.2013.3

The Campbell Collaboration www.campbellcollaboration.org

4 Forest plots

The Campbell Collaboration www.campbellcollaboration.org

Forest plots

The Campbell Collaboration www.campbellcollaboration.org

5 Forest plots

The Campbell Collaboration www.campbellcollaboration.org

Forest plots

The Campbell Collaboration www.campbellcollaboration.org

6 Forest plots

The Campbell Collaboration www.campbellcollaboration.org

Forest plots

The Campbell Collaboration www.campbellcollaboration.org

7 Forest plots

Source: Maynard, B. R., McCrea, K. T., Pigott, T. D., & Kelly, M. S. (2012). Indicated truancy interventions: Effects on school attendance among chronic truant students. Campbell Systematic Reviews, 10. doi:10.4073/csr. 2012.10 The Campbell Collaboration www.campbellcollaboration.org

Forest plots

8 Forest plots with subgroups

• Display effect size estimates and confidence intervals for each study, split by some grouping variable • Useful for depicting results from subgroup or moderator analyses • May include the overall summary effect across groups, if appropriate • Results from statistical tests of moderation

(e.g., QB or b from a meta-regression) should be summarized on the graph or in footnotes, when appropriate

The Campbell Collaboration www.campbellcollaboration.org

Study Hedges' g (95% CI)

subgroups Multi-Session Intervention Chang, 1997 0.44 (0.20, 0.68) Liu, 1992 0.49 (0.21, 0.77) Mapleson, 2001 0.65 (0.41, 0.89) Steiner, 2005 0.71 (0.40, 1.02) Lancaster, 2009 0.74 (0.53, 0.95) Subtotal 0.61 (0.49, 0.73) . (0.36, 0.86)

-1.02 0 1.02 Favors Control Favors Treatment

Source: Fictional data

The Campbell Collaboration www.campbellcollaboration.org

9 Study Hedges' g (95% CI)

-1.02 0 1.02 Favors Control Favors Treatment

Source: Fictional data

The Campbell Collaboration www.campbellcollaboration.org

Study Hedges' g (95% CI)

Single Session Intervention Jones, 2012 0.11 (-0.09, 0.31) Wilson, 2008 0.22 (-0.12, 0.56) Smith, 2011 0.34 (0.06, 0.62) Forest Walters, 2000 0.45 (0.21, 0.69) Milton, 1999 0.48 (0.28, 0.68) Subtotal 0.32 (0.17, 0.48) plots with . (-0.16, 0.81) subgroups Multi-Session Intervention Chang, 1997 0.44 (0.20, 0.68) Liu, 1992 0.49 (0.21, 0.77) Mapleson, 2001 0.65 (0.41, 0.89) Steiner, 2005 0.71 (0.40, 1.02) Lancaster, 2009 0.74 (0.53, 0.95) Subtotal 0.61 (0.49, 0.73) . (0.36, 0.86)

Overall 0.46 (0.33, 0.60) . (0.04, 0.89)

-1.02 0 1.02 Favors Control Favors Treatment

Note: Significant difference in mean effect sizes between groups (b = .28, se = .10, 95% CI [.05, .52]).

Source: Fictional data

The Campbell Collaboration www.campbellcollaboration.org

10 Study Hedges' g (95% CI)

Overall 0.46 (0.33, 0.60) . (0.04, 0.89)

-1.02 0 1.02 Favors Control Favors Treatment

Note: Significant difference in mean effect sizes between groups (b = .28, se = .10, 95% CI [.05, .52]).

Source: Fictional data

The Campbell Collaboration www.campbellcollaboration.org

Study Hedges' g (95% CI)

Overall 0.46 (0.33, 0.60) . (0.04, 0.89)

-1.02 0 1.02 Favors Control Favors Treatment

Note: Significant difference in mean effect sizes between groups (b = .28, se = .10, 95% CI [.05, .52]).

Source: Fictional data

The Campbell Collaboration www.campbellcollaboration.org

11 Summary forest plots

• Display summary (mean) effect sizes and confidence intervals for different groups of studies • Does not include effect size estimates from individual studies • Useful for very large reviews where traditional forest plots may not be feasible, but effects can be categorized into meaningful groups (e.g., across intervention, study, participant types) • May include the overall summary effect across groups, if appropriate

• Results from statistical tests of moderation (e.g., QB or b from a meta-regression) should be summarized on the graph or in footnotes, when appropriate

The Campbell Collaboration www.campbellcollaboration.org

Study Location Hedges' g (95% CI)

United States Subtotal 0.32 (0.22, 0.43)

Canada Summary Subtotal 0.61 (0.50, 0.72) forest plots South America Subtotal 0.61 (0.50, 0.72)

United Kingdom Subtotal 0.02 (-0.10, 0.13)

Spain Subtotal 0.08 (-0.04, 0.21)

China Subtotal 0.35 (0.24, 0.46)

Africa Subtotal 0.46 (0.35, 0.57)

Overall 0.36 (0.32, 0.40)

-.723 0 .723 Favors Control Favors Treatment Source: Fictional data

The Campbell Collaboration www.campbellcollaboration.org

12 Cumulative meta-analysis forest plots

• Display results from iterative estimation of summary (mean) effect sizes, cumulatively adding one study at a time • Useful for showing the accumulation of evidence over time, or the in/stability of intervention effects over time • May also be used to explore small sample bias, cumulatively adding studies according to sample size of primary studies • Title should clearly specify it is a forest plot showing results from a cumulative meta-analysis

The Campbell Collaboration www.campbellcollaboration.org

Study Log odds Name ratio (95% CI)

Fletcher (1959) -1.84 (-4.23, 0.55) Dewar (1963) -1.04 (-2.26, 0.18) 1st European (1969) -0.01 (-0.65, 0.63) Heikinheimo (1971) 0.10 (-0.36, 0.56) Italian (1971) 0.07 (-0.31, 0.46) Cumulative 2nd European (1971) -0.21 (-0.47, 0.05) 2nd Frankfurt (1973) -0.30 (-0.54, -0.06) 1st Australian (1973) -0.30 (-0.52, -0.07) meta- NHLBI SMIT (1974) -0.27 (-0.49, -0.05) Valere (1975) -0.25 (-0.47, -0.04) Frank (1975) -0.24 (-0.46, -0.03) analysis UK Collab (1976) -0.22 (-0.41, -0.03) Klein (1976) -0.21 (-0.40, -0.02) forest plots Austrian (1977) -0.27 (-0.45, -0.10) Lasierra (1977) -0.28 (-0.45, -0.11) N German (1977) -0.21 (-0.37, -0.05) Witchitz (1977) -0.21 (-0.37, -0.05) 2nd Australian (1977) -0.21 (-0.36, -0.06) 3rd European (1977) -0.26 (-0.41, -0.11) ISAM (1986) -0.24 (-0.38, -0.11) GISSI-1 (1986) -0.23 (-0.31, -0.14) ISIS-2 (1988) -0.26 (-0.32, -0.19)

-4.23 0 4.23 Favors Treatment Favors Control

Source: Data accessed at http://www.stata-press.com/data/mais.html from Lau, J., Elliott, M., Antman, M. D., Jimenez-Silva, J., Kupelnick, B., Mosteller, F., & Chalmers, T. C. (1992). Cumulative meta-analysis of therapeutic trials for myocardial infarction. The New England Journal of Medicine, 327, 248-254. The Campbell Collaboration www.campbellcollaboration.org

13 General suggestions – forest plots

• Always include forest plots (or summary forest plots) if possible/appropriate • Not recommended with fewer than 2 studies • Plot ratio effect size measures on the log scale, but include axis labels on the original anti-logged scale • Include reference lines at the null value • State the confidence level for confidence intervals • Blocks for each study should be proportionate to study weight • Sort studies in a meaningful order (e.g., effect size magnitude) • State the direction of results • Include prediction intervals for random effects analyses • Include numerical data on plots (if possible)

The Campbell Collaboration www.campbellcollaboration.org

What’s wrong with this forest plot?

Study ID

1 2 3 4 5 6 7 8 9 10 Overall (I-squared = 74.6%, p = 0.000)

-3.31 0 3.31

The Campbell Collaboration www.campbellcollaboration.org

14 What’s wrong with this forest plot?

Study Uninformative study labels ID

1 2 3 4 5 6 7 8 9 10 Overall (I-squared = 74.6%, p = 0.000)

-3.31 0 3.31

The Campbell Collaboration www.campbellcollaboration.org

What’s wrong with this forest plot?

Study Uninformative study labels ID Seemingly random order 1 of effect sizes 2 3 4 5 6 7 8 9 10 Overall (I-squared = 74.6%, p = 0.000)

-3.31 0 3.31

The Campbell Collaboration www.campbellcollaboration.org

15 What’s wrong with this forest plot?

Study Uninformative study labels ID Seemingly random order 1 of effect sizes 2 3 4 Unclear direction of 5 effect sizes 6 7 8 9 10 Overall (I-squared = 74.6%, p = 0.000)

-3.31 0 3.31

The Campbell Collaboration www.campbellcollaboration.org

What’s wrong with this forest plot?

Study Uninformative study labels ID Seemingly random order 1 of effect sizes 2 3 4 Unclear direction of 5 effect sizes 6 7 Does not include data 8 9 10 Overall (I-squared = 74.6%, p = 0.000)

-3.31 0 3.31

The Campbell Collaboration www.campbellcollaboration.org

16 What’s wrong with this forest plot?

Study Uninformative study labels ID Seemingly random order 1 of effect sizes 2 3 4 Unclear direction of 5 effect sizes 6 7 Does not include data 8 9 10 Unspecified confidence Overall (I-squared = 74.6%, p = 0.000) level

-3.31 0 3.31

The Campbell Collaboration www.campbellcollaboration.org

What’s wrong with this forest plot?

General aesthetics -3.31 0 3.31 (white space)

The Campbell Collaboration www.campbellcollaboration.org

17 FUNNEL PLOTS

The Campbell Collaboration www.campbellcollaboration.org

Funnel plots

• Exploratory tool used to visually assess the possibility of publication/small study bias in a meta-analysis • Scatter plot of effect size (x-axis) against some measure of study size (y-axis) – x-axis: use log scale for ratio effect size measures, e.g., ln(OR), ln(RR) – y-axis: the standard error of the effect size is generally recommended (see Sterne et al., 2005 for a review of additional y-axis options) • Not recommended in very small meta-analyses (e.g., n < 10)

The Campbell Collaboration www.campbellcollaboration.org

18 Funnel plots

• If publication bias is present, you would expect null or ‘negative’ findings from small n studies to be suppressed (i.e., missing from the plot) • Asymmetry in the funnel plot for small n studies may provide evidence of possible publication bias • Symmetry in the funnel plot provides some evidence against the possibility of publication bias

The Campbell Collaboration www.campbellcollaboration.org

Funnel plot with pseudo 95% confidence limits 0 1

Funnel 2 plots 3 Standard Error for LOR Standard for Error 4 5

-10 -5 0 5 10 Log Odds Ratio

Source: Wilson, S. J., Tanner-Smith, E. E., Lipsey, M. W., Steinka-Fry, K., & Morrison, J. (2011). Dropout prevention and intervention programs: Effects on school completion and dropout among school aged children and youth. Campbell Systematic Reviews, 8. doi: 10.4073/csr.2011.8

The Campbell Collaboration www.campbellcollaboration.org

19 Funnel plots

Source: Mazerolle , L., Bennett, S., Davis, J., Sargeant, E., & Manning, M. (2013). Legitimacy in policing: A systematic review. Campbell Systematic Reviews,1. doi:10.4073/csr.2013.1

The Campbell Collaboration www.campbellcollaboration.org

Funnel plots

• Asymmetry could be due to factors other than publication bias, e.g., – Poor methodological quality – Other reporting biases – Artefactual variation – Chance – True heterogeneity • Assessing funnel plot symmetry relies entirely on subjective visual judgment

The Campbell Collaboration www.campbellcollaboration.org

20 Contour enhanced funnel plots

• Funnel plot with additional contour lines associated with ‘milestones’ of statistical significance: p = .001, .01, .05, etc. – If studies are missing in areas of statistical non-significance, publication bias may be present – If studies are missing in areas of statistical significance, asymmetry may be due to factors other than publication bias – If there are no studies in areas of statistical significance, publication bias may be present • Can help distinguish funnel plot asymmetry due to publication bias versus other factors

The Campbell Collaboration www.campbellcollaboration.org

0 Studies p < 1% Contour 1% < p < 5% 5% < p < 10% enhanced p > 10% .5 funnel plots Standard error 1

1.5 -4 -2 0 2 4 Log odds ratio (lor)

Source: Data accessed at http://www.stata-press.com/data/mais.html

The Campbell Collaboration www.campbellcollaboration.org

21 General suggestions – funnel plots

• Not recommended with fewer than 10 studies • Plot effect sizes on the horizontal axis • Plot the standard error of the effect size on the vertical axis (generally) • Plot ratio effect size measures on the log scale, but include axis labels on the original anti-logged scale • All points should be the same size (weights/precision represented in the vertical axis) • Include 95% pseudo-confidence limits from a fixed effect analysis • Include contours if possible • Data in graphs should generally be available elsewhere in the review (except in very large reviews) • Use different plotting symbols to distinguish subgroups, when appropriate The Campbell Collaboration www.campbellcollaboration.org

What’s wrong with this funnel plot? .2 .1 0 Effect size Effect -.1 -.2 0 .05 .1 .15 .2 Standard error

The Campbell Collaboration www.campbellcollaboration.org

22 What’s wrong with this funnel plot?

Effect size on vertical axis .2 .1 0 Effect size Effect -.1 -.2 0 .05 .1 .15 .2 Standard error

The Campbell Collaboration www.campbellcollaboration.org

What’s wrong with this funnel plot? Effect size

-.2 -.1 0 .1 .2

0 Effect size on vertical axis .05 Standard error Standard .1 .15 .2

The Campbell Collaboration www.campbellcollaboration.org

23 What’s wrong with this funnel plot?

Effect size on vertical axis .2

Points are not all the same

.1 size 0 Effect size Effect -.1 -.2 0 .05 .1 .15 .2 Standard error

The Campbell Collaboration www.campbellcollaboration.org

What’s wrong with this funnel plot?

Effect size on vertical axis .2

Points are not all the same

.1 size

Vague labeling of axes and

0 reference line Effect size Effect -.1 -.2 0 .05 .1 .15 .2 Standard error

The Campbell Collaboration www.campbellcollaboration.org

24 What’s wrong with this funnel plot?

Effect size on vertical axis .2

Points are not all the same

.1 size

Vague labeling of axes and

0 reference line Effect size Effect No confidence bands -.1 -.2 0 .05 .1 .15 .2 Standard error

The Campbell Collaboration www.campbellcollaboration.org

Bubble plots

• Scatter plot of a study covariate (x-axis) against effect size (y-axis) • Useful to characterize covariates that may be a source of heterogeneity • Provides a visual representation of results from a bivariate meta-regression model

The Campbell Collaboration www.campbellcollaboration.org

25 Bubble plots 0.00 -1.00 -2.00 Standardizedmean differencesize effect -3.00 4 6 8 10 12 Duration of follow-up (weeks)

Source: Data accessed at http://www.stata-press.com/data/mais.html from Thompson, S. G., & Sharp, S. G. (1999). Explaining heterogeneity in meta- analysis: A comparison of methods. Statistics in Medicine, 18, 2693-2708.

The Campbell Collaboration www.campbellcollaboration.org 1 Bubble plots 0 -1 -2 Standardizedmeandifference -3

4 6 8 10 12 Duration of follow-up (weeks)

Confidence interval Linear prediction SMD effect size Prediction including random effects

The Campbell Collaboration www.campbellcollaboration.org

26 General suggestions – bubble plots

• Plot effect sizes on the vertical axis • Plot the covariate on the horizontal axis • Plot ratio effect size measures on the log scale, but include axis labels on the original anti-logged scale • Points should be proportionate to study weight • Include fitted meta-regression line (if appropriate) • Data in graphs should generally be available elsewhere in the review (except in very large reviews)

The Campbell Collaboration www.campbellcollaboration.org

Other graphs – Galbraith/radial plots

• Scatter plot of inverse standard error (x-axis) against a standardized effect size (i.e., effect size divided by its standard error) (y-axis) • Includes an unweighted regression line constrained through the origin with slope equal to the fixed effect summary effect size estimate • Useful for displaying heterogeneity and aiding detection of outliers • Useful for displaying effect sizes in very large reviews where forest plots may be impractical

The Campbell Collaboration www.campbellcollaboration.org

27 Other graphs – Galbraith/radial plots

2 Favors control Favorscontrol

0 g/SE(g)

-2

Favorstreatment

-6.21956

0 4.16667 1/SE(g)

The Campbell Collaboration www.campbellcollaboration.org

Other graphs – Galbraith/radial plots

More precise

2 estimates lie farther from the Favors control Favorscontrol

origin

0 g/SE(g)

-2

Favorstreatment

-6.21956

0 4.16667 1/SE(g)

The Campbell Collaboration www.campbellcollaboration.org

28 Other graphs – Galbraith/radial plots

More precise

2 estimates lie farther from the Favors control Favorscontrol

origin

0 g/SE(g)

-2 Vertical scatter

Favorstreatment illustrates heterogeneity

-6.21956

0 4.16667 1/SE(g)

The Campbell Collaboration www.campbellcollaboration.org

Other graphs – Galbraith/radial plots

• Points should be the same size for study (weight/precision is represented in the horizontal axis) • Include confidence intervals around the fixed effect summary effect line • Use different plotting symbols to distinguish subgroups, when appropriate

The Campbell Collaboration www.campbellcollaboration.org

29 Other graphs – L’abbé plots

• Plot of control group risk (x-axis) against treatment group risk (y-axis) • Commonly used to depict risks, but can also be plotted on log risk or log odds scales • Most commonly used for binary outcome data, but can be extended to depict means for continuous outcomes or ROC plot for diagnostic/screening test accuracy • Can also be used to contrast different effect size metrics (odds ratio, risk ratio, risk difference)

The Campbell Collaboration www.campbellcollaboration.org

Other graphs – L’abbé plots 1 .75 .5 Treatment groupTreatmentrate event .25 0

0 .25 .5 .75 1 Control group event rate

The Campbell Collaboration www.campbellcollaboration.org

30 Other graphs

• Density strips • Summary receiver-operator • Raindrop plots curve (SROC) graphs • Graphical display of study • Cross hairs ROC plot heterogeneity (GOSH) • Harvest plot • CUSUM chart • Baujat plots • Veritas plot

The Campbell Collaboration www.campbellcollaboration.org

Software resources

• CMA • OpenMeta • R • RevMan • SAS • SPSS • Stata

The Campbell Collaboration www.campbellcollaboration.org

31 Software resources

Forest Summary Cumulative Funnel Contour Trim and Galbraith l’Abbé plot forest forest plot plot funnel fill funnel plot plot plot plot plot CMA ü ü ü ü û ü û û OpenMeta ü ü ü ü û û û ü MIX ü ü ü ü ü ü ü ü R ü ü ü ü ü ü ü ü RevMan ü ü û ü ü û û û SAS ü ü ü ü û ü ü û SPSS ü û û ü û û û û Stata ü ü ü ü ü ü ü ü

Adapted from: Schild, A. H. E., & Voracek, M. (2013). Less is less: A systematic review of graph use in meta-analyses. Research Synthesis Methods, in press.

The Campbell Collaboration www.campbellcollaboration.org

Summary

• Graphs are an important part of any meta-analysis and can greatly facilitate interpretation, when used appropriately • Forest plots should (almost always) be included in a Campbell review • Funnel plots, bubble plots, Galbraith plots, L’Abbe plots, or other various plots may also be appropriate • Always follow standard graphing principles, and strive for accuracy, simplicity, clarity, aesthetic appeal, and good structure • Making good graphs can take time/effort – using software defaults is rarely sufficient!

The Campbell Collaboration www.campbellcollaboration.org

32 Recommended reading

Anzures-Cabrera, J., & Higgins, J. P. T. (2010). Graphical displays in meta-analysis: An overview with suggestions for practice. Research Synthesis Methods, 1, 66-80. Borman, G. D., & Grigg, J. A. (2009). Visual and narrative interpretation. Pp. 497-519 in H. Cooper, L. V. Hedges, & J. C. Valentine (Eds). The handbook of research synthesis and meta-analysis. New York: Russell Sage. Galbraith, R. F. (1988). A note on graphical presentation of estimated odds ratios from several clinical trials. Statistics in Medicine, 7, 889-894. Higgins, J. P. T. (2003). Considerations and recommendations for figures in Cochrane reviews: Graphs of statistical data. Cochrane Statistical Methods Group. Available online at: http://www.cochrane.org/training/cochrane-handbook#supplements Lane, P. W. et al. (2012). Graphics for meta-analysis. Pp. 295-308 in A. Krause & M. O’Connell (Eds.), A picture is worth a thousand tables: Graphics in life sciences. New York: Springer. Schild, A. H. E., & Voracek, M. (c. 2013). Less is less: A systematic review of graph use in meta- analyses. Research Synthesis Methods, in press. Schriger, D. L., et al. (2010). Forest plots in reports of systematic reviews: A cross-sectional study reviewing current practice. International Journal of Epidemiology, 39, 421-429.

The Campbell Collaboration www.campbellcollaboration.org

P.O. Box 7004 St. Olavs plass 0130 Oslo, Norway

E-mail: [email protected] http://www.campbellcollaboration.org

The Campbell Collaboration www.campbellcollaboration.org