Graphical Representation of Meta-analysis Findings Emily E. Tanner-Smith Associate Editor, Campbell Methods Coordinating Group Research Assistant Professor, Vanderbilt University Campbell Collaboration Colloquium Chicago, IL May 22nd, 2013 The Campbell Collaboration www.campbellcollaboration.org Outline • Introduction • Forest plots • Funnel plots • Bubble plots • Other graphs • Software resources • Summary The Campbell Collaboration www.campbellcollaboration.org 1 Introduction • Graphs are an essential tool for conveying the results of a meta-analysis to readers • But if poorly constructed, graphs can be misleading and/or confuse readers • Graphs should strive for accuracy, simplicity, clarity, and aesthetics • This workshop will provide an overview of expectations and guidelines for graphical displays of meta-analysis results in Campbell Collaboration reviews The Campbell Collaboration www.campbellcollaboration.org Introduction: Basic Graphing Principles • Descriptive titles and/or captions • Use of legends (when appropriate) • Representative range of scale • Properly labeled axes • Inclusion of reference points on axes • Graphs should reflect the statistical precision of results • Explicit mention of any excluded data • Data in graphs should generally be available elsewhere in the review (except in very large reviews) • Aesthetics (line thickness, symbol size, symbol types, parsimony) The Campbell Collaboration www.campbellcollaboration.org 2 FOREST PLOTS The Campbell Collaboration www.campbellcollaboration.org Forest plots • The “workhorse” graph in meta-analysis • Display effect size estimates and confidence intervals for each study included in the meta-analysis • Effect size estimates typically shown with blocks proportionate to the weight assigned to a given study – Functions to draw the eye toward studies with larger sample size/larger weights, and away from smaller studies with wider confidence intervals The Campbell Collaboration www.campbellcollaboration.org 3 Forest plots • Estimated mean effect size with confidence interval shown at the bottom, typically with a diamond • In random effects meta-analyses, prediction intervals can be used to display dispersion in the estimated effect • Studies should be ordered in a meaningful way – Effect size magnitude – Study weight (precision) – Chronological order – Other meaningful study characteristic The Campbell Collaboration www.campbellcollaboration.org Forest plots Source: Regehr, C., Alaggia, R., Dennis, J., Pitts, A., & Saini, M. (2013). Interventions to reduce distress in adult victims of sexual violence and rape. Campbell Systematic Reviews, 3. doi:10.4073/csr.2013.3 The Campbell Collaboration www.campbellcollaboration.org 4 Forest plots Source: Regehr, C., Alaggia, R., Dennis, J., Pitts, A., & Saini, M. (2013). Interventions to reduce distress in adult victims of sexual violence and rape. Campbell Systematic Reviews, 3. doi:10.4073/csr.2013.3 The Campbell Collaboration www.campbellcollaboration.org Forest plots Source: Regehr, C., Alaggia, R., Dennis, J., Pitts, A., & Saini, M. (2013). Interventions to reduce distress in adult victims of sexual violence and rape. Campbell Systematic Reviews, 3. doi:10.4073/csr.2013.3 The Campbell Collaboration www.campbellcollaboration.org 5 Forest plots Source: Regehr, C., Alaggia, R., Dennis, J., Pitts, A., & Saini, M. (2013). Interventions to reduce distress in adult victims of sexual violence and rape. Campbell Systematic Reviews, 3. doi:10.4073/csr.2013.3 The Campbell Collaboration www.campbellcollaboration.org Forest plots Source: Regehr, C., Alaggia, R., Dennis, J., Pitts, A., & Saini, M. (2013). Interventions to reduce distress in adult victims of sexual violence and rape. Campbell Systematic Reviews, 3. doi:10.4073/csr.2013.3 The Campbell Collaboration www.campbellcollaboration.org 6 Forest plots Source: Regehr, C., Alaggia, R., Dennis, J., Pitts, A., & Saini, M. (2013). Interventions to reduce distress in adult victims of sexual violence and rape. Campbell Systematic Reviews, 3. doi:10.4073/csr.2013.3 The Campbell Collaboration www.campbellcollaboration.org Forest plots Source: Regehr, C., Alaggia, R., Dennis, J., Pitts, A., & Saini, M. (2013). Interventions to reduce distress in adult victims of sexual violence and rape. Campbell Systematic Reviews, 3. doi:10.4073/csr.2013.3 The Campbell Collaboration www.campbellcollaboration.org 7 Forest plots Source: Maynard, B. R., McCrea, K. T., Pigott, T. D., & Kelly, M. S. (2012). Indicated truancy interventions: Effects on school attendance among chronic truant students. Campbell Systematic Reviews, 10. doi:10.4073/csr. 2012.10 The Campbell Collaboration www.campbellcollaboration.org Forest plots Source: Maynard, B. R., McCrea, K. T., Pigott, T. D., & Kelly, M. S. (2012). Indicated truancy interventions: Effects on school attendance among chronic truant students. Campbell Systematic Reviews, 10. doi:10.4073/csr. 2012.10 The Campbell Collaboration www.campbellcollaboration.org 8 Forest plots with subgroups • Display effect size estimates and confidence intervals for each study, split by some grouping variable • Useful for depicting results from subgroup or moderator analyses • May include the overall summary effect across groups, if appropriate • Results from statistical tests of moderation (e.g., QB or b from a meta-regression) should be summarized on the graph or in footnotes, when appropriate The Campbell Collaboration www.campbellcollaboration.org Study Hedges' g (95% CI) Single Session Intervention Jones, 2012 0.11 (-0.09, 0.31) Wilson, 2008 0.22 (-0.12, 0.56) Smith, 2011 0.34 (0.06, 0.62) Forest Walters, 2000 0.45 (0.21, 0.69) Milton, 1999 0.48 (0.28, 0.68) Subtotal 0.32 (0.17, 0.48) plots with . (-0.16, 0.81) subgroups Multi-Session Intervention Chang, 1997 0.44 (0.20, 0.68) Liu, 1992 0.49 (0.21, 0.77) Mapleson, 2001 0.65 (0.41, 0.89) Steiner, 2005 0.71 (0.40, 1.02) Lancaster, 2009 0.74 (0.53, 0.95) Subtotal 0.61 (0.49, 0.73) . (0.36, 0.86) -1.02 0 1.02 Favors Control Favors Treatment Source: Fictional data The Campbell Collaboration www.campbellcollaboration.org 9 Study Hedges' g (95% CI) Single Session Intervention Jones, 2012 0.11 (-0.09, 0.31) Wilson, 2008 0.22 (-0.12, 0.56) Smith, 2011 0.34 (0.06, 0.62) Forest Walters, 2000 0.45 (0.21, 0.69) Milton, 1999 0.48 (0.28, 0.68) Subtotal 0.32 (0.17, 0.48) plots with . (-0.16, 0.81) subgroups Multi-Session Intervention Chang, 1997 0.44 (0.20, 0.68) Liu, 1992 0.49 (0.21, 0.77) Mapleson, 2001 0.65 (0.41, 0.89) Steiner, 2005 0.71 (0.40, 1.02) Lancaster, 2009 0.74 (0.53, 0.95) Subtotal 0.61 (0.49, 0.73) . (0.36, 0.86) -1.02 0 1.02 Favors Control Favors Treatment Source: Fictional data The Campbell Collaboration www.campbellcollaboration.org Study Hedges' g (95% CI) Single Session Intervention Jones, 2012 0.11 (-0.09, 0.31) Wilson, 2008 0.22 (-0.12, 0.56) Smith, 2011 0.34 (0.06, 0.62) Forest Walters, 2000 0.45 (0.21, 0.69) Milton, 1999 0.48 (0.28, 0.68) Subtotal 0.32 (0.17, 0.48) plots with . (-0.16, 0.81) subgroups Multi-Session Intervention Chang, 1997 0.44 (0.20, 0.68) Liu, 1992 0.49 (0.21, 0.77) Mapleson, 2001 0.65 (0.41, 0.89) Steiner, 2005 0.71 (0.40, 1.02) Lancaster, 2009 0.74 (0.53, 0.95) Subtotal 0.61 (0.49, 0.73) . (0.36, 0.86) Overall 0.46 (0.33, 0.60) . (0.04, 0.89) -1.02 0 1.02 Favors Control Favors Treatment Note: Significant difference in mean effect sizes between groups (b = .28, se = .10, 95% CI [.05, .52]). Source: Fictional data The Campbell Collaboration www.campbellcollaboration.org 10 Study Hedges' g (95% CI) Single Session Intervention Jones, 2012 0.11 (-0.09, 0.31) Wilson, 2008 0.22 (-0.12, 0.56) Smith, 2011 0.34 (0.06, 0.62) Forest Walters, 2000 0.45 (0.21, 0.69) Milton, 1999 0.48 (0.28, 0.68) Subtotal 0.32 (0.17, 0.48) plots with . (-0.16, 0.81) subgroups Multi-Session Intervention Chang, 1997 0.44 (0.20, 0.68) Liu, 1992 0.49 (0.21, 0.77) Mapleson, 2001 0.65 (0.41, 0.89) Steiner, 2005 0.71 (0.40, 1.02) Lancaster, 2009 0.74 (0.53, 0.95) Subtotal 0.61 (0.49, 0.73) . (0.36, 0.86) Overall 0.46 (0.33, 0.60) . (0.04, 0.89) -1.02 0 1.02 Favors Control Favors Treatment Note: Significant difference in mean effect sizes between groups (b = .28, se = .10, 95% CI [.05, .52]). Source: Fictional data The Campbell Collaboration www.campbellcollaboration.org Study Hedges' g (95% CI) Single Session Intervention Jones, 2012 0.11 (-0.09, 0.31) Wilson, 2008 0.22 (-0.12, 0.56) Smith, 2011 0.34 (0.06, 0.62) Forest Walters, 2000 0.45 (0.21, 0.69) Milton, 1999 0.48 (0.28, 0.68) Subtotal 0.32 (0.17, 0.48) plots with . (-0.16, 0.81) subgroups Multi-Session Intervention Chang, 1997 0.44 (0.20, 0.68) Liu, 1992 0.49 (0.21, 0.77) Mapleson, 2001 0.65 (0.41, 0.89) Steiner, 2005 0.71 (0.40, 1.02) Lancaster, 2009 0.74 (0.53, 0.95) Subtotal 0.61 (0.49, 0.73) . (0.36, 0.86) Overall 0.46 (0.33, 0.60) . (0.04, 0.89) -1.02 0 1.02 Favors Control Favors Treatment Note: Significant difference in mean effect sizes between groups (b = .28, se = .10, 95% CI [.05, .52]). Source: Fictional data The Campbell Collaboration www.campbellcollaboration.org 11 Summary forest plots • Display summary (mean) effect sizes and confidence intervals for different groups of studies • Does not include effect size estimates from individual studies • Useful for very large reviews where traditional forest plots may not be feasible, but effects can be categorized into meaningful groups (e.g., across intervention, study, participant types) • May include the overall summary effect across groups, if appropriate • Results from statistical tests of moderation (e.g., QB