A Guide to Understanding Meta-Analysis
Total Page:16
File Type:pdf, Size:1020Kb
[ clinical commentary ] HEIDI ISRAEL, FNP, PhD1 • RANDY R. RICHTER, PT, PhD2 A Guide to Understanding Meta-Analysis eta-analysis is a popular and frequently used statistical able one to combine data and summarize technique used to combine data from several studies and the findings of several clinical trials that reexamine the effectiveness of treatment interventions. As evaluate the effectiveness or efficacy of a similar treatment approach on similar the number of articles using meta-analysis increases, under- M group of patients. This technique can standing of the benefits and drawbacks of the technique is essential. prove especially useful when there are several similar clinical trials with or with- Well-conducted systematic reviews of high level of evidence, and enhanced clin- out consistent outcomes, or when there randomized controlled trials are regarded ical interpretation of treatment effects, are smaller to medium-sized trials with as representing a high level of evidence.28 interpreting a meta-analysis is an impor- inconclusive results. Practicing in an evidence-based manner tant skill for physical therapists. The pur- By combining the results from 2 or is a recognized goal for the profession.3 pose of this commentary is to expand on more studies, a meta-analysis can in- Systematic reviews are used to answer existing articles describing meta-analysis crease statistical power18 and provide questions7,10,40 about the evidence sup- interpretation,6,13,14,42,61 discuss differences a single numerical value of the overall porting or refuting the effectiveness or in the results of a meta-analysis based on treatment effect. The meta-analysis result efficacy of an intervention. When certain the treatment questions, explore special may show either a benefit or lack of ben- conditions are met, a systematic review cases in the use of meta-analysis, and efit of a treatment approach that will be may be extended to include a meta- provide physical therapists guidance in indicated by the effect size, which is the analysis, a statistical procedure used to interpreting a meta-analysis. term used to describe the treatment ef- numerically summarize the included fect of an intervention. Treatment effect is studies’ treatment effect.56 A meta-anal- WHY META-ANALYSIS the gain (or loss) seen in the experimental ysis provides a single, overall measure of group relative to the control group. The the treatment effect, enhancing the clini- number of reasons exist for overall positive or negative change may cal interpretation of findings across sev- considering the use of meta-anal- be hard to discern from individual stud- eral studies. Because of its increasing use, A ysis techniques. Meta-analyses en- ies. For example, Clare et al15 used a meta- analysis to examine the treatment effect of McKenzie therapy for spinal pain. TTSYNOPSIS: With the focus on evidence- meta-analysis, and strengths and weaknesses of based practice in healthcare, a well-conducted meta-analysis. Common components like forest Three studies supported the use of McK- systematic review that includes a meta-analysis plot interpretation, software that may be used, enzie therapy for short-term pain. Two of where indicated represents a high level of evidence special cases for meta-analysis, such as subgroup the 3 studies reported a small but similar for treatment effectiveness. The purpose of this analysis, individual patient data, and meta-regres- reduction in pain, which was statistically commentary is to assist clinicians in understand- sion, and a discussion of criticisms, are included. significant for only 1 of the 2 studies. The ing meta-analysis as a statistical tool using both J Orthop Sports Phys Ther 2011;41(7):496-504. third study reported a reduction of pain published articles and explanations of components doi:10.2519/jospt.2011.3333 that was twice the magnitude of the other of the technique. We describe what meta-analysis studies. The results of the meta-analysis is, what heterogeneity is, and how it affects meta- TTKEY WORDS: forest plot, literature review, analysis, effect size, the modeling techniques of statistical analysis, systematic review indicated an overall treatment effect that was statistically significant and closer 1Assistant Professor, Saint Louis University, Department of Orthopaedic Surgery, St Louis, MO. 2Associate Professor, Saint Louis University, Program in Physical Therapy, St Louis, MO. Address correspondence to Dr Heidi Israel, Saint Louis University, Department of Orthopaedic Surgery, 3635 Vista FDT7N, St Louis, MO 63104. E-mail: [email protected] 496 | july 2011 | volume 41 | number 7 | journal of orthopaedic & sports physical therapy 41-07 Israel.indd 496 6/15/2011 3:14:46 PM in magnitude to the 2 studies reporting a small reduction in pain. This example illustrates the potential value of meta- analysis when direction of the treatment effect is the same across all studies but the magnitude and statistical significance of the treatment effect varies. A meta-anal- ysis can also be used to show changes in the treatment effect that occur over time. For example, Zhang et al,60 in an update of the management of hip and knee os- teoarthritis (2006 to 2009), determined that the treatment effect sizes for exercise and acupuncture did not change at mul- tiple time points, while the treatment ef- fect sizes for weight reduction eventually reached statistical significance at later time points, and the treatment effect size for electromagnetic therapy was no lon- ger significant at later time points. These type of results obtained from meta-anal- ysis can be used to make better informed treatment decisions. To interpret a meta-analysis, the read- er needs to understand several concepts, including effect size, heterogeneity, the inal units, the interpretation is clearer. size estimates.36 Finally, variables such as model used to conduct the meta-analy- When the effect sizes are in standardized gender, age differences, or differences in sis, and the forest plot, a graphical rep- units, the interpretation is more difficult the intervention provided, such as dose, resentation of the meta-analysis. These and published guidelines for interpret- can influence the magnitude and direc- concepts are discussed below and sum- ing effect sizes may be used.17 Whether tion of the effect size. marized in TABLE 1. standardized or not, the overall effect size Use of the confidence interval can lend derived from the meta-analysis is calcu- insight into the precision of the treatment EFFECT SIZE lated by combining the effect sizes of the included studies. There are several types tudies included in a meta-anal- of effect sizes. For dichotomous data, ysis must have common outcome such as improved or not improved, odds Sstatistics that allow their results to ratios or relative risks are used for effect be combined.31 Effect sizes, which re- sizes. Other types of effect sizes are often flect the magnitude and direction of the reported in meta-analysis, and these are treatment effect for each study, serve this described in TABLE 2. purpose. When all the studies to be in- Because several factors, such as sam- cluded in a meta-analysis have the same ple size, variance, and reliability of the outcome measure, an effect size in the outcome measures, can influence the original units may be calculated. For ex- magnitude and direction of the effect ample, if all studies in the meta-analysis size, the estimates of the effect sizes will measure a continuous outcome, such as vary among studies. In addition, the ef- range of motion, the mean difference fect size of the individual studies may be can be used as the effect size. Standard- somewhat imprecise and, therefore, lead ization of the effect size is needed when to an unstable finding when multiple treatments are not measured in the same small studies are utilized. Weighting of units. Standardization makes data unit- the standard error based on sample size less. When the effect sizes are in the orig- allows for the best precision of the effect journal of orthopaedic & sports physical therapy | volume 41 | number 7 | july 2011 | 497 498 | july However, despite having statistical tests for statistical heterogeneity, there are no accepted guidelines for when a meta- analysis should not be completed due to statistical heterogeneity, and it is left to the author’s discretion to determine if a meta-analysis is appropriate. Clinical heterogeneity refers to dif- ferences in study methods that affect the ability to compare and/or combine data from different studies. Examples of dif- ferences in study methods that may lead to clinical heterogeneity include differ- ences in participant demographics, such as risk or severity of disease, the settings in which the research was conducted, the journal of orthopaedic & sports physical therapy | volume 41 | number 7 | july 2011 | 499 [ clinical commentary ] assumes a distribution of treatment ef- study is accounted for in the calcula- fects, answers the question “What is the tion.22,26 A calculation method under the average treatment effect?” The random- effects model assumes a distribution of the treatment effect for some popula- tions, meaning that the treatment effect falls along a range of values, not a single value, as in the fixed-effects model. -Be cause of this distribution, the effect size may be positive for some populations but may be negative or harmful for others.18,29 Studies included in the meta-analysis using a random-effects model are as- sumed to represent a random sample of a population of studies. The results of each study included in the meta-analysis represent a study-specific effect size that varies around a mean population ef- fect size.18,26 In other words, the results of each study in the meta-analysis are assumed to represent a unique effect. Because of this assumption, larger stud- ies are given proportionally less weight, while smaller studies are given propor- tionally more weight.11 In the random- effects model, the unique effect of each 500 | july 2011 | volume 41 | number 7 | journal of orthopaedic & sports physical therapy sizes.