CONCLI-01195; No of Pages 9 Contemporary Clinical Trials xxx (2015) xxx–xxx Contents lists available at ScienceDirect Contemporary Clinical Trials journal homepage: www.elsevier.com/locate/conclintrial Advances in the meta-analysis of heterogeneous clinical trials I: The inverse variance heterogeneity model Suhail A.R. Doi a,⁎, Jan J. Barendregt b,c, Shahjahan Khan d, Lukman Thalib e, Gail M. Williams c a Research School of Population Health, Australian National University, Canberra, Australia b Epigear International, Sunrise Beach, Australia c School of Population Health, University of Queensland, Brisbane, Australia d School of Agricultural, Computational and Environmental Sciences, University of Southern Queensland, Toowoomba, Australia e Department of Community Medicine, Kuwait University, Kuwait article info abstract Article history: This article examines an improved alternative to the random effects (RE) model for meta-analysis of heterogeneous Received 5 February 2015 studies. It is shown that the known issues of underestimation of the statistical error and spuriously overconfident Received in revised form 10 May 2015 estimates with the RE model can be resolved by the use of an estimator under the fixed effect model assumption Accepted 15 May 2015 with a quasi-likelihood based variance structure — the IVhet model. Extensive simulations confirm that this estima- Available online xxxx tor retains a correct coverage probability and a lower observed variance than the RE model estimator, regardless of heterogeneity. When the proposed IVhet method is applied to the controversial meta-analysis of intravenous mag- Keywords: – Fixed effect nesium for the prevention of mortality after myocardial infarction, the pooled OR is 1.01 (95% CI 0.71 1.46) which Heterogeneity not only favors the larger studies but also indicates more uncertainty around the point estimate. In comparison, Meta-analysis under the RE model the pooled OR is 0.71 (95% CI 0.57–0.89) which, given the simulation results, reflects underes- Quasi-likelihood timation of the statistical error. Given the compelling evidence generated, we recommend that the IVhet model re- Random effects place both the FE and RE models. To facilitate this, it has been implemented into free meta-analysis software called MetaXL which can be downloaded from www.epigear.com. © 2015 Elsevier Inc. All rights reserved. 1. Introduction from which they depart under the influence of the random error only [4]. As such, only within-study variation is considered to be present. In the era of evidence based medicine, meta-analyses of well- In practice, this model ensures that the larger studies (with the lowest designed and executed randomized controlled trials have the potential probability of random error) have the greatest influence on the pooled to provide high levels of evidence to support therapeutic interventions estimate. The drawback however is that this model also demonstrates in all areas of clinical practice. Despite the potential of the outcome of increasing overdispersion as heterogeneity increases. Overdispersion such trials to guide decision making, they may sometimes fail to pro- here refers to an estimator that has a greater observed variance (true duce credible conclusive results or may disagree if there were multiple variance often assessed through simulation) than that theoretically ex- independent trials that investigate the same clinical question. In this sit- pected which is based on the statistical model (used in the confidence uation, a meta-analysis of the trial results has the potential to combine interval computation). conceptually similar and independent studies with the purpose of de- In an attempt to tackle the issue of overdispersion, the RE approach riving more reliable statistical conclusions (based on a much larger sam- was suggested which attempts to create a more fully specified model ple data) than any of the individual studies [1,2]. Today, clinical decision [6]. This model makes the additional assumption that the true treatment making relies heavily on this methodology as is evident by the work of effects in the individual studies are different from each other and these the Cochrane collaboration and the high volume of publications for differences follow a normal distribution with a common variance. The meta-analyses outside the collaboration [3]. assumption of normally distributed random effects is not justified [7] Meta-analyses, customarily, are performed using either the fixed ef- because the underlying effects included in the meta-analysis do not fect (FE) or the random effects (RE) models [4,5]. The FE model of meta- constitute a random sample from the population. This model neverthe- analysis is underpinned by the assumption that one identical true treat- less ignores the need for randomization in statistical inference [8] and ment effect is common to every study included in the meta-analysis, the variance of these underlying effects is usually approximated by a moment-based estimator [9]. The application of this common variance to the model has the unintended effect of redistributing the study ⁎ Corresponding author at: Chronic Disease Epidemiology Division, Research School of Population Health, Australian National University, Mills Road, Acton ACT 2601, Australia. weights in only one direction: from larger to smaller studies [10].Thus E-mail address: [email protected] (S.A.R. Doi). the studies with the lowest probability of random error are penalized http://dx.doi.org/10.1016/j.cct.2015.05.009 1551-7144/© 2015 Elsevier Inc. All rights reserved. Please cite this article as: S.A.R. Doi, et al., Advances in the meta-analysis of heterogeneous clinical trials I: The inverse variance heterogeneity model, Contemp Clin Trials (2015), http://dx.doi.org/10.1016/j.cct.2015.05.009 2 S.A.R. Doi et al. / Contemporary Clinical Trials xxx (2015) xxx–xxx fl ρ ^ and do not in uence the combined estimates as strongly. The inclusion the extent of bias in an estimator. wδ will only be greater than zero if of this estimated common between-studies variance also seems to be both the weights and the study effects are correlated and of course, the mechanism that attempts to address overdispersion with increasing there will be such a correlation if the weights are derived from the data. heterogeneity, yielding wider confidence intervals and lesser statistical For the inverse variance weighted FE model, the weighted estimator, significance than would be attained through the conventional fixed ef- fect model. Yet, given the faulty assumptions, it does not work as ex- Xk ^ ^ pected and as heterogeneity increases, the coverage of the confidence θ FE ¼ w jδ j; interval drops well below the nominal level [7,11].Thoughcorrections j¼1 have been suggested [12,13] they have not been easy to implement and this estimator substantially underestimates the statistical error has weights that sum to 1 given by: and remains potentially overconfident in its conclusions [13,14]. , A careful look at the inputs to these models demonstrates that both Xk ¼ 1 1 ; ð Þ use inverse variance weighting to decrease observed variance of the es- w j υ υ 2 timator. However, the approach taken with the RE estimator disadvan- j j¼1 j tages it because as heterogeneity increases, the inverse variance weights are moved towards equality thus increasing estimator variance. where the sampling error variance of the jth study is υj. Given the previ- This also leads to a failure to specify the theoretical variance correctly so ous discussion, this inverse variance estimator is likely to be biased (un- that it now falls short of the observed variance and nominal coverage is less there are equal variances across studies). However, despite the not achieved. expected bias, the FE estimator does improve over the arithmetic While alternative frequentist RE models that attempt to improve on mean estimator because the weights don't just increase the bias, but the conventional theoretical model in various ways have been described (by doing so) they also make the variance of the estimator much smaller in the literature [11], they all continue to be based on the assumption of [20] and trade off this bias through reduction in the mean squared error normally distributed random effects which, as mentioned above, leads (MSE). The point of having such increased precision despite, on average, to several problems. There is therefore the need for a better method the estimator being biased is that while the estimator is biased “on av- and this paper argues that the random effects model should be replaced erage”, it will still be “more correct” most of the time if the MSE of the by a distributional assumption free model. Such a model has been pro- biased estimator is smaller than that (MSE) of the unbiased estimator. posed by us as a variant of the quality effects model that sets quality Since, as researchers, we usually have only one set of sample data for to equal (called the IVhet model) [15]. This paper reviews the model's meta-analysis, the bias in the estimator is overshadowed by the de- theoretical construct and presents an evaluation of its performance creased MSE. using standard performance measures [16]. This works well under the model assumption that studies suffer only from sampling errors. When heterogeneity due to systematic errors is 2. Difference between empirically weighted means and the also present, this model suffers from overdispersion. To resolve this, arithmetic mean the RE weighted estimator was put forward by DerSimonian and Laird in 1986 [9] where the pooled effect is computed in the same way: Consider a collection of k independent studies, the jth of which has ^ estimated effect size δ which varies from its true effect size, δ through Xk j j ^ Ã^ θRE ¼ w δ j; random error. Also consider that the true effects, δjs, also vary from an j j¼1 underlying common effect, θ, through bias. There is the possibility of some diversity of true effects (which remain similar) across studies (in which case θ would simply be the mean of the true (unbiased) ef- but where the weights that sum to 1 are now given by: fects).