Evid Based Mental Health: first published as 10.1136/ebmental-2019-300129 on 5 March 2020. Downloaded from Statistics in practice Challenges in meta- analyses with observational studies Silvia Metelli,1,2 Anna Chaimani 1,3 1Université de Paris, Research ABSTRact mainly related to the high risk for within- study and Center of Epidemiology and Objective Meta- analyses of observational studies across- study biases, as well as to the presence of Statistics (CRESS- UMR1153), increased heterogeneity. INSERM, INRA, F-75004, Paris, are frequently published in the literature, but they are France generally considered suboptimal to those involving A major issue of observational evidence is that 2Assistance Publique - Hôpitaux randomised controlled trials (RCTs) only. This is due to it is known to have limited internal validity as it de Paris (APHP), Paris, France is subject to both bias and confounding. Overall, 3 the increased risk of biases that observational studies Cochrane France, Paris, France may entail as well as because of the high heterogeneity observational study designs are not the most appro- that might be present. In this article, we highlight aspects priate to assess the causal relationship between an Correspondence to Dr Anna Chaimani, Université of meta- analyses with observational studies that need intervention and an outcome as several characteris- de Paris, Research Center of more careful consideration in comparison to meta- tics might differ or might change over time between Epidemiology and Statistics analyses of RCTs. the different intervention groups. So, the inclusion Sorbonne Paris Cité (CRESS- Methods We present an overview of recommendations of observational studies in a meta- analysis might UMR1153), INSERM, INRA, Paris from the literature with respect to how the different introduce bias in the summary effect. To mitigate the 75004, France; anna. chaimani@ parisdescartes. fr steps of a meta- analysis involving observational studies risk of confounding and to make more comparable should be comprehensively conducted. We focus more on the different study groups, investigators usually Received 27 November 2019 issues arising at the step of the quantitative synthesis, in adjust the relative effects for several characteristics Revised 4 February 2020 terms of handling heterogeneity and biases. We briefly that may be related to the outcome and/or to the Accepted 5 February 2020 Published Online First describe some sophisticated synthesis methods, which intervention. Propensity scores (ie, the probability 5 March 2020 may allow for more flexible modelling approaches than of treatment assignment conditional on observed common meta- analysis models. We illustrate the issues baseline characteristics) are now also being used encountered in the presence of observational studies frequently in the analysis of observational studies using an example from mental health, which assesses as they likely allow reduction of confounding and the risk of myocardial infarction in antipsychotic drug selection bias. Despite the fact that these methods users. have the potential to produce less biassed results, Results The increased heterogeneity observed among at the meta-anal ysis level they increase the meth- studies challenges the interpretation of the diamond, odological heterogeneity as often different studies while the inclusion of short exposure studies may lead use different analysis methods or different adjust- to an exaggerated risk for myocardial infarction in this ment factors and the comparability of their results population. is questionable. Apart from methodological hetero- Conclusions In the presence of observational geneity, clinical heterogeneity is also expected to be study designs, prior to synthesis, investigators should much higher than in meta-analyses of RCTs since http://ebmh.bmj.com/ carefully consider whether all studies at hand are observational studies are based on less stringent able to answer the same clinical question. The inclusion criteria. potential for a quantitative synthesis should be guided An additional problem in the synthesis of through examination of the amount of clinical and observational studies is that it is always chal- methodological heterogeneity and assessment of lenging or even impossible to assess the risk of possible biases. bias both within and across studies. The latter is because preregistration and protocol prepara- on October 1, 2021 by guest. Protected copyright. tion are not mandatory for observational studies, and as a result, unpublished studies or partly INTRODUCTION unpublished results cannot be identified. This Systematic reviews and meta-anal yses help to estab- leads to an increased risk of publication bias and lish evidence- based clinical practice and resolve other reporting biases such as selective outcome contradictory research outcomes while supporting reporting. With respect to within- study bias, in research planning and prioritisation.1 Therefore, contrast to RCTs, the lack of widely agreed quality meta- analysis is being increasingly used in most criteria and the absence of sufficient empirical medical fields with the aim to reach an overall evidence to support the focus on particular study features render the assessment of the risk of bias © Author(s) (or their understanding of clinical outcomes and their sources employer(s)) 2020. No of variation. Although synthesis of randomised for observational studies and their meta- analysis commercial re- use. See rights controlled trials (RCTs) is generally considered rather challenging. To date, more than 80 tools and permissions. Published as the highest level of clinical evidence, there are have been proposed to assess the credibility of by BMJ. several concerns regarding the potential of meta- observational studies, but most of them have not To cite: Metelli S, analysis of observational studies to provide reli- been used in practice. Recently, a draft Cochrane Chaimani A. Evid Based Ment able results.2 The reasons making researchers being risk of bias tool for non-randomised studies was Health 2020;23:83–87. sceptical with synthesising observational studies are also developed that considers each observational Metelli S, Chaimani A. Evid Based Ment Health 2020;23:83–87. doi:10.1136/ebmental-2019-300129 83 Evid Based Mental Health: first published as 10.1136/ebmental-2019-300129 on 5 March 2020. Downloaded from Statistics in practice study as an attempt to mimic a hypothetical pragmatic Synthesising data and controlling for bias randomised trial.3 Nevertheless, this tool has not been final- Observational studies usually have larger sample sizes than RCTs ised yet. and might yield highly precise results.2 Given all the aforemen- Despite all the above issues, observational data not only tioned issues of observational evidence (ie, bias and confounding), offer a valuable source of supplementary information to RCTs this phenomenon might lead to spurious inferences because but also, for some clinical questions, provide the most reliable usually the more precise the summary effects, the stronger the data (eg, safety of interventions and long-term outcomes) or conclusions of the investigators. Further, when observational even the only source of available evidence (eg, effectiveness of and randomised studies are synthesised using typical methods transplantation). In addition, results from observational studies (eg, classical fixed or random effects meta-analysis), the weight might be more directly applicable to the general populations of observational studies would be larger than that of the RCTs, as they are designed under a more real- life setting than RCTs, although the latter usually give more reliable results. Thus, it is which usually involve very restricted populations treated under of great importance to consider very carefully the setting of each 4 highly standardised care. Therefore, meta- analysis of observa- study before proceeding with the synthesis of the results, and tional data alone or in combination with RCTs (when possible) whether it is appropriate to answer the research question of the 5 is often desirable. However, a methodological systematic review meta- analysis. of meta- analyses of observational studies in the field of psychi- At the stage of data synthesis, the main issues in the pres- atry found several deficiencies in terms of assessing study quality, ence of observational studies are (1) how to accommodate the publication bias and risk for confounding, while the majority of possibly large heterogeneity that may be present especially 6 the meta- analyses found significant heterogeneity. when different types of observational studies, or also RCTs, In this article, we review methods and recommendations in are combined in the same analysis and (2) how to account for the literature for the different steps of a meta- analysis involving different biases. Overall, using a fixed effect meta- analysis does observational studies. Using an example from mental health, we not seem a reasonable approach considering that observational present and discuss the issues that frequently arise due to the studies generally have very variable populations, which are nature of these data. followed under different conditions. In case of multiple obser- vational designs or combination with RCTs, these discrepancies would likely be magnified. The random effects model accounts METHODS better for this apparent heterogeneity. Subgroup analysis or meta- Searching for relevant studies regression by study design and by type