European Journal of Epidemiology (2019) 34:543–546 https://doi.org/10.1007/s10654-019-00505-6 COMMENTARY Umbrella reviews: what they are and why we need them Stefania Papatheodorou1,2 Received: 15 February 2019 / Accepted: 22 February 2019 / Published online: 9 March 2019 © Springer Nature B.V. 2019 Introduction Knowledge gap Evidence in clinical research and public health is hierar- The information that systematic reviews and meta-analy- chical. From animal studies to observational epidemiol- ses are conveying is directly associated with the quantity, ogy, randomized control trials and their synthesis, there quality and comparability of the available evidence from is a pipeline of diferent study designs providing diferent the primary studies. In observational and experimental nature and quality of information. Systematic reviews and research, diferences in interventions, metrics, outcomes, meta-analyses stand on top of this hierarchy most of the designs, participants and settings-collectively character- times and are key components of evidence-based medicine ized as sources of heterogeneity—as well as confounding, [1]. Their production and publication has increased expo- known and unknown biases, are often infated in a meta- nentially in the last decade across almost all disciplines with analysis [6]. Because of the increase in the sample size and approximately 20,000 records labeled as meta-analysis in statistical power, the pooled estimates can have spuriously 2018 compared to 3300 in 2008. One would expect that this tight confdence intervals. Sometimes it is doubtful whether growth would refect the increase in the number of primary statistical signifcance is real or a function of the additive or observational and experimental studies but this is not true cumulative impact of biases. In addition, meta-analyses or because all Pubmed-indexed items increased by 153% from randomized trials, usually to ensure comparability, examine 1991 to 2014 [2]. the association between one treatment option and one out- This abundance of published evidence synthesis cannot come. However, usually there are multiple treatments for the be considered only positive. It is not uncommon for meta- same condition and multiple clinically relevant outcomes analyses on the same research question to reach diferent and the meaningful question is what intervention is the best conclusions even when published within the same year. This for every outcome. Single risk factors are not specifc as well unavoidably leads to confusion and debate in clinicians and since many risk factors apply to several outcomes. Therefore public health policy makers on where to base their deci- a more comprehensive approach is needed. sions on. There are several preventive measures proposed An umbrella review systematically collects and evalu- in order to minimize bias in meta-analyses. A priori pub- ates information from multiple systematic reviews and meta- lication of protocols is highly encouraged and each ana- analyses on all clinical outcomes for which these have been lytical step and every subjective judgment call should be performed [7]. The number of papers tagged as “umbrella reported [3]. Finally there are tools to appraise the reporting review” in Pubmed from 2007 until today has increased [4] and quality [5] of the published systematic reviews and (Fig. 1). There is a very wide range of topics covered includ- meta-analyses. ing nutrition [8], psychiatry [9] and neurology [10], internal medicine [11] and Obstetrics and Gynecology [12]. There is also a variety of studies that can be included in the umbrella review depending on the research question: meta-analyses of observational studies examining risk factors for disease [13], interventions [14] or incorporating evidence from Mendelian * Stefania Papatheodorou Randomization studies [15]. [email protected] Not all published umbrella reviews follow a standard- 1 Department of Epidemiology, Harvard T.H. Chan School ized methodology despite the clear description of this of Public Health, Boston, MA, USA study design several years ago [7]. Such a broad synthesis 2 Cyprus University of Technology, Limassol, Cyprus and analysis of data from many systematic reviews and Vol.:(0123456789)1 3 544 S. Papatheodorou Number of papers tagged as "Umbrella Review" in Pubmed need to be evaluated in terms of their quality by using vali- 60 dated tools [5]. Measures of association and exposure‑outcome categories 40 Systematic reviews and meta-analyses use diferent meas- ures of association depending on the nature of the research 20 question, the design and the analytical approach. In large- Number of papers scale assessments, we can have relative and absolute meas- ures describing the same exposure-outcome association. The use of diferent measures of association should not 0 2007 2008 2009 2010 20112012201320142015201620172018 prohibit the researchers from synthesizing them in an umbrella review as long as they use the established meth- Fig. 1 Number of papers tagged as “umbrella reviews” in Pubmed ods for transformations [16]. That will allow a straight- forward presentation and interpretation of the evidence. In most cases, the defnition of the risk factors of any multiple meta-analyses is not simple and it requires both nature (clinical, environmental, biomarkers and others) subject-matter experts and experienced methodologists. is very heterogeneous. One possible option is to use the The basic steps on performing an umbrella review are pre- defnitions as presented in the primary studies without fur- sented below. ther categorization. This approach may reduce the risk of introducing newly defned factors not originally described in the literature. However, in some cases, it makes more sense to refer to categories of exposure for example in biomarkers. So instead of evaluating biomarkers one by Methods one, we can evaluate large categories of biomarkers such as hormones, diet, infammatory markers, IGF/insulin sys- The need for a new umbrella review has to be identifed a tem [17]. The advantage of this method is that we may be priori based on several factors: usually it is performed in able to collectively evaluate the state of the evidence in topics that are highly controversial or when the biases that broad categories of research, which may make more sense afect a certain research feld have not been systematically in clinical practice than evaluating biomarkers one by one. evaluated. Fields with many meta-analyses having incon- This is not always possible and it requires very thoughtful clusive evidence are clearly suitable for an umbrella review and justifed decisions from the analysts’ side. The same assessment, as it may shed light on the robustness of epide- reasoning applies in the defnition and categorization of miologic evidence by using a ranking approach. the outcomes. Systematic reviews can be summarized by using a descriptive approach [18]. The conclusions can be cat- egorized in the following categories: defnite association, Inclusion and exclusion criteria suggestive (possible) association, no association or incon- clusive association (insufcient evidence). As in all synthesis methods, the protocol must be registered in an open-access database such as PROSPERO (https :// www.crd.york.ac.uk/PROSP ERO/). The authors should Heterogeneity and other biases describe clearly the type of systematic reviews and meta- analyses included in their assessment (observational studies, Heterogeneity should be recorded and evaluated. In the randomized trials, mendelian randomization studies or all of presence of large between study heterogeneity, the results them). Certain criteria regarding the defnition of the expo- of the meta-analysis might not be applicable to any of the sure/intervention and the outcome need to be determined synthesized studies or in future studies. Publication bias just like in standard systematic reviews. The search strategy and small study efects are incorporated in the grading cri- and the databases searched for records must be fully reported teria used for ranking the epidemiologic evidence. Finally, for all databases for replication purposes. In addition, the the excess‐of‐statistical‐signifcance test is performed to included studies need to provide the data in enough detail evaluate whether there was a relative excess of formally so as to perform the statistical analysis. The included studies 1 3 545 Umbrella reviews: what they are and why we need them signifcant fndings in the published literature for any rea- Conclusions son as per Ioannidis and Trikalinos [19]. Umbrella reviews have the potential to provide the highest quality of evidence, if performed and interpreted properly. Grading criteria With almost 200 articles characterized as umbrella reviews in Pubmed and several others registered at PROSPERO data- Finally, the credibility of each proposed association is base, this is a fascinating feld with a great potential for large graded based on the following categories: contributions in the hierarchy of evidence. • Convincing evidence (Class I): Associations with a sta- tistical signifcance of P < 10 − 6, more than 1000 cases included (or more than 20,000 participants for continu- References ous outcomes), the largest component study reporting a signifcant result P < 0.05, a 95% prediction interval 1. Sackett DL, Rosenberg WM, Gray JA, Haynes RB, Richardson that excluded the null, absence of large heterogeneity WS. Evidence based medicine: what it is and what it isn’t. BMJ. P 1996;312(7023):71–2.