Pussegoda et al. Systematic Reviews (2017) 6:117 DOI 10.1186/s13643-017-0507-6 RESEARCH Open Access Identifying approaches for assessing methodological and reporting quality of systematic reviews: a descriptive study Kusala Pussegoda1, Lucy Turner1, Chantelle Garritty1,2, Alain Mayhew1, Becky Skidmore1, Adrienne Stevens1,2, Isabelle Boutron3, Rafael Sarkis-Onofre4, Lise M. Bjerre5,6,7, Asbjørn Hróbjartsson8, Douglas G. Altman9 and David Moher10* Abstract Background: The methodological quality and completeness of reporting of the systematic reviews (SRs) is fundamental to optimal implementation of evidence-based health care and the reduction of research waste. Methods exist to appraise SRs yet little is known about how they are used in SRs or where there are potential gaps in research best-practice guidance materials. The aims of this study are to identify reports assessing the methodological quality (MQ) and/or reporting quality (RQ) of a cohort of SRs and to assess their number, general characteristics, and approaches to ‘quality’ assessment over time. Methods: The Cochrane Library, MEDLINE®, and EMBASE® were searched from January 1990 to October 16, 2014, for reports assessing MQ and/or RQ of SRs. Title, abstract, and full-text screening of all reports were conducted independently by two reviewers. Reports assessing the MQ and/or RQ of a cohort of ten or more SRs of interventions were included. All results are reported as frequencies and percentages of reports. Results: Of 20,765 unique records retrieved, 1189 of them were reviewed for full-text review, of which 76 reports were included. Eight previously published approaches to assessing MQ or reporting guidelines used as proxy to assess RQ were used in 80% (61/76) of identified reports. These included two reporting guidelines (PRISMA and QUOROM) and five quality assessment tools (AMSTAR, R-AMSTAR, OQAQ, Mulrow, Sacks) and GRADE criteria. The remaining 24% (18/76) of reports developed their own criteria. PRISMA, OQAQ, and AMSTAR were the most commonly used published tools to assess MQ or RQ. In conjunction with other approaches, published tools were used in 29% (22/76) of reports, with 36% (8/22) assessing adherence to both PRISMA and AMSTAR criteria and 26% (6/22) using QUOROM and OQAQ. Conclusions: The methods used to assess quality of SRs are diverse, and none has become universally accepted. The most commonly used quality assessment tools are AMSTAR, OQAQ, and PRISMA. As new tools and guidelines are developed to improve both the MQ and RQ of SRs, authors of methodological studies are encouraged to put thoughtful consideration into the use of appropriate tools to assess quality and reporting. Keywords: Reporting quality, Methodological quality, Systematic reviews, Guideline adherence * Correspondence: [email protected] 10Centre for Journalology, Canadian EQUATOR Centre, Clinical Epidemiology Program, Centre for Practice-Changing Research, Ottawa Hospital Research Institute, Ottawa, Ontario, Canada Full list of author information is available at the end of the article © The Author(s). 2017 Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated. Pussegoda et al. Systematic Reviews (2017) 6:117 Page 2 of 12 Background address several conceptual and methodological advances With the global annual expenditure of biomedical re- in the conduct and reporting of SRs. The development search estimated to be in excess of 100 billion USD [1], and adoption of SR MQ and RQ tools aim to assess, and it is no surprise that the extent of published literature is hopefully improve, the design, conduct, and reporting of growing each year, with PubMed® housing over 24 mil- SRs. Which tools are accepted and used by SR authors to lion citations, for example [2]. Researchers and decision assess MQ and completeness of reporting was unclear. makers have recognized that although there are hun- We set out to identify methodological evaluations asses- dreds of thousands of studies of healthcare interven- sing the MQ and/or RQ of SRs published from 1990 to tions, the quality of research and reporting is variable. 2014 in order to determine the approaches that were used. Evidence indicates that unless research is adequately de- signed and reported, the resources invested in research are not used effectively [1]. One estimate suggests that Methods at least 50% of published research studies were poorly Definitions and important concepts conducted making them difficult to interpret and use to We defined SRs and meta-analyses in line with that pro- inform best practice [1]. vided by the Cochrane Collaboration and the PRISMA ‘ ’ Systematic reviews (SRs) are considered the gold statement [12, 13]. We adopted the term overview,to standard for healthcare decision-making as they evaluate mean a summary of evidence from more than one SR, in- the quality and confidence of all of the available evidence cluding the combination of different populations, different addressing specific questions, such as the benefits and interventions, different outcomes (both favourable ones harms of specific health care interventions. When SR and adverse events), or different conditions [14, 15]. It is ‘ ’ conduct is optimal, that is, when best practices are synonymous with systematic review of systematic reviews, ‘ ’ ‘ ’ employed to minimize biases in the process of collecting, reviews of reviews,oran umbrella review .Wehavein- ‘ ’ appraising, and synthesizing the evidence, researchers cluded publications of methodological overviews, meaning can best understand whether or not they can be research that has assessed the MQ and/or RQ of a cohort ‘ ’ confident in the findings [3, 4]. Further, when SR report- of SRs and refer to these publications simply as reports . ing is optimal, the essential information is presented for practice guideline developers and other stakeholders, Methodological quality and completeness of reporting such as policy makers and clinicians to facilitate transla- It is necessary to make clear the distinction between MQ tion into guidance and improved patient care. and RQ. MQ addresses how well a SR was designed and Criteria for assessing the quality of primary research conducted (e.g. literature search, selection criteria, pooling emerged in the late 1980s with the rise of evidence- of data) [8]. RQ evaluates the description of the method- based medicine. This set the stage for guidelines asses- ology and findings [11]. Moreover, to distinguish from MQ, sing quality of SR conduct to be developed. Several sets the concept of risk of bias to assess primary studies is used of criteria had been developed early on including to refer to systematic flaws or limitations in the design, con- Mulrow [5] and Sacks criteria [6]. It was not until duct, or analysis of research that distort the findings [16]. Oxman and Guyatt developed the Overview Quality As- Examples are the Cochrane Risk of Bias tool for random- sessment Questionnaire (OQAQ) [7] in 1991, that a vali- ized controlled trials [17], ROBINS-I for non-randomized dated tool for assessing methodological quality (MQ) studies [18], QUADAS-2 [19] for diagnostic studies, and existed for SRs of intervention studies. More than a dec- ROBIS for SRs [16]). ade after OQAQ, A Measurement Tool to Assess Sys- tematic Reviews (AMSTAR) [8] was developed and validated in 2007 to address additional SR quality criteria Objectives including potential sources of bias that were not in- The objectives of this study are to identify reports asses- cluded in the OQAQ tool. In 2010, AMSTAR was re- sing the MQ and/or RQ of SRs and to assess their general vised (R-AMSTAR) to provide a quantitative scoring characteristics and approaches used. method to assess quality [9]. With criteria available for assessing SR conduct, it was apparent that SR authors address the standards for improving reporting quality Eligibility criteria (RQ) as well. In 1999, the Quality of Reporting of Meta- Inclusion criteria analyses (QUOROM) statement [10] was created to evalu- We included any methodological report published be- ate the completeness of reporting of meta-analysis of tween January 1990 and October 2014 whose stated pri- randomized trials. Subsequently, in 2009, QUOROM was mary intent was to assess the quality of methodology, updated as the Preferred Reporting Items of Systematic reporting, or other self-identified quality indicator(s) of a reviews and Meta-Analyses (PRISMA) statement [11] to cohort of SRs of interventions. Pussegoda et al. Systematic Reviews (2017) 6:117 Page 3 of 12 Exclusion criteria two reviewers in duplicate; a 10% random sample of re- We excluded reports of clinical interventions, whose pri- ports was assessed for accuracy. A pre-extraction meet- mary intent was not to look at methodological quality or ing was held for each extraction stage along with pilot reporting and rather to summarise SR evidence for use testing to ensure consistency across reviewers. The fol- in healthcare decision-making; reports assessing the lowing general characteristics were extracted: year of quality of SRs of diagnostic, screening, etiological, or publication; number of included SRs; specified medical prognostic studies only; and evaluations of SRs that in- area; databases searched; language restrictions; SR defin- clude study designs other than randomized controlled ition; reporting of availability of study protocol; and trials such as, narrative reviews, rapid reviews, network source of funding. The method of assessing MQ or RQ meta-analyses, and editorials. Reports in languages other of SRs was extracted. Additional items pertaining to the than English were excluded due to budget constraints evaluated reviews were extracted including the following: (Additional file 1) [20–31].