Characteristics, Quality and Volume of the First 5 BMJ EBM: First Published As 10.1136/Bmjebm-2021-111710 on 3 June 2021
Total Page:16
File Type:pdf, Size:1020Kb
Evidence synthesis Characteristics, quality and volume of the first 5 BMJ EBM: first published as 10.1136/bmjebm-2021-111710 on 3 June 2021. Downloaded from months of the COVID-19 evidence synthesis infodemic: a meta- research study Rebecca Abbott ,1 Alison Bethel,1 Morwenna Rogers,1 Rebecca Whear,1 Noreen Orr,1 Liz Shaw,2 Ken Stein,1 1 Jo Thompson Coon 10.1136/bmjebm-2021-111710 Abstract Summary box Objective The academic and scientific community has reacted at pace to gather evidence to help ► Additional supplemental What is already known about this and inform about COVID-19. Concerns have been material is published online subject? only. To view, please visit raised about the quality of this evidence. The aim ► Poorly conducted systematic reviews the journal online (http:// of this review was to map the nature, scope and can lead to inaccurate representations dx. doi. org/ 10. 1136/ quality of evidence syntheses on COVID-19 and of the evidence, inaccurate estimates bmjebm-2021- 111710). to explore the relationship between review quality of treatment effectiveness, misleading and the extent of researcher, policy and media 1NIHR ARC South West conclusions and reduced applicability. interest. Peninsula, University of Exeter Medical School, University of Design and setting A meta-research: systematic What are the new findings? Exeter, Exeter, UK review of reviews. ► Most COVID-19 evidence syntheses 2University of Exeter Medical Information sources PubMed, Epistemonikos described as systematic or rapid School, University of Exeter, COVID-19 evidence, the Cochrane Library of review were of low quality and missed Exeter, UK Systematic Reviews, the Cochrane COVID-19 out cornerstones of best practice. Study Register, EMBASE, CINAHL, Web of Science Less than a half reported critically Core Collection and the WHO COVID-19 database, appraising their included studies, Correspondence to: searched between 10 June 2020 and 15 June 2020. and a third had no reproducible Dr Rebecca Abbott, NIHR Eligibility criteria Any peer-reviewed article search strategy. Review conduct and ARC South West Peninsula, University of Exeter Medical reported as a systematic review, rapid review, publication were rapid. Interest, as School, University of Exeter, overview, meta-analysis or qualitative evidence measured by altmetrics and citations, Exeter EX2 1LS, UK; r. a. synthesis in the title or abstract addressing a was high, and not correlated with abbott@ exeter. ac. uk research question relating to COVID-19. Articles quality. http://ebm.bmj.com/ described as meta-analyses but not undertaken as How might it impact clinical practice in part of a systematic or rapid review were excluded. the foreseeable future? Study selection and data extraction Abstract ► By being reported as ‘systematic and full text screening were undertaken by two reviews’, many readers may regard independent reviewers. Descriptive information evidence syntheses as high-quality on review type, purpose, population, size, evidence, irrespective of the actual on September 24, 2021 by guest. Protected copyright. citation and attention metrics were extracted methods undertaken. The challenge along with whether the review met the definition especially in times such as this of a systematic review according to six key pandemic is to provide indications of methodological criteria. For those meeting trustworthiness in evidence that is all criteria, additional data on methods and available in ‘real time’. Researchers, publication metrics were extracted. peer reviewers and journal editors Risk of bias For articles meeting all six criteria need to ensure that robust methods © Author(s) (or their required to meet the definition of a systematic have been used for research denoted employer(s)) 2021. Re- use review, AMSTAR-2 ((A MeaSurement Tool to as systematic reviews. permitted under CC Assess systematic Reviews, version 2.0) was used BY- NC. No commercial to assess the quality of the reported methods. re- use. See rights and Results 2334 articles were screened, resulting permissions. Published in 280 reviews being included: 232 systematic critical flaws: only a third reported registering by BMJ. reviews, 46 rapid reviews and 2 overviews. Less a protocol, and less than one in five searched To cite: Abbott R, Bethel A, than half reported undertaking critical appraisal named COVID-19 databases. Review conduct Rogers M, et al. BMJ and a third had no reproducible search strategy. and publication were rapid, with 52 of the 88 Evidence- Based Medicine There was considerable overlap in topics, with systematic reviews reported as being conducted Epub ahead of print: discordant findings. Eighty- eight of the 280 within 3 weeks, and a half published within [please include Day Month reviews met all six systematic review criteria. Of 3 weeks of submission. Researcher and media Year]. doi:10.1136/ these, just 3 were rated as of moderate or high interest, as measured by altmetrics and citations, bmjebm-2021-111710 quality on AMSTAR-2, with the majority having was high, and was not correlated with quality. BMJ Evidence- Based Medicine Month 2021 | volume 0 | number 0 | 1 Evidence synthesis Discussion This meta- research of early published COVID-19 2. In what ways are established systematic review methods being BMJ EBM: first published as 10.1136/bmjebm-2021-111710 on 3 June 2021. Downloaded from evidence syntheses found low- quality reviews being published compromised in an effort to inform transmission, diagnosis, at pace, often with short publication turnarounds. Despite being treatment and care of people with COVID-19? And what is the of low quality and many lacking robust methods, the reviews potential impact of these methodological shortcuts? received substantial attention across both academic and public 3. What is the methodological and reporting quality of published platforms, and the attention was not related to the quality of systematic reviews addressing research questions related to review methods. COVID-19? Interpretation Flaws in systematic review methods limit the 4. To what extent have published systematic reviews addressing validity of a review and the generalisability of its findings. et,Y COVID-19 research questions received attention from other by being reported as ‘systematic reviews’, many readers may well researchers, policy- makers and the media and what is its regard them as high- quality evidence, irrespective of the actual relationship with the methodological and reporting quality of methods undertaken. The challenge especially in times such as this published systematic reviews? pandemic is to provide indications of trustworthiness in evidence We planned to conduct a living systematic mapping review,14 that is available in ‘real time’. with initial searches from December 2019, regular and frequent PROSPERO registration number CRD42020188822. update searches and an online summary of relevant evidence. However, our available resources were unable to meet the demands of producing a living systematic review due to the volume of new COVID-19 systematic reviews being published. We, therefore, present, within this paper, a snapshot of the evidence from December Introduction 2019 to June 2020. Since the emergence of the COVID-19 in December 2019 in Wuhan, China, there has been a proliferation of research related Methods to its epidemiology, diagnosis, treatment, prevention and impact. This review is reported according to Preferred Reporting Items for As of 10 January 2021, there were over 77 000 records on PubMed Systematic Reviews and Meta- Analyses (PRISMA) guidelines.15 An a alone that included COVID-19 somewhere in the title or abstract. priori protocol was developed and registered on PROSPERO. On 5 May 2020, when we first drafted the protocol for this review, there were more than 60 published systematic reviews on Information sources and search strategy COVID-19 when searching by title in PubMed alone, and as of We searched PubMed (https:// pubmed. ncbi. nlm. nih. gov/), Episte- 10 January 2021, this stands at 1820. The COVID-19 Evidence monikos COVID-19 evidence (https:// app. iloveevidence. com/ loves/ Reviews resource (http:// covid19reviews. org/ index. cfm) suggests 5e6f db96 69c0 0e4a c072701d? utm= epdb_ en), the Cochrane Library that at this time, there are over 4000 systematic reviews, rapid of Systematic Reviews, the Cochrane COVID-19 Study Register reviews and evidence summaries on COVID-19 that have either (https:// COVID- 19. cochrane. org/), EMBASE (via OvidSP), CINAHL been published, or in the process of being carried out. Complete (via EBSCOHost) and Web of Science (WoS) Core Collection Making sense of research by bringing together studies in system- (Clarivate) and the WHO COVID-19 database (https://search. bvsalud. atic reviews, with or without meta-analysis, is a well- established org/ global- literature- on- novel- coronavirus- 2019- ncov/) for studies http://ebm.bmj.com/ method in medicine and health research.1 Cochrane and other described as systematic reviews or rapid reviews, published from evidence- based health programmes have promoted the use of December 2019. Terms for COVID-19 (eg, Coronavirus OR ‘corona systematic review methods globally.2–4 However, poorly conducted virus’ OR ‘2019 coronavirus’ OR ‘corona virus disease’ OR ‘novel reviews can lead to inaccurate representations of the evidence, inac- coronavirus’ OR ‘wuhan coronavirus’ OR ‘novel coronavirus’ OR curate estimates of treatment effectiveness, misleading conclusions ‘wuhan coronavirus’ OR ‘severe