A Protocol of a Cross-Sectional Study Evaluating an Online Tool for Early Career Peer Reviewers Assessing Reports of Randomised Controlled Trials
Total Page:16
File Type:pdf, Size:1020Kb
Open Access Protocol BMJ Open: first published as 10.1136/bmjopen-2017-017462 on 15 September 2017. Downloaded from A protocol of a cross-sectional study evaluating an online tool for early career peer reviewers assessing reports of randomised controlled trials Anthony Chauvin,1,2,3 David Moher,4,5 Doug Altman,6 David L Schriger,7 Sabina Alam,8 Sally Hopewell,9 Daniel R Shanahan,10 Alessandro Recchioni,10 Philippe Ravaud,1,2 Isabelle Boutron1,2,11 To cite: Chauvin A, Moher D, ABSTRACT Strengths and limitations of this study Altman D, et al. A protocol Introduction Systematic reviews evaluating the impact of a cross-sectional study of interventions to improve the quality of peer review for ► To the best of our knowledge, this will be the first evaluating an online tool for biomedical publications highlighted that interventions were early career peer reviewers study that will assess the efficacy of using an limited and have little impact. This study aims to compare assessing reports of randomised innovative online tool to help early career peer the accuracy of early career peer reviewers who use an controlled trials. BMJ Open reviewers to assess the quality of reporting of innovative online tool to the usual peer reviewer process 2017;7:e017462. doi:10.1136/ randomised controlled trials. bmjopen-2017-017462 in evaluating the completeness of reporting and switched ► A randomised selection of trials from a large panel of primary outcomes in completed reports. biomedical journals. ► Prepublication history and Methods and analysis This is a cross-sectional additional material for this ► The absence of background for choosing the a priori study of individual two-arm parallel-group randomised paper are available online. To effect size. view these files, please visit controlled trials (RCTs) published in the BioMed Central the journal online (http:// dx. doi. series medical journals, BMJ, BMJ Open and Annals of org/ 10. 1136/ bmjopen- 2017- Emergency Medicine and indexed with the publication type 017462). ‘Randomised Controlled Trial’. First, we will develop an online editorial peer review process is described as tool and training module based (a) on the Consolidated journal editors relying on the views of inde- Received 2 May 2017 Standards of Reporting Trials (CONSORT) 2010 checklist pendent experts in making decisions on, for Revised 20 July 2017 and the Explanation and Elaboration document that would example, the publication of submitted manu- http://bmjopen.bmj.com/ Accepted 28 July 2017 be dedicated to junior peer reviewers for assessing the scripts or presentation of reports at meetings.2 completeness of reporting of key items and (b) the Centre The peer review system is considered the best for Evidence-Based Medicine Outcome Monitoring Project method for helping scientific editors decide process used to identify switched outcomes in completed on the acceptability of a manuscript for reports of the primary results of RCTs when initially publication.3 Nevertheless, the effectiveness submitted. Then, we will compare the performance of early of the system has been questioned. In 2010, career peer reviewers who use the online tool to the usual peer review process in identifying inadequate reporting and a report commissioned by the UK House of switched outcomes in completed reports of RCTs at initial Commons showed that the cost to the UK on September 25, 2021 by guest. Protected copyright. journal submission. The primary outcome will be the mean Higher Education Institutions in terms of number of items accurately classified per manuscript. The staff time was £110 to £165 million per year secondary outcomes will be the mean number of items for peer review and up to £30 million per year accurately classified per manuscript for the CONSORT items for the work done by editors and editorial and the sensitivity, specificity and likelihood ratio to detect boards.4 Worldwide, peer review is estimated the item as adequately reported and to identify a switch in to cost £1.9 billion annually and accounts outcomes. We aim to include 120 RCTs and 120 early career for about one-quarter of the overall costs of peer reviewers. scholarly publishing and distribution.5 The Ethics and dissemination The research protocol was human resources were estimated to be about approved by the ethics committee of the INSERM Institutional 6 Review Board (21 January 2016). The study is based on 15 million hours by 2010. voluntary participation and informed written consent. Furthermore, the peer review system may Trial registration number NCT03119376. fail to identify important flaws in the quality For numbered affiliations see of submitted manuscripts and published end of article. articles.7 8 A recent study by Hopewell et al9 Correspondence to INTRODUCTION showed that peer reviewers failed to detect Dr Anthony Chauvin; The peer review process is a cornerstone important deficiencies in the reporting of anthony. chauvin@ aphp. fr of biomedical research publication.1 The methods and results of randomised trials. Chauvin A, et al. BMJ Open 2017;7:e017462. doi:10.1136/bmjopen-2017-017462 1 Open Access BMJ Open: first published as 10.1136/bmjopen-2017-017462 on 15 September 2017. Downloaded from There is also considerable indirect evidence of the failure be reported extracted from the Explanation and Elab- of peer review. Systematic reviews of clinical questions oration document, as well as (an) example(s) of good highlight a high prevalence of key information not being reporting. provided in peer-reviewed articles. Moreover, there is The peer reviewer tool will use the existing CobWeb evidence to suggest that peer reviewers are not able to elements reworded to be appropriate for peer reviewers detect fraud,10 mistakes11 12 or spin.13 14 Systematic reviews and for the items not considered in CobWeb, we will evaluating the impact of interventions to improve the extract the bullet points from the Explanation and Elab- quality of peer review for biomedical publications high- oration document of the CONSORT 2010 following the lighted few randomised controlled trials (RCTs) and that same process.21 interventions such as training have little or no impact.15 16 The tool will feature bullet points eliciting the meaning How the peer review process is currently organised can of each checklist item. This tool will allow for standard- explain the difficulties encountered. A number of tasks ising the completion of this task, be an opportunity to are expected from peer reviewers when assessing reports train early career peer reviewers to become more expert of an RCT.17 They have to assess the completeness of peer reviewers and be used to provide standardised reporting, whether outcomes were switched from that and personalised feedback to authors based on the reported in the trial registry, the choice of the compar- CONSORT items and bullet points; the authors would ator and intervention, the relevance of outcomes, the receive a request to report each item (with associated risk of bias, the external validity, the presence of spin, the bullet points) that the early career peer reviewer identi- originality, importance and relevance of the RCT to the fied as being inadequately reported. journal’s readers, etc. It may be unrealistic to expect a The objectives of this project are as follows: peer reviewer to complete all of these tasks. 1. Develop an online tool and training module based Furthermore, the assessment of the completeness of (a) on the CONSORT 2010 checklist and the reporting should be a prerequisite in the peer review Explanation and Elaboration document modified for process, because lack of transparency is a barrier to the early career peer reviewers (ie, early stage researchers: assessment of important aspects of a trial such as the risk of masters students, PhD students, residents involved bias18 and reproducibility.19 Another essential task of peer in clinical research during their study and clinicians review is the identification of switched outcomes (ie, a involved in clinical research who never peer reviewed discrepancy between the prespecified outcomes reported a manuscript) for assessing the completeness of in the trial registry or protocol and outcomes reported reporting of 10 key items and (b) the COMPare in the manuscript). To avoid reporting bias(es),20 all process used to identify switched in primary outcomes prespecified outcomes should be reported in the manu- in completed reports of RCTs. script, and authors should report any changes to trial 2. Compare the performance of early career peer 21 outcomes after the trial started, with reasons. The Centre reviewers who use the peer reviewer tool with usual http://bmjopen.bmj.com/ for Evidence-Based Medicine Outcome Monitoring peer reviewers in identifying inadequate reporting Project (COMPare) assessed RCTs published in the five and switched outcomes in completed reports of RCTs. general medical journals with the highest impact factor. The planning, conduct, analysis and reporting of the They compared the outcomes published to the outcomes study were discussed by the study’s scientific committee. recorded in trial registries or protocols. Among the 67 RCTs examined in July 2016 (www. COMPare- Trials. org), only nine had no discrepancies; 354 outcomes were not METHODS AND ANALYSIS 18 Study design reported and 357 new outcomes were ‘silently’ added. on September 25, 2021 by guest. Protected copyright. Despite being essential, the assessment of the complete- We approached the issue of peer reviewing as if it