An Investigation Into the Impact and Implications of Published Papers from Retracted Research: Systematic Search of Affected Literature
Total Page:16
File Type:pdf, Size:1020Kb
BMJ Open: first published as 10.1136/bmjopen-2019-031909 on 30 October 2019. Downloaded from PEER REVIEW HISTORY BMJ Open publishes all reviews undertaken for accepted manuscripts. Reviewers are asked to complete a checklist review form (http://bmjopen.bmj.com/site/about/resources/checklist.pdf) and are provided with free text boxes to elaborate on their assessment. These free text comments are reproduced below. ARTICLE DETAILS TITLE (PROVISIONAL) An investigation into the impact and implications of published papers from retracted research: systematic search of affected literature. AUTHORS Avenell, Alison; Stewart, Fiona; Grey, Andrew; Gamble, Greg; Bolland, Mark VERSION 1 – REVIEW REVIEWER David Robert Grimes Queen's University Belfast / University of Oxford REVIEW RETURNED 03-Jul-2019 GENERAL COMMENTS Firstly, my apologies for the delay in responding to this excellent and important paper. In medical science, we simply do no know enough yet about the proliferation of dubious research, and this paper is a commendable step in that direction.I admit I am not overly familiar with the subject area, but the theme of meta- research is one that I have some stake in, and I am heartened to see it being taken seriously. I cannot fault the paper on content, and the scope is perfect for http://bmjopen.bmj.com/ such an analysis. Other authors, for example, have looked at things like the Wakefield Lancet paper citations before and after retraction; but this is tainted by the fact that many of those citations would be to condemn or dismiss the work, and its infamy makes it an outlier already. Even so, I have two small suggestions to make this work even more powerful, if the authors care for them. on September 30, 2021 by guest. Protected copyright. (1) It might be worth doing a search of citing papers AFTER the date of retraction, to see if any of them clearly dismiss the findings of the suspect papers. Citations are a blunt instrument, and while I suspect it's unlikely here, it'd be worth checking to see. This would quickly answer the question whether bad science propagates faster than retractions allow. As the authors seem to largely answer this question, it's a minor point, but might be worth a keyword search. (2) On a related note, some visualisation would not go amiss if possible. For example, a graph of annual citations post-and pre retraction might give some insight into the extent of the problem we have in medical science, using this as a microcosm of a wider problem. One could even look at fitting whether the retractions had any impact on citation rate! Again, optional but happy to advise if it helps. 1 BMJ Open: first published as 10.1136/bmjopen-2019-031909 on 30 October 2019. Downloaded from Other than that, I think this is an extremely valuable contribution, and I look forward to seeing it in press. REVIEWER Jodi Schneider School of Information Sciences, University of Illinois at Urbana- Champaign, USA REVIEW RETURNED 09-Jul-2019 GENERAL COMMENTS This is really excellent work, and very novel. The main, fundamental issue I see is that results need to be presented more clearly for non-specialists. In particular, for best impact, it should be accessible to those who don't know the topic, and who don't know evidence-based medicine/systematic reviews/meta-analysis either. Try to get a colleague outside of evidence-based medicine to read this, and don't be afraid to summarize first and then explain. Do you think that, in the end, there are clinical impacts of the diffusion of error here? Be more clear about that. Overall, I encourage you to make the results punchier and easier to read. I have two, less fundamental objections with the current version: 1. The methods could be made more reproducible. Currently, it is difficult to understand exactly what you searched for and how. The most important questions, to me, are: -- how you found certain literature that does not appear to be in Scopus and Web of Science; -- whether it matters to you if your search is comprehensive (it may not); and -- whether you are sure you found ALL clinical guidelines or possibly missed some 2. Some relevant literature should be considered, particularly on http://bmjopen.bmj.com/ harms, readers'/authors' ignorance of retraction, and CrossMark (see below). You may also find it relevant to read about (though not necessarily to cite) cleaning up the literature in other fields; I've listed a few references in anesthesiology that your work reminded me of. I have also made numerous minor "in-line" comments with page/line references below. on September 30, 2021 by guest. Protected copyright. LITERATURE HARMS While you marshal a lot of very significant literature I think it's important to reference previous work on harms. These two come to mind: Steen RG. Retractions in the medical literature: how can patients be protected from risk?. Journal of medical ethics. 2012 Apr 1;38(4):228-32. Marcus Adam. A scientist's fraudulent studies put patients at risk. Science. 2018 Oct 26;362(6413): 394 DOI: 10.1126/science.362.6413.394-a READERS'/AUTHORS' IGNORANCE OF RETRACTION 2 BMJ Open: first published as 10.1136/bmjopen-2019-031909 on 30 October 2019. Downloaded from Work on ignorance of retraction is also available and needed. This is the most obvious: Davis PM. The persistence of error: a study of retracted articles on the Internet and in personal libraries. Journal of the Medical Library Association: JMLA. 2012 Jul;100(3):184. Alternately, you could consider citing work on citation of retraction. For instance, from one recent paper, "the vast majority of citations to retracted articles are positive despite of the clear retraction notice on the publisher’s platform and regardless of the reason for retraction." Bar-Ilan J, Halevi G. Post retraction citations in context: A case study. Scientometrics. 2017 Oct 1;113(1):547-65. A less-recent paper showed this poignantly in the case of Reuben's papers: Bornemann-Cimenti H, Szilagyi IS, Sandner-Kiesling A. Perpetuation of retracted publications using the example of the Scott S. Reuben case: Incidences, reasons and possible improvements. Science and engineering ethics. 2016 Aug 1;22(4):1063-72. CROSSMARK Add a link to CrossMark (or to a paper about it). I have not heard it called "The CrossMark policy" but just "CrossMark". The official site seems to be: https://www.crossref.org/services/crossmark/ This paper was written before it went live (I think) but is representative: Meyer CA. Distinguishing published scholarly content with CrossMark. Learned Publishing. 2011 Apr;24(2):87-93 http://bmjopen.bmj.com/ OTHER FIELDS' RESPONSE TO FRAUD Less important to cite but perhaps relevant to read and think about would be editorials such as the following: White PF, Kehlet H, Liu S. Perioperative analgesia: what do we still know?. Anesth Analg. 2009 May;108(5):1364-7. doi: 10.1213/ane.0b013e3181a16835. on September 30, 2021 by guest. Protected copyright. Polaner DM, Shafer SL. Anesth Analg. 2019 Apr;128(4):613-614. doi: 10.1213/ANE.0000000000004037. Page 1 line 1: Which are the 33 trial reports? I do not think we should need to go to your prior publication in order to identify these. We have to get to Table 1 or its discussion before we see the citations to the 12 of these articles you focus on. Page 1 line 10: Clarify the sentence structure to make it clear that "ostensibly involving" refers to the trial reports (not the review). Page 1 line 28: There has been limited work on patient harms. The paper you cite is a great one, but I find it related to your work primarily for focusing on diffusion of information through (citation) networks over time. 3 BMJ Open: first published as 10.1136/bmjopen-2019-031909 on 30 October 2019. Downloaded from Consider better supporting the following sentence -- perhaps with the Steen and/or Marcus (above). I thought, in fact, that you were citing Steen when I first read this: "Patients and research participants may be put at risk if decisions are based on findings that are later retracted because they were incorrect or unreliable.[3]" Consider adding one or both of the following papers, which are among the few to more pointedly find patient harms and clinical outcomes. To my mind, only limited support is available from the paper you cite ([3] van der Vet PE, Nijveen H. Propagation of errors in citation networks: a study involving the entire citation network of a widely cited paper published in, and later retracted from, the journal Nature. Res Integr Peer Rev 2016;1:3.). They do not center on this question though they have an in-passing literature review which provides a very good summary: "Particularly in the medical literature, there is the danger that patients are put at risk by what is concluded in articles that later have to be retracted. The Wakefield case is probably the most famous example. Wakefield and co-workers claimed to have found an association between measles vaccine and autism [20] but their article was retracted because of fraud 12 years later [21]. The false association has lingered on since then and may have caused unnecessary deaths through parents refusing measles vaccination of their children [22]. Treatments based on retracted articles put patients at risk [23] [note - this is the Steen above]. Neale and co- authors find no such cases in their study involving 102 articles retracted because of misconduct [17], while Begley and co-authors conclude the opposite [24] and Couzin and co-workers provide a concrete example [9]." Page 1 line 33: "unaware of the retraction" I would suggest that you cite on this point; perhaps ideal on this http://bmjopen.bmj.com/ point: Davis PM.