An Investigation Into the Impact and Implications of Published Papers from Retracted Research: Systematic Search of Affected Literature

BMJ Open: first published as 10.1136/bmjopen-2019-031909 on 30 October 2019. Downloaded from

PEER REVIEW HISTORY

BMJ Open publishes all reviews undertaken for accepted manuscripts. Reviewers are asked to complete a checklist review form (http://bmjopen.bmj.com/site/about/resources/checklist.pdf) and are provided with free text boxes to elaborate on their assessment. These free text comments are reproduced below.

ARTICLE DETAILS

TITLE (PROVISIONAL) An investigation into the impact and implications of published papers from retracted research: systematic search of affected literature. AUTHORS Avenell, Alison; Stewart, Fiona; Grey, Andrew; Gamble, Greg; Bolland, Mark

VERSION 1 – REVIEW

REVIEWER David Robert Grimes Queen's University Belfast / University of Oxford REVIEW RETURNED 03-Jul-2019

GENERAL COMMENTS Firstly, my apologies for the delay in responding to this excellent and important paper. In medical science, we simply do no know enough yet about the proliferation of dubious research, and this paper is a commendable step in that direction.I admit I am not overly familiar with the subject area, but the theme of meta- research is one that I have some stake in, and I am heartened to see it being taken seriously.

I cannot fault the paper on content, and the scope is perfect for http://bmjopen.bmj.com/ such an analysis. Other authors, for example, have looked at things like the Wakefield Lancet paper citations before and after retraction; but this is tainted by the fact that many of those citations would be to condemn or dismiss the work, and its infamy makes it an outlier already.

Even so, I have two small suggestions to make this work even

more powerful, if the authors care for them. on September 30, 2021 by guest. Protected copyright.

(1) It might be worth doing a search of citing papers AFTER the date of retraction, to see if any of them clearly dismiss the findings of the suspect papers. Citations are a blunt instrument, and while I suspect it's unlikely here, it'd be worth checking to see. This would quickly answer the question whether bad science propagates faster than retractions allow. As the authors seem to largely answer this question, it's a minor point, but might be worth a keyword search.

(2) On a related note, some visualisation would not go amiss if possible. For example, a graph of annual citations post-and pre retraction might give some insight into the extent of the problem we have in medical science, using this as a microcosm of a wider problem. One could even look at fitting whether the retractions had any impact on citation rate! Again, optional but happy to advise if it helps.

BMJ Open: first published as 10.1136/bmjopen-2019-031909 on 30 October 2019. Downloaded from

Other than that, I think this is an extremely valuable contribution, and I look forward to seeing it in press.

REVIEWER Jodi Schneider School of Information Sciences, University of Illinois at Urbana- Champaign, USA REVIEW RETURNED 09-Jul-2019

GENERAL COMMENTS This is really excellent work, and very novel.

The main, fundamental issue I see is that results need to be presented more clearly for non-specialists. In particular, for best impact, it should be accessible to those who don't know the topic, and who don't know evidence-based medicine/systematic reviews/meta-analysis either. Try to get a colleague outside of evidence-based medicine to read this, and don't be afraid to summarize first and then explain. Do you think that, in the end, there are clinical impacts of the diffusion of error here? Be more clear about that. Overall, I encourage you to make the results punchier and easier to read.

I have two, less fundamental objections with the current version:

1. The methods could be made more reproducible. Currently, it is difficult to understand exactly what you searched for and how. The most important questions, to me, are: -- how you found certain literature that does not appear to be in Scopus and Web of Science; -- whether it matters to you if your search is comprehensive (it may not); and -- whether you are sure you found ALL clinical guidelines or possibly missed some

2. Some relevant literature should be considered, particularly on http://bmjopen.bmj.com/ harms, readers'/authors' ignorance of retraction, and CrossMark (see below). You may also find it relevant to read about (though not necessarily to cite) cleaning up the literature in other fields; I've listed a few references in anesthesiology that your work reminded me of.

I have also made numerous minor "in-line" comments with

page/line references below. on September 30, 2021 by guest. Protected copyright.

LITERATURE

HARMS While you marshal a lot of very significant literature I think it's important to reference previous work on harms. These two come to mind:

Steen RG. Retractions in the medical literature: how can patients be protected from risk?. Journal of medical ethics. 2012 Apr 1;38(4):228-32.

Marcus Adam. A scientist's fraudulent studies put patients at risk. Science. 2018 Oct 26;362(6413): 394 DOI: 10.1126/science.362.6413.394-a

READERS'/AUTHORS' IGNORANCE OF RETRACTION

BMJ Open: first published as 10.1136/bmjopen-2019-031909 on 30 October 2019. Downloaded from

Work on ignorance of retraction is also available and needed. This is the most obvious: Davis PM. The persistence of error: a study of retracted articles on the Internet and in personal libraries. Journal of the Medical Library Association: JMLA. 2012 Jul;100(3):184.

Alternately, you could consider citing work on citation of retraction. For instance, from one recent paper, "the vast majority of citations to retracted articles are positive despite of the clear retraction notice on the publisher’s platform and regardless of the reason for retraction." Bar-Ilan J, Halevi G. Post retraction citations in context: A case study. Scientometrics. 2017 Oct 1;113(1):547-65.

A less-recent paper showed this poignantly in the case of Reuben's papers: Bornemann-Cimenti H, Szilagyi IS, Sandner-Kiesling A. Perpetuation of retracted publications using the example of the Scott S. Reuben case: Incidences, reasons and possible improvements. Science and engineering ethics. 2016 Aug 1;22(4):1063-72.

CROSSMARK Add a link to CrossMark (or to a paper about it). I have not heard it called "The CrossMark policy" but just "CrossMark".

The official site seems to be: https://www.crossref.org/services/crossmark/

This paper was written before it went live (I think) but is representative: Meyer CA. Distinguishing published scholarly content with CrossMark. Learned Publishing. 2011 Apr;24(2):87-93

http://bmjopen.bmj.com/ OTHER FIELDS' RESPONSE TO FRAUD Less important to cite but perhaps relevant to read and think about would be editorials such as the following:

White PF, Kehlet H, Liu S. Perioperative analgesia: what do we still know?. Anesth Analg. 2009 May;108(5):1364-7. doi: 10.1213/ane.0b013e3181a16835. on September 30, 2021 by guest. Protected copyright.

Polaner DM, Shafer SL. Anesth Analg. 2019 Apr;128(4):613-614. doi: 10.1213/ANE.0000000000004037.

Page 1 line 1: Which are the 33 trial reports? I do not think we should need to go to your prior publication in order to identify these. We have to get to Table 1 or its discussion before we see the citations to the 12 of these articles you focus on.

Page 1 line 10: Clarify the sentence structure to make it clear that "ostensibly involving" refers to the trial reports (not the review).

Page 1 line 28: There has been limited work on patient harms. The paper you cite is a great one, but I find it related to your work primarily for focusing on diffusion of information through (citation) networks over time.

BMJ Open: first published as 10.1136/bmjopen-2019-031909 on 30 October 2019. Downloaded from

Consider better supporting the following sentence -- perhaps with the Steen and/or Marcus (above). I thought, in fact, that you were citing Steen when I first read this: "Patients and research participants may be put at risk if decisions are based on findings that are later retracted because they were incorrect or unreliable.[3]" Consider adding one or both of the following papers, which are among the few to more pointedly find patient harms and clinical outcomes.

To my mind, only limited support is available from the paper you cite ([3] van der Vet PE, Nijveen H. Propagation of errors in citation networks: a study involving the entire citation network of a widely cited paper published in, and later retracted from, the journal Nature. Res Integr Peer Rev 2016;1:3.). They do not center on this question though they have an in-passing literature review which provides a very good summary: "Particularly in the medical literature, there is the danger that patients are put at risk by what is concluded in articles that later have to be retracted. The Wakefield case is probably the most famous example. Wakefield and co-workers claimed to have found an association between measles vaccine and autism [20] but their article was retracted because of fraud 12 years later [21]. The false association has lingered on since then and may have caused unnecessary deaths through parents refusing measles vaccination of their children [22]. Treatments based on retracted articles put patients at risk [23] [note - this is the Steen above]. Neale and co- authors find no such cases in their study involving 102 articles retracted because of misconduct [17], while Begley and co-authors conclude the opposite [24] and Couzin and co-workers provide a concrete example [9]."

Page 1 line 33: "unaware of the retraction"

I would suggest that you cite on this point; perhaps ideal on this http://bmjopen.bmj.com/ point: Davis PM. The persistence of error: a study of retracted articles on the Internet and in personal libraries. Journal of the Medical Library Association: JMLA. 2012 Jul;100(3):184. The citation of retracted papers (widely studied) also provides some evidence.

Page 5 line 10: It would be better to give full search strings, if you on September 30, 2021 by guest. Protected copyright. still have them. If not, consider recording those when you do similar research in the future.

Also, are guidelines typically formally published in the sorts of venues that Scopus and Web of Science index? Later (page 6 line 40), you mention AHRQ reports but it's not clear how you found those, or whether you attempted to be exhaustive in finding other guidelines by searching "grey literature" in some way.

Page 5, line 31: "quantitative findings" could be made more explicit. For instance, forest plots?

Page 5, line 53: Make more clear how you checked for "any notice, update, correction, or retraction resulting from recognition that the publication was potentially influenced by the affected trial reports" For instance: Did you repeat your Scopus/Web of Science search? Look on publisher websites? Use CrossMark? Or...?

BMJ Open: first published as 10.1136/bmjopen-2019-031909 on 30 October 2019. Downloaded from

Page 6, line 15: Personally, I find "Twelve trial reports...met our inclusion criteria" a bit confusing, because throughout you have been referring to 33 "affected trial reports". consider being even more clear here to help the reader. It's not clear why you limit to the 12, instead of all 33. From Table 1, I see that your inclusion criteria relate to impact factor and being "marked as retracted on both PuMed and Medline by November 2018". Give further explanation of these inclusion criteria and why you chose them.

Page 6, line 23: "both Medline and PubMed" is unclear. Were you using, e.g. Ovid Medline? (At the PubMed website, Medline is searched as a subset of PubMed, see "MEDLINE, PubMed, and PMC (PubMed Central): How are they different?" https://www.nlm.nih.gov/bsd/difference.html )

Page 6, line 65: "excluded" could be glossed for those not familiar with systematic reviews. What is important is not that they were excluded (per se), but the consequence: that the conclusions of the reviews do not depend on the "excluded affected trial reports".

Page 7 lines 17-23: In the discussion section you make a factual observation about documented best practices which is closely connected to what you say here. I'd repeat it here: "recommendations for caution in deriving conclusions from data from a very limited number of authors and centres [72,73]". Many readers don't know this, and it's a relevant comment to make twice (here, as well as where it currently is, in the discussion section, where it clarifies your comments at lines page 12 line 17).

Page 7 lines 43-50: Very important. Consider rewriting this whole paragraph to make the point more clearly. The fact that a sensitivity analysis was done is less important than the impact. Further, as a non-expert, I don't know whether the .23 vs. .30 odds ratio is meaningful (e.g., is this statistically significant? Clinically http://bmjopen.bmj.com/ significant? Would it impact decision-making?). Gloss this for those not expert in hip fracture prevention, please.

Page 7 paragraph starting line 53: As above, this paragraph could be strengthened and made more clear for those not expert in the hip fracture literature. The last sentence seems to be the most important.

on September 30, 2021 by guest. Protected copyright. Page 8 paragraph starting line 9: The most important part of this paragraph is a bit buried to my taste: "Data provided in these publications were insufficient to permit re-analysis after removal of the affected trial reports." Important on its own. And this generates another suggestion, that sufficient data be included in meta- analyses in order to enable such analyses.

Page 8 line 24: Bold the subheading here.

Page 8 paragraph starting line 26. Can you show HOW MUCH the findings "would be different without the affected trial report"? Again, a more journalistic style (finding first, then justification) would be more readable here.

Page 8 line 50: Bold the subheading here.

Page 9 line 35: Bold the subheading here.

BMJ Open: first published as 10.1136/bmjopen-2019-031909 on 30 October 2019. Downloaded from

Page 9 line 36: Again, I'd advocate a gloss for why "did not include" is important.

Page 10 lines 33 & 41: How did you find [52] and [54], since presumably (given the "no citation count available"), it wasn't from Scopus or Web of Science?

Page 12 line 33: It is most likely that the authors/publishers have no knowledge. One suggestion that could arise from your work is, even if no corrections are needed, an update be issued stating that the authors have examined the work in line with the correction/retraction, and certify no changes are needed or that changes have been made as needed. You mention this in Box 1, but consider adding a mention to the text. One way to facilitate this would be "amendments", versioning articles, as suggested in this paper: Barbour V, Bloom T, Lin J, Moylan E. Amending published articles: time to rethink retractions and corrections?. F1000Research. 2017 Nov 6;6. https://f1000research.com/articles/6-1960/v1/iparadigms

Page 14 line 10: One very new tool is the Zotero Retraction Watch integration: https://www.zotero.org/blog/retracted-item-notifications/

Out of self-promotion, I will point out an unrelated prototype: https://github.com/nikolausn/ReTrackers which students & I wrote about in this conference paper to be presented: Yi-Yun Cheng, Nikolaus Parulian, Tzu-Kun Hsiao, Ly Dinh, Janina Sarol, Jodi Schneider. “ReTracker: actively and automatically matching retraction metadata in Zotero”. In ASIS&T 2019, Melbourne, Australia, October.

http://jodischneider.com/pubs/asist2019retracker.pdf http://bmjopen.bmj.com/

STRENGTHS AND LIMITATIONS OF THIS STUDY - You mention that you "only examined the effect on published research". It's not clear to me whether guidelines are always published. Did you get ALL guidelines, or just some? That could be made more clear.

- This work is quite novel; limited work has on September 30, 2021 by guest. Protected copyright.

BOX 1 - I have not heard it called "The CrossMark policy" but just "CrossMark". Check and add a citation please.

I think (and hope) my main concerns will be relatively straightforward to address, and that revising the results section in particular will make the paper have much more impact in the long- run. I very much look forward to the next version!

VERSION 1 – AUTHOR RESPONSE

Reviewer(s)' Comments to Author:

Reviewer: 1

BMJ Open: first published as 10.1136/bmjopen-2019-031909 on 30 October 2019. Downloaded from

Reviewer Name: David Robert Grimes Institution and Country: Queen's University Belfast / University of Oxford Please state any competing interests or state ‘None declared’: None

Please leave your comments for the authors below Firstly, my apologies for the delay in responding to this excellent and important paper. In medical science, we simply do no know enough yet about the proliferation of dubious research, and this paper is a commendable step in that direction.I admit I am not overly familiar with the subject area, but the theme of meta-research is one that I have some stake in, and I am heartened to see it being taken seriously.

I cannot fault the paper on content, and the scope is perfect for such an analysis. Other authors, for example, have looked at things like the Wakefield Lancet paper citations before and after retraction; but this is tainted by the fact that many of those citations would be to condemn or dismiss the work, and its infamy makes it an outlier already.

Even so, I have two small suggestions to make this work even more powerful, if the authors care for them.

Reply – thank you for this comment. To further answer the question as to whether bad science propagates faster than retractions allow, we have now added citations to the Discussion clarifying that concerns were expressed about research from this group in Japan in five publications as far back as 2004 – 2007. Thus, there was also a long contributory delay before investigations started and retractions occurred.

We have added the dates of retractions to Table 1, which did not begin until June 2016. All of the citing publications, except for Binks (published August 2016 includes one affected trial report retracted in June 2016) and Qaseem (published June 2017 includes affected trial report retracted in June and

July 2016) were published before affected trial reports were retracted. We have also added that we http://bmjopen.bmj.com/ examined 40 citations after affected trial reports were retracted, 34 (85%) expressed no concerns about the affected trial reports, and 6 (15%) quoted the affected trial reports but discounted their findings.

The following wording has been added to the Methods:

‘In July 2019, we searched Web of Science for publications that cited the affected trial reports after they had been retracted, to examine whether these publications mentioned that the affected trial on September 30, 2021 by guest. Protected copyright. reports had been retracted.’

To the Results:

‘We examined 40 publications in July 2019 that cited any of the 12 affected trial reports after they were retracted. Thirty-four publications (85%) expressed no concern about the affected trial reports, and six (15%) cited the affected trial reports but discounted their findings as a result of misconduct.’

Reply – follow-up time post retraction is still too limited to provide graphical data, but we have provided data as above. We have also expanded our discussion of the 2019 paper by Mott, which examines citation rates post retraction, and included a reference by Mistry et al.[90]

BMJ Open: first published as 10.1136/bmjopen-2019-031909 on 30 October 2019. Downloaded from

‘Mott and colleagues found a 46% reduction in citations of randomised clinical trial reports in the first year after retraction,[83] and retractions also reduce subsequent publication by authors associated with misconduct.[90]’

Other than that, I think this is an extremely valuable contribution, and I look forward to seeing it in press.

Reviewer: 2 Reviewer Name: Jodi Schneider Institution and Country: School of Information Sciences, University of Illinois at Urbana-Champaign, USA Please state any competing interests or state ‘None declared’: None declared.

Please leave your comments for the authors below This is really excellent work, and very novel.

The main, fundamental issue I see is that results need to be presented more clearly for non- specialists. In particular, for best impact, it should be accessible to those who don't know the topic, and who don't know evidence-based medicine/systematic reviews/meta-analysis either. Try to get a colleague outside of evidence-based medicine to read this, and don't be afraid to summarize first and then explain. Do you think that, in the end, there are clinical impacts of the diffusion of error here? Be more clear about that. Overall, I encourage you to make the results punchier and easier to read.

Reply – thank you. We have revised our document in response to your comments, particularly with regard to readability in the results. We have given the revised paper to others with no background in evidence-based medicine for their comments, and amended the paper accordingly. We have made it clear that we think that there are clinical impacts, and expanded on our discussion by covering in more details the limitations of our research, and also mentioning delays in retracting articles contributing to impact.

Thank you for all your suggested references, which we carefully read. We have included some of these and also included reference to a new publication by Fanelli and Moher (Fanelli D, Moher D. What difference do retractions make? An estimate of the epistemic impact of retractions on recent meta-analyses. bioRxiv 2019;https://doi.org/10.1101/734137), which examines the likely impact of retracted studies on effect sizes in meta-analyses.[92] http://bmjopen.bmj.com/

I have two, less fundamental objections with the current version:

Reply – we have improved our description of the methods (please see below) and added to the discussion

‘We have probably missed guidelines in our evaluation of citing publications, since these are poorly covered by indexing databases. ARHQ full guidelines[51,54] were identified through linked journal articles, and SIGN guidelines from personal databases.[50] Thus, we have probably underestimated the impact of these 12 trial reports.’

We have also made it clear in the Discussion that we did not set out to include all available impact.

2. Some relevant literature should be considered, particularly on harms, readers'/authors' ignorance of retraction, and CrossMark (see below). You may also find it relevant to read about (though not necessarily to cite) cleaning up the literature in other fields; I've listed a few references in anesthesiology that your work reminded me of.

Reply – responses to this comment are given under individual items below.

BMJ Open: first published as 10.1136/bmjopen-2019-031909 on 30 October 2019. Downloaded from

I have also made numerous minor "in-line" comments with page/line references below.

LITERATURE

HARMS While you marshal a lot of very significant literature I think it's important to reference previous work on harms. These two come to mind:

Steen RG. Retractions in the medical literature: how can patients be protected from risk?. Journal of medical ethics. 2012 Apr 1;38(4):228-32.

Marcus Adam. A scientist's fraudulent studies put patients at risk. Science. 2018 Oct 26;362(6413): 394 DOI: 10.1126/science.362.6413.394-a

Reply – these references have been added to the Background.

READERS'/AUTHORS' IGNORANCE OF RETRACTION Work on ignorance of retraction is also available and needed. This is the most obvious: Davis PM. The persistence of error: a study of retracted articles on the Internet and in personal libraries. Journal of the Medical Library Association: JMLA. 2012 Jul;100(3):184.

Reply – we think this the most relevant reference to quote on this topic, so have added its reference to the Background.

Reply – this reference is already included in the Discussion with respect to the Reuben case.

CROSSMARK Add a link to CrossMark (or to a paper about it). I have not heard it called "The CrossMark policy" but just "CrossMark".

on September 30, 2021 by guest. Protected copyright. The official site seems to be: https://www.crossref.org/services/crossmark/ This paper was written before it went live (I think) but is representative: Meyer CA. Distinguishing published scholarly content with CrossMark. Learned Publishing. 2011 Apr;24(2):87-93

Reply – the link to CrossMark has been added for the most up to date information.

OTHER FIELDS' RESPONSE TO FRAUD Less important to cite but perhaps relevant to read and think about would be editorials such as the following:

White PF, Kehlet H, Liu S. Perioperative analgesia: what do we still know?. Anesth Analg. 2009 May;108(5):1364-7. doi: 10.1213/ane.0b013e3181a16835.

Polaner DM, Shafer SL. Anesth Analg. 2019 Apr;128(4):613-614. doi: 10.1213/ANE.0000000000004037.

BMJ Open: first published as 10.1136/bmjopen-2019-031909 on 30 October 2019. Downloaded from

Reply – thank you for these references, which are interesting reading. We added the White reference to our mention of the Reuben case in the Discussion.

Reply – these have now been added as an Appendix, with a link provided in the introduction.

Page 1 line 10: Clarify the sentence structure to make it clear that "ostensibly involving" refers to the trial reports (not the review).

Reply – changed to ‘with RCT reports ostensibly involving large numbers…’

To my mind, only limited support is available from the paper you cite ([3] van der Vet PE, Nijveen H. Propagation of errors in citation networks: a study involving the entire citation network of a widely cited paper published in, and later retracted from, the journal Nature. Res Integr Peer Rev 2016;1:3.). They do not center on this question though they have an in-passing literature review which provides a very good summary: "Particularly in the medical literature, there is the danger that patients are put at risk by what is concluded in articles that later have to be retracted. The Wakefield case is probably the most famous example. Wakefield and co-workers claimed to have found an association between measles vaccine and autism [20] but their article was retracted because of fraud 12 years later [21]. The false http://bmjopen.bmj.com/ association has lingered on since then and may have caused unnecessary deaths through parents refusing measles vaccination of their children [22]. Treatments based on retracted articles put patients at risk [23] [note - this is the Steen above]. Neale and co-authors find no such cases in their study involving 102 articles retracted because of misconduct [17], while Begley and co-authors conclude the opposite [24] and Couzin and co-workers provide a concrete example [9]."

Reply – thank you for providing these references, we have removed the citation to van der Vet from the Background and replaced it with the references from Marcus and Steen. on September 30, 2021 by guest. Protected copyright.

Page 1 line 33: "unaware of the retraction" I would suggest that you cite on this point; perhaps ideal on this point: Davis PM. The persistence of error: a study of retracted articles on the Internet and in personal libraries. Journal of the Medical Library Association: JMLA. 2012 Jul;100(3):184. The citation of retracted papers (widely studied) also provides some evidence.

Reply – this reference has been inserted.

Page 5 line 10: It would be better to give full search strings, if you still have them. If not, consider recording those when you do similar research in the future.

Reply – unfortunately we no longer have these, but will make sure this doesn’t happen again.

BMJ Open: first published as 10.1136/bmjopen-2019-031909 on 30 October 2019. Downloaded from

Reply – please see earlier comment.

Page 5, line 31: "quantitative findings" could be made more explicit. For instance, forest plots?

Reply – inserted ‘such as summary risk ratios in forest plots’

Reply – we have inserted ‘publishers’ websites’.

Page 6, line 15: Personally, I find "Twelve trial reports...met our inclusion criteria" a bit confusing, because throughout you have been referring to 33 "affected trial reports". consider being even more clear here to help the reader. It's not clear why you limit to the 12, instead of all 33. From Table 1, I see that your inclusion criteria relate to impact factor and being "marked as retracted on both PubMed and Medline by November 2018". Give further explanation of these inclusion criteria and why you chose them.

Reply – we have revised the methods to say:

‘We studied the impact of a subgroup of the 33 affected trial reports whose integrity was analysed in our systematic review.[1] This subgroup of trial reports was used because these trials had hip fracture as an outcome, arguably the most important consequence of osteoporosis, and affected trial reports on this outcome are likely to have the greatest impact. We included all affected trial reports with hip fracture outcomes that had also been published in higher impact journals (ISI Web of Knowledge impact factor > 4).’

We have revised the results to say:

‘Twelve trial reports from the original 33 were identified by us for evaluation. These 12 affected trial reports all had hip fracture outcomes and were published in journals with impact factors > 4 between http://bmjopen.bmj.com/ 1997 and 2011, with 3182 reported participants (see Table 1).’

Reply – we have clarified that this is Ovid Medline. We don’t know why there should be a difference between Ovid Medline and PubMed in their listings, one displaying a retraction and not the other. on September 30, 2021 by guest. Protected copyright.

Reply – changed to ‘cited but did not include any data from affected trial reports in their analyses.’

Reply – this paragraph has been simplified and the comment about caution placed at the beginning of the paragraph.

BMJ Open: first published as 10.1136/bmjopen-2019-031909 on 30 October 2019. Downloaded from non-expert, I don't know whether the .23 vs. .30 odds ratio is meaningful (e.g., is this statistically significant? Clinically significant? Would it impact decision-making?). Gloss this for those not expert in hip fracture prevention, please.

Reply – this has been re-ordered and rewritten to clarify the impact.

Reply – we have re-ordered this paragraph and made it clearer.

Page 8 paragraph starting line 9: The most important part of this paragraph is a bit buried to my taste: "Data provided in these publications were insufficient to permit re-analysis after removal of the affected trial reports." Important on its own. And this generates another suggestion, that sufficient data be included in meta-analyses in order to enable such analyses.

Reply -we have moved this point to the beginning of the paragraph and added a comment about authors providing sufficient data, stating that

‘Cases where we were unable to reanalyse meta-analyses after removal of affected trial reports, would have been facilitated by authors providing open access to all their data.’

Page 8 line 24: Bold the subheading here.

Reply – our understanding is that journal formatting requires third order headings to not be in bold.

Reply – reworded to

‘The results from one affected trial report [10] changed the findings for calcium, vitamin D and vitamin http://bmjopen.bmj.com/ K given together for falls prevention. One Cochrane review on the prevention of falls in the community[30] (756 citations, November 2018) included an unpooled meta-analysis of data from one this trial report[10] and one other trial of calcium alone, relating to the number of fractures caused by falling. The analysis shows a large, statistically significant, reduction in fracture risk in the intervention group from the eligible trial report (Risk Ratio 0.13, 95% CI 0.04 to 0.43), and a null effect in the other trial[46] (Risk Ratio 0.90, 95% CI 0.69 to 1.16).

Page 8 line 50: Bold the subheading here. on September 30, 2021 by guest. Protected copyright.

Reply – as above.

Page 9 line 35: Bold the subheading here.

Reply – as above.

Page 9 line 36: Again, I'd advocate a gloss for why "did not include" is important.

Reply – changed to

‘A further four systematic reviews cited but did not include affected trial reports in their reviews as a result of existing concerns with data,[44-46] or awaiting responses to enquiries about data.[47]’

Page 10 lines 33 & 41: How did you find [52] and [54], since presumably (given the "no citation count available"), it wasn't from Scopus or Web of Science?

BMJ Open: first published as 10.1136/bmjopen-2019-031909 on 30 October 2019. Downloaded from

Reply – these came from Google Scholar and references in journal articles, details now given in the text.

Reply – This has been amended to

‘With one exception,[18-20] authors and/or journals of citing publications have either not identified that their publications have been compromised, or decided no action is required, although the latter seems unlikely. To our knowledge, bibliographic database/journal/publisher/guideline developer structures are not established that permit systematic identification and correction of publications that are affected by the inclusion of research with compromised integrity. Even if removing the affected trial reports did not influence their conclusions, citing authors should publish an update. This should give details of their examination of the impact of the correction or retraction on their own work, and confirm that changes are not required or have been made. This would remove uncertainty in the interpretation of their work.[76,77] This could be aided by publishing an amended article, with an updated version number, as has been suggested by Barbour and colleagues.[78]’

Page 14 line 10: One very new tool is the Zotero Retraction Watch integration: https://www.zotero.org/blog/retracted-item-notifications/

Australia, October. http://jodischneider.com/pubs/asist2019retracker.pdf http://bmjopen.bmj.com/

Reply – we have added ‘Zotero software that is linked to Retraction Watch’s database,[92] or ReTracker linked to Retractions in PubMed[93] might facilitate authors’ awareness of retractions.’

Reply – Article Summary reworded to ‘We only examined the effect on published research we were able to identify, and may have not found all publications, especially guidelines. We did not examine impact on other forms of influence, e.g. grant applications, drug company documents.’

- This work is quite novel; limited work has

Reply – We are not sure what comment the Reviewer was intending to make here.

BOX 1 - I have not heard it called "The CrossMark policy" but just "CrossMark". Check and add a citation please.

Reply – This has been changed to Crossmark and a reference provided.

BMJ Open: first published as 10.1136/bmjopen-2019-031909 on 30 October 2019. Downloaded from

VERSION 2 – REVIEW

REVIEWER David Robert Robert Grimes Dublin City University, Ireland University of Oxford, UK REVIEW RETURNED 22-Aug-2019

GENERAL COMMENTS The authors have fully addressed by questions. I would love to see a follow up on this in future, perhaps 5 years from now to see how much these dubious articles have propagated in terms of citations - and whether those citations, being a blunt instrument, note the papers were retracted or not.

VERSION 2 – AUTHOR RESPONSE

Reviewer(s)' Comments to Author:

Reviewer: 2 Reviewer Name: Jodi Schneider

Institution and Country: UIUC, USA

Please state any competing interests or state ‘None declared’: None declared

Please leave your comments for the authors below All my previous comments were addressed. Thank you to the authors! It's nice, too, to see additional very recent work (like [92] on meta- analysis/retraction).

I have only one very small substantive point: Page 6 line 35: "on both" -- were any affected trial reports marked as retracted on one but not the other? (I suppose not, but consider mentioning that.) http://bmjopen.bmj.com/

Reply – added in ‘and two further affected trial reports were marked as retracted on PubMed but not on Medline.[11,16]’

I have a handful of very minor comments regarding wordings: Page 6 line 27: consider moving "between 1997 and 2011" after "published" for clarity (While few would misunderstand this, there is a plausible misreading, because impact factors are associated with

Reply – thank you, this change has been made.

Page 9 line 15: I'd suggest revising the first sentence. This is difficult to read, and much more difficult to read than the deleted sentence at the end of the paragraph. Perhaps something like: "The results from one affected trial report [10] changed the findings for a combined treatment (calcium, vitamin D and vitamin K for falls prevention)." Or "The findings about the efficacy of a falls prevention treatment (calcium, vitamin D and vitamin K given together) changed, based on the results from one affected trial report [10].

Thank you – we have replaced the first sentence with ‘The results from one affected trial report [10] changed the findings for a combined treatment (calcium, vitamin D and vitamin K for falls prevention).’

Page 13 line 47: Maybe missing a word here: "We only assessed the impact of 12 likely most influential affected trial reports..." (Perhaps "the 12"?)

Reply – changed to ‘the 12 likely….’

BMJ Open: first published as 10.1136/bmjopen-2019-031909 on 30 October 2019. Downloaded from

Just an FYI: For future reference (not necessary to cite here, but related to the Mott [83] that you cite and to [90]), there is econometrics work on the impact of retraction on a field (they used PubMed's "related articles" as a proxy for "field"): Azoulay P, Furman JL, Krieger JL, Murray F. Retractions. Review of Economics and Statistics. 2015 Dec 1;97(5):1118-36. From their abstract: "Following retraction and relative to carefully selected controls, related articles experience a lasting 5% to 10% decline in the rate of citations received. This penalty is more severe when the retracted article involves fraud or misconduct rather than honest mistakes.")

Reply – thank you for this interesting paper, with interesting information on grant funding – something we’d like to do work on in due course.

Looking forward to including this in a review I'm working on! Your paper really fills a gap in the literature!

Reviewer: 1 Reviewer Name: David Robert Robert Grimes

Institution and Country: Dublin City University, Ireland University of Oxford, UK

Please state any competing interests or state ‘None declared’: None declared

Please leave your comments for the authors below. The authors have fully addressed by questions. I would love to see a follow up on this in future, perhaps 5 years from now to see how much these dubious articles have propagated in terms of citations - and whether those citations, being a blunt instrument, note the papers were retracted or not.

Reply – thank you for your comment. Rest assured, we are continuing to undertake research work in this area.

Open access Correction

Correction: An investigation into the impact and implications of published papers from retracted research: systematic search of affected literature

Avenell A, Stewart F, Grey A, et al. An investigation into the impact and implications of published papers from retracted research: systematic search of affected literature. BMJ Open 2019;9:e031909. doi: 10.1136/bmjopen-2019-031909

This article has been corrected since it was first published online. Details of retractions and correspondence relating to some references were inadvertently omitted in the first version; the following references have since been corrected to:

[6]Sato Y, Maruoka H, Oizumi K. Amelioration of hemiplegia-associated osteopenia more than 4 years after stroke by 1 alpha-hydroxyvitamin D3 and calcium supplementation. Stroke 1997;28:736–9. [Retraction in Sato Y, Maruoka H, Oizumi K, et al. Stroke 2019 in press. DOI: 10.1161/01.STR.28.4.736]. [7]Sato Y, Honda Y, Kuno H, et al. Menatetrenone ameliorates osteopenia in disuse- affected limbs of vitamin D- and K-deficient stroke patients. Bone 1998;23:291–6. [Retraction in: Sato Y, Honda Y, Kuno H, et al. Bone 2019;124:167]. [8]Sato Y, Manabe S, Kuno H, et al. Amelioration of osteopenia and hypovitaminosis D by 1alpha- hydroxyvitamin D3 in elderly patients with Parkinson's disease. J Neurol Neurosurg Psychiatry 1999;66:64–8. [Retraction in Sato Y, Manabe S, Kuno H, et al. J Neurol Neurosurg Psychiatry 2018;89:E5]. [9]Sato Y, Honda Y, Kaji M, et al. Amelioration of osteoporosis by menatetrenone in elderly female Parkinson's disease patients with vitamin D deficiency. Bone 2002;31:114– 8. [Retraction in: Sato Y, Honda Y, Kaji M, et al. Bone 2018;106:212]. [10]Sato Y, Kanoko T, Satoh K, et al. Menatetrenone and vitamin D2 with calcium supplements prevent nonvertebral fracture in elderly women with Alzheimer's disease. Bone 2005;36:61–8. [Retraction in: Sato Y, Kanoko T, Satoh K, et al. Bone 2018;106:213]. [11]Sato Y, Iwamoto J, Kanoko T, et al. Risedronate therapy for prevention of hip fracture after stroke in elderly women. Neurology 2005;64:811–6. [Retraction in Sato Y, Iwamoto J, Kanoko T, et al. Neurology 2016;87:239]. [12]Sato Y, Iwamoto J, Kanoko T, et al. Risedronate sodium therapy for prevention of hip fracture in men 65 years or older after stroke. Arch Intern Med 2005;165:1743–8. [Retraction in: Sato Y, Iwamoto J, Kanoko T, et al. JAMA Intern Med 2016;176:1256]. [13]Sato Y, Kanoko T, Satoh K, et al. The prevention of hip fracture with risedronate and ergocalciferol plus calcium supplementation in elderly women with Alzheimer disease: a randomized controlled trial. Arch Intern Med 2005;165:1737–42. [Retraction in: Sato Y, Kanoko T, Satoh K, et al. JAMA Intern Med 2016;176:1256]. [14]Sato Y, Honda Y, Iwamoto J, et al. Effect of folate and mecobalamin on hip fractures in patients with stroke: a randomized controlled trial. JAMA 2005;293:1082–8. [Retraction in: Sato Y, Honda Y, Iwamoto J, et al. JAMA 2016;315:2405]. [15]Sato Y, Iwamoto J, Kanoko T, et al. Alendronate and vitamin D2 for prevention of hip fracture in Parkinson's disease: a randomized controlled trial. Mov Disord 2006;21:924–9. [Retraction in: Sato Y, Iwamoto J, Kanoko T, et al. Mov Disord;31:1077]. [16]Sato Y, Honda Y, Iwamoto J. Risedronate and ergocalciferol prevent hip fracture in elderly men with Parkinson disease. Neurology 2007;68:911–5. [Retraction in: Sato Y, Honda Y, Iwamoto J. Neurology 2016;87:239]. [17]Sato Y, Iwamoto J, Honda Y. Once-weekly risedronate for prevention of hip fracture in women with Parkinson's disease: a randomised controlled trial. J Neurol Neurosurg Psychiatry 2011;82:1390-3. [Retraction in: Sato Y, Iwamoto, Honda Y. J Neurol Neurosurg Psychiatry 2018;89:E4]. [85]Clarke CE. Efficacy of methyprednisolone pulse therapy on neuroleptic malig- nant syndrome in Parkinson’s disease. J Neurol Neurosurg Psychiatry 2004;75:510–11; author reply 511.

BMJ Open 2019;9:e031909corr1. doi:10.1136/bmjopen-2019-031909corr1 1 Open access

[86]Poole KE, Warburton EA, Reeve J. Risedronate therapy for prevention of hip fracture after stroke in elderly women. Neurology 2005;65:1513–14; author reply 1513–4. [87]Caffrey N, Casey M, Walsh B. Substandard studies or substandard “standard-of- care.” J Bone Miner Res 2006; 21:491; author reply 492. [88]Halbekath JM, Schenk S, von Maxen A, et al. Risedronate for the prevention of hip fractures: concern about validity of trials. Arch Intern Med 2007;167:513–14; author reply 514–515.

Open access This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/. © Author(s) (or their employer(s)) 2019. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.

BMJ Open 2019;9:e031909corr1. doi:10.1136/bmjopen-2019-031909corr1

2 BMJ Open 2019;9:e031909corr1. doi:10.1136/bmjopen-2019-031909corr1