Cobey et al. BMC Medicine (2020) 18:88 https://doi.org/10.1186/s12916-020-01550-9 RESEARCH ARTICLE Open Access Stress testing journals: a quasi-experimental study of rejection rates of a previously published paper Kelly D. Cobey1,2*, Danielle B. Rice1,3, Manoj M. Lalu1,4,5, Daniel Abramowitz6, Nadera Ahmadzai1, Heather Cunningham7, Ana Patricia Ayala7, Hana Raffoul8, Faizan Khan2, Larissa Shamseer2 and David Moher1,2 Abstract Background: When a journal receives a duplicate publication, the ability to identify the submitted work as previously published, and reject it, is an assay to publication ethics best practices. The aim of this study was to evaluate how three different types of journals, namely open access (OA) journals, subscription-based journals, and presumed predatory journals, responded to receiving a previously published manuscript for review. Methods: We performed a quasi-experimental study in which we submitted a previously published article to a random sample of 602 biomedical journals, roughly 200 journals from each journal type sampled: OA journals, subscription-based journals, and presumed predatory journals. Three hundred and three journals received a Word version in manuscript format, while 299 journals received the formatted publisher’s PDF version of the published article. We then recorded responses to the submission received after approximately 1 month. Responses were reviewed, extracted, and coded in duplicate. Our primary outcome was the rate of rejection of the two types of submitted articles (PDF vs Word) within our three journal types. Results: We received correspondence back from 308 (51.1%) journals within our study timeline (32 days); (N =46 predatory journals, N = 127 OA journals, N = 135 subscription-based journals). Of the journals that responded, 153 received the Word version of the paper, while 155 received the PDF version. Four journals (1.3%) accepted our paper, 291 (94.5%) journals rejected the paper, and 13 (4.2%) requested a revision. A chi-square test looking at journal type, and submission type, was significant (χ2 (4) = 23.50, p < 0.001). All four responses to accept our article came from presumed predatory journals, 3 of which received the Word format and 1 that received the PDF format. Less than half of journals that rejected our submissions did so because they identified ethical issues such as plagiarism with the manuscript (133 (45.7%)). Conclusion: Few journals accepted our submitted paper. However, our findings suggest that all three types of journals may not have adequate safeguards in place to recognize and act on plagiarism or duplicate submissions. Keywords: Scholarly publishing models, Open access, Predatory journals, Plagiarism * Correspondence: [email protected] 1Centre for Journalology, Clinical Epidemiology Program, The Ottawa Hospital Research Institute, Ottawa, Canada 2School of Epidemiology and Public Health, Faculty of Medicine, University of Ottawa, Ottawa, Canada Full list of author information is available at the end of the article © The Author(s). 2020 Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data. Cobey et al. BMC Medicine (2020) 18:88 Page 2 of 9 Background OMICS, a large and well-known predatory publisher and Several spoof papers submitted to “predatory” journals predatory conference organizer, was ordered by a federal have highlighted the self-interest these journals have: they judge in 2019 to pay a fine of more than 50 million dollars seem to publish anything if authors are willing to pay an to resolve Federal Trade Commission charges in the USA, article processing fee. When spoof papers get accepted at which found them guilty of making deceptive claims about predatory journals, they tend to attract a great deal of at- their operations and fee structure [9]. It is unclear if these tention. Whether it is a Seinfeld-themed “case report” [1], actions have influenced predatory journals, or if predatory or a bogus Star Wars themed paper with movie quotes [2], journals have adapted during this timeframe. these papers make us laugh, but then encourage reflection Here, we describe the results of a quasi-experimental on this darker side of publishing. They force us to acknow- study in which we submitted a previously published article ledge the potential threat predatory journals play in de- to more than 600 journals from three journal groups: grading the integrity of the open access publishing model presumed predatory journals, open access journals, and and their impact on the scholarly literature more broadly. subscription-based journals. A previous study has found a Scientifically designed studies, which systematically substantial level of plagiarism within presumed predatory evaluate predatory journals, are rare. Submitting a single nursing journals [10]. We submitted a Microsoft Word low-quality paper to a journal, or a handful of journals, version of the article to approximately half of these jour- is relatively feasible to do. This however says little about nals, and the publishers formatted PDF version to the the scale of the problem and does not allow us to track other half. Our primary outcome was the rate of rejection how predatory journals and legitimate journals have of the previously published article at each of the journal changed over time. The results of these spoof papers do types. We predicted that articles submitted to predatory inform us about the stability of particular journals to with- journals would be rejected at a lower rate than articles stand worrisome publication practices. We can conceive of submitted to open access or subscription-based journals, such studies as “stress tests” similar to what banks were re- but that no journal type would be immune from accepting quired to withstand following the 2008 financial banking the article. As a secondary outcome, we looked at the type crisis. The first investigation to examine acceptance rates of editorial and peer review within the three journal types. of spoof articles at a large group of journals was conducted We predicted that more of the subscription-based and in 2013, by journalist John Bohannon, and published by open access journals, as compared to predatory journals, Science Magazine [3]. Bohannon submitted articles with would immediately detect the paper as being plagiarized obvious methodological and reporting issues, and made- and reject it without sending it for peer review. We pre- up author and university names, to several presumed dicted that differences in rejection rates would be greatest predatory journals and to several open access journals. In when considering the PDF article submission as compared total he submitted versions of his flawed paper to 304 jour- to the Word article submission. nals, 157 of which accepted his paper. Most journals that accepted his article were presumed to be predatory Methods (69.42%; 84/121); however, many open access journals that Ethics, consent, and permissions were presumed not to be predatory also accepted the This study, which included deliberate deception and subse- paper (38.32%; 64/167). Bohannon’s sting brought atten- quent debriefing, received ethical approval from the Ottawa tion to the problem of predatory journals, but it also Health Science Network Research Ethics Board (REB #: highlighted potential concerns with open access publishing 20180266-01H). We developed a study protocol and and peer review more generally. The DOAJ (Directory of uploaded it to the Open Science Framework [11]priorto Open Access Journals) [4], which curates a list of legiti- initiating data collection (See: https://osf.io/4ngk3/). Given mate open access journals, required all their journals to the use of deception, participants (journal editorial team reapply for indexing as a consequence of this finding. members) did not provide consent. It has been more than 5 years since these data were collected. Bohannon, who is a journalist, did not design Journal sampling his study scientifically and therefore it had elements of We sought to sample 200 journals from each of the three bias in journal selection, in journal communication, and journal types, as described below. All journal titles selected with lack of transparency in reporting. for the sample were collated into an Excel file along with No recent study auditing a broad range of journals with their respective URL. We checked for duplication of jour- different publication models has been rigorously con- nals across our three randomly selected samples. We ex- ducted. The number of predatory journals is growing