Publish Peer Reviews Jessica K

COMMENT HISTORY Seventy-five years TECHNOLOGY Why are PUBLISHING Preprints — what is OBITUARY Burton Richter, since physicist Schrödinger governments in favour of good for science is good for charm-quark Nobel laureate, made waves in biology p.548 digital insecurity? p.550 the public p.553 remembered p.554 ILLUSTRATION BY DAVID PARKINS DAVID BY ILLUSTRATION

Publish peer reviews Jessica K. Polka and colleagues call on journals to sign a pledge to make reviewers’ anonymous comments part of the official scientific record.

ong shrouded in secrecy, the contents (the latter is owned by Springer Nature) have in Chevy Chase, Maryland — and ASAPbio, of peer review are coming into the been doing so for even longer (see ‘Revealing a non-profit organization that encourages open. In the past decade, outlets peer review’). Last year, the organizers of innovation in life-sciences publishing. We Lsuch as eLife, F1000Research, Royal Society Peer Review Week embraced the topic in a are convinced that publishing referee reports Open Science, Annals of Anatomy, Nature broader discussion of transparency. would better inform authors and readers, Communications, PeerJ and EMBO Press We are representatives of two biomedical improve review practices and boost trust in have begun to publish referee reports. Pub- funders — the UK Wellcome Trust and the science. Right now, less than 3% of scientific lishers including Copernicus, BMJ and BMC Howard Hughes Medical Institute (HHMI) journals allow peer reviews to be

COMMENT

published (see go.nature.com/2weh6vn). example. And the public would not be asked CAPITALSREVEALING OF To increase these numbers, our organiza- to place its faith in hidden assessments. tions held a meeting in February this year Studies of published peer reviews are PEERCULTIVATION REVIEW of around 90 invitees from the life sciences, small and often also involve open identi- predominantly from North America and ties or other innovations, making effects Journal editors have long consulted Europe. Scientific authors, reviewers and hard to ascertain. Nonetheless, evidence so referees to select and improve papers. readers participated, along with journal far suggests that the scientific community e focus has shifted to sharing them. editors and leaders of granting agencies. We finds published reports valuable. At The took care to include conservative voices, but EMBO Journal, peer-review files receive the nature of the meeting attracted people about 10% of the hits the papers themselves 1700 ready for change. The ideas in this article do4. A pilot by the publisher Elsevier found were honed at that event, with later assistance that one-third of its website visitors accessed 1750s: The UK Royal Society establishes a committee to vote on from HHMI president Erin O’Shea; molecu- peer-review reports, and several editors 1800 what is published in its journal, lar biologist Needhi Bhalla at the University said they used published reports as instruc- Philosophical Transactions. of California, Santa Cruz; Kenneth Gibbs, tive examples for inexperienced reviewers director of postgraduate training at the (see go.nature.com/2oujfgv). Editors at the US National Institute of General Medical European Journal of Neuroscience, which 1890s: UK scientic societies debate 1900 and abandon the adoption of a Sciences; and researcher Tony Ross-Hellauer launched transparent review at the end of standardized referee system to curb at Know-Center in Graz, Austria. 2016, report that referees are writing better “veritable sewage thrown into the Attendees agreed that the current lack of reviews and returning them more promptly pure stream of science”. 1940 transparency around peer review does not (see go.nature.com/2oxgtyf). serve science, and several journals com- mitted to publishing reviews (although not BARRIERS, PERCEIVED AND REAL necessarily reviewers’ identities) and author So why is the practice still rare? There are rebuttals. Here, we invite more journals to several reasons — some inertial, some 1940s–1960s: Formal peer take up the cause. It is time for transparency conceptual. 1950 review comes to be considered the linchpin of science. Science, to become the norm. Some disciplines are more keen than Nature and the Journal of the others. Nature Communications found that, American Medical Association take DEFINING REVIEW up the practice. given a choice, authors (and reviewers) of The term ‘open review’ has many more than 70% of its evolution and ecology interpretations. ‘Open identities’ means submissions opted for published reports. 1960 disclosing reviewers’ names; ‘open reports’ The figure was less than 50% for submissions (also called transparent reviews or published in atomic, particle and theoretical physics5. 1970s: The term peer review peer review) means publishing the content One concern is that, even if public reviews becomes widely used. of reviews. Journals might offer one or the are anonymous, they might make reviewers other, neither or both1. reluctant to accept assignments or to criticize In a 2016 survey2, 59% of 3,062 respond- freely, because authors could resent criticism 1970 ents were in favour of open reports. Only and retaliate against their presumed review- 31% favoured open identities, which they ers. The BMJ found that publishing peer- feared could cause reviewers to weaken their review reports with reviewers’ names did not criticisms or could lead to retaliation from change the quality of the peer reviews, sug- authors. Here, we advocate for open reports gesting that reviewers were not intimidated6.

1980 as the default and for open identities to be What is more, authors read unsigned review- optional, not mandatory. ers’ reports during standard review anyway. WWW.PEERREVIEWCONGRESS.ORG The vast majority of scientists think that A bigger concern is that published reviews peer review is essential for vetting research might be used unfairly in subsequent evalu- papers3. The process gives authors con- ation of the authors for grants, jobs, awards structive feedback, offers editors insight or promotions. There are few data about 1990 and assures readers of the trustworthiness whether and how authors’ ethnicity, gender, 1989: Inaugural Peer Review of research. Generally, however, only edi- country of origin or institution affect the Congress organized to evaluate the process. It is held every four tors, authors and (sometimes) reviewers see evaluations of papers. Yet there is evidence years. referee reports. That enables several forms of of bias in scientific publishing. Women, abuse: referees might be superficial, rude or for example, are less likely to be first or last biased; authors might respond inadequately authors in high-profile journals, and are 2000 1999–2003: The BMJ decides to 7,8 disclose reviewers’ names after to reasonable criticism; editors might not less likely to be asked for peer reviews assessing e ects in a randomized hold authors or reviewers to account; and (see also go.nature.com/2pzyvcw). And trial. The publisher BMC begins predatory publishers will charge fees without workplace evaluations of female profes- publishing signed reviewer reports. Atmospheric Chemistry and Physics providing quality review. sionals also show gender bias (see go.nature. promotes open discussion of Many benefits would accrue from pub- com/2ppat2k). So the concern is that indi- 2010 submissions. lishing peer reviews (see ‘Potential benefits viduals from under-represented minorities 2006–16: Several journals and of published review’). The scientific commu- could receive biased reviews. Assessors for platforms start publishing reviewer nity would learn from reviewers’ and editors’ funding, hiring and promotions could pay comments. They include Biology insights. Social scientists could collect data more attention to negative comments when Direct (2006), eLife (2011), The EMBO Journal (2010), F1000 Research (for example, on biases among reviewers authors are from under-represented groups (2012), PeerJ (2013) and Nature or the efficiency of error identification by or less-prestigious institutions. Some fields 2020 Communications (2016). reviewers) that might improve the process. are also more critical or competitive, which Early-career researchers could learn by might skew reviews.

COMMENT

Making referee reports open could allow peer-review materials can take approximately manuscript-management platforms should more-effective research into how competi- 25 minutes per manuscript. This is obviously develop workflows that optimize and auto- tion and bias affect the process. Meanwhile, much less than the time spent coordinating, mate the process of publishing peer reviews, anyone participating in open peer review — conducting and assessing reviews, but is still reducing burdens on journal staff, authors or evaluating it after the fact — should be significant. Most publishing platforms are and reviewers. Indexing services, such as aware of this potential for unfairness. not set up to display, organize or assign digi- PubMed, should find ways to prominently Another risk is the ‘weaponization’ of tal object identifiers link peer reviews to the original paper. reviewer reports. Opponents of certain “It is time for (DOIs) to reviewer Appropriate infrastructure is already being types of research (for example, on geneti- transparency reports and related built: CrossRef began assigning DOIs for cally modified organisms, climate change to become materials, and peer reviews in late 2017, and reviewer and vaccines) could take critical remarks in the norm.” making changes to reports (with or without reviewer identi- peer reviews out of context or mischaracter- such systems can be ties) can be archived in PubMed Central ize disagreements to undermine public trust onerous. Still, we expect that journals could and Europe PubMed Central. in the paper, the field or science as a whole. streamline these tasks and, potentially, build For robust systems to develop, however, Queries to eLife, The BMJ and EMBO Press in transparent costs for dealing with extra published reviews must become more com- about this problem revealed only one, mild work. Editors will have to learn to handle mon. Even if today’s implementations are example (see go.nature.com/2piygkb). But reviews containing inappropriate material, less than ideal, they will drive demand and weaponization could be a greater concern such as libellous comments or unpublished pave the way for better iterations. for journals that publish work that is more results, and to become comfortable with More than 20 editors and publishers likely to be politicized. making some correspondence public. Many representing more than 100 journals One precaution would be to add a dis- have already done so. have already signed an open letter (see claimer explaining the peer-review process http://asapbio.org/letter) to show that they and its role in scientific discussion. Open- MOVING FORWARD have begun to publish peer reviews, with or ing up materials and establishing dialogues We think that the value of published review without reviewers’ identities, or that they plan with journalists, politicians and the public reports to referees, authors, the public and to. We invite others to join. (Nature Commu- is an opportunity to build trust and enhance editors far outweighs the risks and toil. In an nications has signed this pledge. Other Nature understanding of the scientific process. ideal world, all published papers would be journals are considering doing so.) Published peer-review reports could accompanied by the contents of their peer- Scientists can stimulate this change by also place editorial decisions under greater review reports. For now, we recommend that requesting that the journals for which they scrutiny and perhaps make editors more the practice is encouraged while the scien- write, review and edit are open about their timid about overriding critical reviews (see tific community assesses whether and how peer-review process. Funders and academic go.nature.com/2bid8ag). Equally, published author characteristics, such as ethnicity and societies could also help to shift attitudes, reports could boost appreciation for the role country of origin, influence reviewer feed- particularly if they implement or pilot pub- of editors in synthesizing and prioritizing back. Any structural barriers to equality lished peer reviews in their society journals. diverse reviewer opinions. Editorial judge- must be eliminated. Science moves forward through criticism ments have huge influence on our percep- Ideally, reviewer reports will be easy to and disagreement. Exposing this inherent tions of scientific quality, and as such are find, and will be organized and archived process, although uncomfortable for some, valuable works of scholarship. Publishing intuitively with related materials (such is a healthy step for science. ■ decision letters, along with author responses, as rebuttals and manuscript versions). could contextualize and rebut criticism, Publishers should provide ways to rec- Jessica K. Polka is executive director of correct misunderstandings and provide ognize reviewers for their contributions. ASAPbio in San Francisco, California, information that does not make it into the Technological innovation could reduce USA. Robert Kiley is head of open research final paper. administrative burdens, tackle informa- at the Wellcome Trust in London. Boyana Finally, there are pragmatic concerns. tion overload and provide additional Konforti is director of scientific strategy Editors report that manually posting links and context for readers. Vendors of and development, and Bodo Stern is chief development and strategy officer at the Howard Hughes Medical Institute in Chevy TRANSPARENT CRITIQUE Chase, Maryland, USA. Ronald D. Vale is president of ASAPbio in San Francisco, Potential benefits of published review California, USA; a professor of cellular and molecular pharmacology at the University ●●Encourages good-quality, constructive reviewers, to make their decisions. of California, San Francisco; and an HHMI comments. The expectation that reviews Published peer review provides a window investigator. will be published will encourage editors and on the process. e-mails: [email protected]; reviewers to hold them to a high standard. ●●Creates a pathway for crediting [email protected] ●●Preserves useful scholarship. Peer reviewing. Reviewers can point (even 1. Ross-Hellauer, T. F1000Research 6, 588 (2017). reviews contain arguments and ideas that privately) to their work as evidence 2. Ross-Hellauer, T., Deppe, A. & Schmidt, B. PLoS can reveal how thinking in a field evolves. of scholarly activity for grants and ONE 12, e0189311 (2017). This material should be preserved and promotions. 3. Publishing Research Consortium. PRC Peer Review Survey 2015 (Mark Ware Consulting, made available to others. ●●Provides a resource for training. Reports 2016). ●●Builds trust. Readers have a right to can show people how to (and how not to) 4. Pulverer, B. EMBO J. 29, 3891–3892 (2010). understand the level of scrutiny that a assess a paper. 5. Nature Commun. 7, 13626 (2016). 6. van Rooyen, S., Delamothe, T. & Evans, S. J. W. paper has undergone. ●●Bolsters systemic study of peer review. Br. Med. J. 341, c5729 (2010). ●●Makes journal decisions more Published reports and rebuttals enable 7. Tregenza, T. Trends Ecol. Evol. 17, 349–350 transparent. Editors must integrate more research on best practices, leading to (2002). 8. Shen, Y. A., Webster, J. M., Shoda, Y. & information from diverse sources, including improvements in the system as a whole. Fine, I. Preprint at bioRxiv https://doi. org/10.1101/275362 (2018).