A Multi-Disciplinary Perspective on Emergent and Future Innovations in Peer Review
Total Page:16
File Type:pdf, Size:1020Kb
F1000Research 2015 - DRAFT ARTICLE (PRE-SUBMISSION) A multi-disciplinary perspective on emergent and future innovations in peer review Jonathan P. Tennant*1, Jonathan M. Dugan2, Daniel Graziotin3, Damien C. Jacques4, Franc¸ois Waldner4, Daniel Mietchen5, Yehia Elkhatib6, Lauren B. Collister7, Christina K. Pikas8, Tom Crick9, Paola Masuzzo10, Anthony Caravaggi11, Devin R. Berg12, Kyle E. Niemeyer13, Tony Ross- Hellauer14, Sara Mannheimer15, Lillian Rigling16, Daniel S. Katz17, Bastian Greshake18, Josmel Pacheco-Mendoza19, Nazeefa Fatima20, Marta Poblet21, Marios Isaakidis22, Dasapta Erwin Irawan23,Sebastien´ Renaut24, Christopher R. Madan25, Lisa Matthias26, Jesper Nørgaard Kjær27, Daniel Paul O’Donnell28, Cameron Neylon29, Sarah Kearns30, Manojkumar Selvaraju31, and Julien Colomb32 [email protected]; Imperial College London, London, UK, ORCID: 0000-0001-7794-0218; ScienceOpen, Berlin, Ger- many (*corresponding author) 2Berkeley Institute for Data Science, University of California, Berkeley, CA USA, ORCID: 0000-0001-8525-6221 3Institute of Software Technology, University of Stuttgart, Stuttgart, Germany; ORCID: 0000-0002-9107-7681 4Earth and Life Institute, Universite´ catholique de Louvain, Louvain-la-Neuve, Belgium; ORCID: 0000-0002-9069-4143 and 0000-0002-5599-7456 5Data Science Institute, University of Virginia, United States of America; ORCID: 0000-0001-9488-1870 6School of Computing and Communications, Lancaster University, Lancaster, UK; ORCID: 0000-0003-4639-436X 7University Library System, University of Pittsburgh, Pittsburgh, PA, USA; ORCID: 0000-0001-5767-8486 8Johns Hopkins University Applied Physics Laboratory, Laurel, MD, USA; ORCID: 0000-0002-4455-8036 9Cardiff Metropolitan University, Cardiff, UK; ORCID: 0000-0001-5196-9389 10Department of Biochemistry, Ghent University, Ghent, Belgium, VIB-UGent Center for Medical Biotechnology, Ghent, Bel- gium; ORCID: 0000-0003-3699-1195 11School of Biological, Earth and Environmental Sciences, University College Cork, Distillery Field, N Mall, Cork, Ireland; ORCID: 0000-0002-1763-8970 12Engineering & Technology Department, University of Wisconsin-Stout, Menomonie, WI, USA; ORCID: 0000-0002-1193-3848 13School of Mechanical, Industrial, and Manufacturing Engineering, Oregon State University, Corvallis, OR, USA; ORCID: 0000-0003-4425-7097 14State and University Library, University of Goettingen, Goettingen, Germany; ORCID: 0000-0003-4470-7027 15Montana State University, Bozeman, Montana; ORCID: 0000-0002-1433-6782 16Western University Libraries, London, Ontario, Canada; ORCID: 0000-0002-9178-0422 17National Center for Supercomputing Applications & Department of Computer Science & Department of Electrical and Com- puter Engineering & School of Information Sciences, University of Illinois Urbana-Champaign, Urbana, Illinois; ORCID: 0000- 0001-5934-7525 18Institute of Cell Biology and Neuroscience, Goethe University Frankfurt, Frankfurt, Germany; ORCID: 0000-0002-9925-9623 19Universidad San Ignacio de Loyola, Lima, Per ´u;ORCID: 0000-0002-2251-8092 20Department of Biology, Faculty of Science, Lund University, Sweden; ORCID: 0000-0001-7791-4984 21Graduate School of Business and Law, RMIT University, Melbourne, Australia; ORCID: 0000-0002-0026989X 22Department of Computer Science, University College London, London, UK; ORCID: 0000-0001-8498-1309 23Department of Groundwater Engineering, Faculty of Earth Sciences and Technology, Institut Teknologi Bandung, Indonesia; ORCID: 0000-0002-1526-0863 24Departement´ de Sciences Biologiques, Institut de Recherche en Biologie Veg´ etale,´ Universite´ de Montreal,´ Montreal, QC, Canada; ORCID: 0000-0002-8111-5082 25School of Psychology, University of Nottingham, Nottingham, UK; ORCID: 0000-0003-3228-6501 26OpenAIRE, University of Gottingen,¨ Gottingen,¨ Germany; ORCID: 0000-0002-2612-2132 27Psychiatric Research Academy, Department of Affective Disorders, Aarhus University Hospital, Risskov, Denmark; ORCID: 0000-0001-9183-9861 28Department of English and Centre for the Study of Scholarly Communications, University of Lethbridge, Lethbridge AB, Canada; ORCID: 0000-0002-0127-4893 29Centre for Culture and Technology, Curtin University, Perth, Australia; ORCID: 0000-0002-0068-716X 30Department of Chemical Biology, University of Michigan, Ann Arbor, Michigan; ORCID: 0000-0001-5640-909X 31Integrated Gulf Biosystems, Riyadh, Saudi Arabia; Saudi Human Genome Program, King Abdulaziz City for Science and Technology (KACST), Riyadh, Saudi Arabia; ORCID: 0000-0001-6439-5652 32Independent scholar, ORCID: 0000-0002-3127-5520 Page 1 of 48 F1000Research 2015 - DRAFT ARTICLE (PRE-SUBMISSION) Abstract Peer review of research articles is a core part of our scholarly commu- nication system. In spite of its importance, the status and purpose of peer review is often contested. What is its role in our modern digital research and communica- tions infrastructure? Does it perform to the high standards with which it is generally regarded? Studies of peer review have shown that it is prone to bias and abuse in numerous dimensions, frequently unreliable, and can fail to detect even fraudulent research. With the advent of web technologies, we are now witnessing a phase of innovation and experimentation in our approaches to peer review. These devel- opments prompted us to examine emerging models of peer review from a range of disciplines and venues, and to ask how they might address some of the issues with our current systems of peer review. We examine the functionality of a range of social Web platforms, and compare these with the traits underlying a viable peer review system: quality control, quantified performance metrics as engagement in- centives, and certification and reputation. Ideally, any new systems will demonstrate that they out-perform and reduce the biases of existing models as much as possi- ble. We conclude that there is considerable scope for new peer review initiatives to be developed, each with their own potential issues and advantages. We also propose a novel hybrid platform model that could, at least partially, resolve many of the socio-technical issues associated with peer review, and potentially disrupt the entire scholarly communication system. Success for any such development relies on reaching a critical threshold of research community engagement with both the process and the platform, and therefore cannot be achieved without a significant change of incentives in research environments. Keywords: Open Peer Review, Social Media, Web 2.0, Open Science, Scholarly Publishing, Incentives, Quality Control. Page 2 of 48 F1000Research 2015 - DRAFT ARTICLE (PRE-SUBMISSION) 1 Introduction disseminate research content rapidly and at relatively negligible cost (Moore et al., 2017). This has led to Peer review is a core part of our self-regulating global the innovation and increasing popularity of digital-only scholarship system. It defines the process in which publication venues that vet submissions based exclusively professional experts (peers) are invited to critically assess on the soundness of the research, often termed “mega- the quality, novelty, theoretical and empirical validity, journals” (e.g., PLOS ONE, PeerJ, the Frontiers series). and potential impact of research by others, typically Such a flexibility in the filter function of peer review while it is in the form of a manuscript for an article, reduces, but does not eliminate, the role of peer review conference, or book (Zuckerman and Merton, 1971; as a selective gatekeeper, and can be considered to be Kronick, 1990; Daniel, 1993; Spier, 2002). For the “impact neutral.” Due to such digital experimentations, purposes of this article, we are exclusively addressing ongoing discussions about peer review are intimately peer review in the context of manuscript selection for linked with contemporaneous developments in Open scientific research articles, with some initial consider- Access (OA) publishing and to broader changes in open ations of other outputs such as software and data. In scholarship (Tennant et al., 2016). this form, peer review is becoming increasingly central as a principle of mutual control in the development The goal of this article is to investigate the historical evo- of scholarly communities that are adapting to digital, lution in the theory and application of peer review in a information-rich, publishing-driven research ecosystems. socio-technological context. We use this as the basis to Consequently, peer review is a vital component at the core consider how specific traits of consumer social Web plat- of research communication processes, with repercussions forms can be combined to create an optimized hybrid for the very structure of academia, which largely operates peer review model that we suggest will be more efficient, through a peer reviewed publication-based reward and democratic, and accountable than existing processes. incentive system (Moore et al., 2017). Different forms of peer review beyond that for manuscripts are also clearly important and used in other contexts such as academic 1.0.1 Methods appointments, measurement time, research ethics or This article provides a general review of conventional research grants (see, e.g., Fitzpatrick, 2011b, p. 16), but journal article peer review and evaluation of recent and a holistic discussion of all forms of peer review is beyond current innovations in the field. It is not a systematic re- the scope of the present article. view or meta-analysis of the empirical literature. Rather, a team of researchers with diverse expertise