Peer Review in Academic Promotion and Publishing: Its Meaning, Locus, and Future A Project Report and Associated Recommendations, Proceedings from a Meeting, and Background Papers Diane Harley and Sophia Krzys Acord with contributions from Sarah Earl-Novell, Shannon Lawrence, and Elise Herrala March 2011 The Future of Scholarly Communication Project University of California, Berkeley http://cshe.berkeley.edu/research/scholarlycommunication © 2005–2011, Center for Studies in Higher Education, UC Berkeley Center for Studies in Higher Education 771 Evans Hall #4650 University of California Berkeley CA 94720-4650 (510) 642-5040 http://cshe.berkeley.edu Citation: Harley, Diane, and Sophia Krzys Acord. 2011. Peer Review in Academic Promotion and Publishing: Its Meaning, Locus, and Future. UC Berkeley: Center for Studies in Higher Education. http://escholarship.org/uc/item/1xv148c8 ACKNOWLEDGEMENTS We would like to thank the Andrew W. Mellon Foundation and all of the workshop participants for their generous support and contributions to this project, including reviewing various iterations of the report. Throughout, we borrowed liberally from the many ideas presented by attendees at both meetings. Their ideas are central to this report, including the background documents. We would also like to thank a number of people who made this project possible, including: Shannon Lawrence and Emily Hilligoss for editorial support; Matthew Winfield at the California Digital Library for his personal help in publishing the report on e-scholarship; and Karen Weinbaum, Christina Herd, Meg Griffith, and Elisabeth Gordon for administrative assistance. HARLEY AND ACORD PEER REVIEW IN ACADEMIC PROMOTION AND PUBLISHING ABSTRACT Since 2005, and with generous support from the A.W. Mellon Foundation, The Future of Scholarly Communication Project at UC Berkeley's Center for Studies in Higher Education (CSHE) has been exploring how academic values—including those related to peer review, publishing, sharing, and collaboration—influence scholarly communication practices and engagement with new technological affordances, open access publishing, and the public good. The current phase of the project focuses on peer review in the Academy; this deeper look at peer review is a natural extension of our findings in Assessing the Future Landscape of Scholarly Communication: An Exploration of Faculty Values and Needs in Seven Disciplines (Harley et al. 2010), which stressed the need for a more nuanced academic reward system that is less dependent on citation metrics, the slavish adherence to marquee journals and university presses, and the growing tendency of institutions to outsource assessment of scholarship to such proxies as default promotion criteria. This investigation is made urgent by a host of new challenges facing institutional peer review, such as assessing interdisciplinary scholarship, hybrid disciplines, the development of new online forms of edition making and collaborative curation for community resource use, heavily computational subdisciplines, large-scale collaborations around grand challenge questions, an increase in multiple authorship, a growing flood of low-quality publications, and the call by governments, funding bodies, universities, and individuals for the open access publication of taxpayer-subsidized research, including original data sets. The challenges of assessing the current and future state of peer review are exacerbated by pressing questions of how the significant costs of high-quality scholarly publishing can be borne in the face of calls for alternative, usually university-based and open access, publishing models for both journals and books. There is additionally the insidious and destructive “trickle down” of tenure and promotion requirements from elite research universities to less competitive and non- research-intensive institutions. The entire system is further stressed by the mounting—and often unrealistic—government pressure on scholars in developed and emerging economies alike to publish their research in the most select peer-reviewed outlets, ostensibly to determine the distribution of government funds (via research assessment exercises) and/or to meet national imperatives to achieve research distinction internationally. The global effect is a growing glut of low-quality publications that strains the efficient and effective practice of peer review, a practice that is, itself, primarily subsidized by universities in the form of faculty salaries. Library budgets and preservation services for this expansion of peer-reviewed publication have run out. Faculty time spent on peer review, in all of its guises, is being exhausted. As part of our ongoing research, CSHE hosted two meetings to address the relationship between peer review in publication and that carried out for tenure and promotion. Our discussions included: The Dominant System of Peer Review: Types, Standards, Uses, Abuses, and Costs; A Very Tangled Web: Alternatives to the Current System of Peer Review; Creating New Models: The Role of Societies, Presses, Libraries, Information Technology Organizations, Commercial Publishers, and Other Stakeholders; and Open Access “Mandates” and Resolutions versus Developing New Models. This report includes (1) an overview of the state of peer review in the Academy at large, (2) a set of recommendations for moving forward, (3) a proposed research agenda to examine in depth the effects of academic status-seeking on the entire academic enterprise, (4) proceedings from the workshop on the four topics noted above, and (5) four substantial and broadly conceived background papers on the workshop topics, with associated literature reviews. The http://cshe.berkeley.edu/research/scholarlycommunication ABSTRACT ⎪ i HARLEY AND ACORD PEER REVIEW IN ACADEMIC PROMOTION AND PUBLISHING document explores, in particular, the tightly intertwined phenomena of peer review in publication and academic promotion, the values and associated costs to the Academy of the current system, experimental forms of peer review in various disciplinary areas, the effects of scholarly practices on the publishing system, and the possibilities and real costs of creating alternative loci for peer review and publishing that link scholarly societies, libraries, institutional repositories, and university presses. We also explore the motivations and ingredients of successful open access resolutions that are directed at peer-reviewed article-length material. In doing so, this report suggests that creating a wider array of institutionally acceptable and cost- effective alternatives to peer reviewing and publishing scholarly work could maintain the quality of academic peer review, support greater research productivity, reduce the explosive growth of low-quality publications, increase the purchasing power of cash-strapped libraries, better support the free flow and preservation of ideas, and relieve the burden on overtaxed faculty of conducting too much peer review. http://cshe.berkeley.edu/research/scholarlycommunication ABSTRACT ⎪ ii HARLEY AND ACORD PEER REVIEW IN ACADEMIC PROMOTION AND PUBLISHING TABLE OF CONTENTS PEER REVIEW IN ACADEMIC PROMOTION AND PUBLISHING: OVERVIEW AND BACKGROUND ..................... 1 What Do We Mean by Peer Review?...................................................................................................................... 1 Road Map to the Report and Methods .................................................................................................................... 5 RECOMMENDATIONS .................................................................................................................................................. 6 Some Suggestions for Further Research .............................................................................................................. 11 WORKSHOP PROCEEDINGS AND BACKGROUND PAPERS SESSION 1. The Dominant System of Peer Review: Types, Standards, Uses, Abuses, and Costs ................... 15 BACKGROUND PAPER 1. Peer Review in Academic Promotion and Publishing: Norms, Complaints, and Costs .................................................................................................................................................................... 21 The Value of Publication-based Peer Review ....................................................................................................... 22 Complaints about Publication-based Peer Review ............................................................................................... 23 The Costs of the Publication-based Peer-Review System and Who Bears Them ................................................ 27 SESSION 2. A Very Tangled Web: Alternatives to the Current System of Peer Review ...................................... 31 BACKGROUND PAPER 2. New Models of Peer Review: Repositories, Open Peer Review, and Post-Publication Metrics ........................................................................................................................................... 41 The Changing Landscape of Peer Review ............................................................................................................ 41 An Overview of Attempts to Reform the Existing Peer-Review System ................................................................ 41 Pre-Publication Peer Review ................................................................................................................................ 43 “Open” Peer-Review Experiments