Acceptance of Open Peer Review by Scientific Publishers

______Proceedings of the Central European Conference on Information and Intelligent Systems 221

Radovan Vrana Faculty of Humanities and Social Sciences, University of Zagreb Department of Information and Communication Sciences Ivana Lučića 3, Zagreb, Croatia [email protected]

Abstract. The paper consists of two parts. The first part with a new one. A recent attempt to made such an describes the importance and problems of the current improvement is introduction and experimental peer review process and characteristics of open peer implementation of open peer review. review. The second part provides results of the Broadly defined, open peer review is “a process in research study of the 100 top scientific publishers which the names of the authors and reviewers are about their acceptance and possible implementation of known to one another” (An Introduction to Open Peer the open peer review. Public documents about open Review, 2014). Furthermore, open peer review process peer review were sought on publishers’ Web sites and is open to experts conducting the review but also the research showed that there are very few of them potentially to anyone else willing to participate in the available. The research confirmed that open peer review process. Open peer review is part of open access review is still not widely accepted even at the major publishing initiative which supports “content free publishing houses. online to readers while supporting operations by financial models that permit this free electronic Keywords. Scientific publishing, peer review, open distribution” (Overview of Open Access Publishing). peer review, scientific publishers Open access publishing provides “an opportunity to those researchers whose work is commendable, but cannot pay high charges for publishing in the leading scientific journals” (Janodia, 2017). When mentioning 1 Introduction charges for publishing in open access scientific journals as obstacles for dissemination of scientific The paper is divided into two parts. The first part information for some scientists, one must have in mind presents a brief introduction and overview of the two types of scientific journals: a type which charges a current peer review process and describes fee called the article-processing charge (APC) which characteristics of open peer review. The second part of covers the publisher's costs and is paid by the author, the paper presents results from the research of the 100 research funder or sometimes by the author's institution top scientific publishers and their acceptance and (Overview of Open Access Publishing). Another type possible implementation of open peer review, one of refers to journals charging nothing at all and being the latest developments in scientific publishing. supported through sponsorship, advertising, are run Scientific publishing is one of the most important using voluntary labour or by selling subscriptions to activities because it transfers new research based the printed form of the journal (Overview of Open knowledge and innovation to the global scientific Access Publishing). More than half of open access community and the general population. Scientific journal publishers do not charge the APC (Overview of publications “are the end-products of the scientific Open Access Publishing). Regardless of the type, work, and their quantity and citability are keys to the scientific journals almost universally accept peer promotion of scientists” (Masic, 2012). The principal review as a basic mechanism for evaluation of article media for dissemination of information about scientific manuscripts submitted by scientists. Peer review is discoveries are still scientific journals. “For more than therefore, still a pillar of trust in science and remains to 350 years, journals have played a pivotal role in the be present in most scientific journals in the world. transmission of scholarly communications among academics and researchers” (Greco, 2016). To verify the quality of scientific information, scientific community has invented peer review. Peer review 2 Peer review process process as we know it today is burdened with problems and scientists in cooperation with scientific publishers Peer review is a “an official and legitimated seek new solutions which would significantly improve mechanism for evaluating and controlling scientific (or possibly replace) the current peer review system production” (Meruane, Gonzales and Pina-Stranger,

______28th CECIIS, September 27-29, 2017, Varaždin, Croatia ______222 Proceedings of the Central European Conference on Information and Intelligent Systems

2016), it is a cornerstone of scholarly communication mistake can be hardly ever corrected (Sandewall, and a guard against the publication of flawed research 2012). and assuring that research meets a standard of rigor Peer review employs reviewers who are experts in all (Wang, Wolfram, Hoyt, Ingwersen, Pöschl, Smith and aspects of a paper and thus can evaluate both its impact Bates, 2016) and it provides feedback to the author(s) and validity and can evaluate the paper prior to obtaining so as to improve the article (Sandewall, 2012). The first answers from the authors or other referees (Lee, 2012). peer review was done the in Medical Essays and Lee (2012) analysed peer review from various aspects and Observations published by the Royal Society of he pointed out, among many other things, that peer review Edinburgh in 1731 (Post-publication peer review). in which experts participate may create a big obstacle for However, some other authors like Nassi-Calò (2017) innovative papers because one negative opinion about the and Csiszar (2016) suggest that the first peer review paper could mean that a single review could determine the was done later, “in 1831 by William Whewell to the future of the paper. In contrast, the same author reminds that successfully published papers are evaluated by the Royal Society of London, suggesting that a committee whole scientific community and that evaluation is much of eminent scholars produced reports on articles more valuable than the single man’s opinion. submitted for publication to the Philosophical One of the most frequent problem for journal editors Transactions journal”. Since then, the term per review is a short list of reliable peer reviewers (How is has become a symbol of "an admirable aspiration to research publishing going to progress in the next 20 intellectual integrity, calm deliberation, and detached years?, 2014) because many researchers decline to evaluation in the accumulation of human knowledge" participate in peer review on the grounds of being too (Iannone, 2013). Today, peer review is accepted by busy (Lajtha and Baveye, 2010). most scientific journals worldwide (except in journals Some reviewers are motivated for participation in which state explicitly that they don’t go through peer the peer review process by self-interest protecting their review process). Since its first implementation, peer own work that submitted article manuscript is review process has gone through many changes. Some challenging; they can also be motivated by a desire to of those changes were not ideal and the current peer "look good" to the editor promoting overly critical review system is now burdened with problems well reviews; they can make cognitive errors and have documented in literature. Because of the problems, limited capacity to process information; they perform peer review is “currently facing a transition moment, unfair review (Cooper, 2009). and many believe that it is necessary to redefine its The referees can also fail to be sufficiently expert, principles and practices so as not to delay or hinder the informed, conscientious or fair (Harnad, 2000). progress of science” (Nassi-Calò, 2017). Reviewers are also criticized for their inability to discover fraud, plagiarism, repetitive publication and for being anti-innovative, non-constructive and 3 Peer review problems causing unacceptable delays in publishing (Lipworth, Kerridge, Carter and Little, 2011). Problems with peer review include slowness, journal Because of the enumerated problems with the peer editors not being pro-active in favour of controversial review process, academic community is searching for or challenging articles, light-touch peer review where new modes of improvement of the peer review process. authors want rigorous peer review, the variable quality One such attempt is open peer review. of reviewing, authors are sometimes misunderstood by the reviewers, a lack of transparency in reviewing, reviewers coming up with conflicting criticisms, open 4 Open peer review refereeing because it inhibits reviewers, authors proposing referees, authors suggesting their friends as The idea of open peer review is present in scientific reviewers (Watkinson et. al, 2015), editors not publishing since 1990s as an attempt to increase “the informing reviewers about the final status of the paper transparency of the article selection process for a after the review process (Lahiri, 2006), censorship, journal, and for obtaining a broader basis for feedback inhibiting or delaying a new work (Wheeler, 2011), to the authors and for the acceptance decision” expensive, wasting academic time, highly subjective, (Sandewall, 2012). In spite of the two decades of its something of a lottery, prone to bias, and easily abused existence, it is quite difficult to find a straightforward (Smith, 2006). and universal definition of open peer review. Ross- This process also suffers from plagiarism, self- Hellauer (2017) suggests that open peer review “has citation, conflicts of interest and sometimes it holds neither a standardized definition nor an agreed schema back competing research (Tattersall, 2015). Some of its features and implementations”. He collected the reviewers also take the advantage of their position and corpus of 122 definitions and analysed them for their do the review only because of the sense of power and content. vulnerability in relations to others (Lipworth, Kerridge, As a result, he proposed the following definition: Carter and Little, 2011). “Open peer review is an umbrella term for a number of Peer review can be manipulated and article rejected overlapping ways that peer review models can be even if its contents actually merit publication and this

______28th CECIIS, September 27-29, 2017, Varaždin, Croatia ______Proceedings of the Central European Conference on Information and Intelligent Systems 223

adapted in line with the ethos of open science, peer review. This may include editorial pre-selection of including making reviewer and author identities open, articles and/or final decision-making for acceptance or publishing review reports and enabling greater rejection of articles; The editor-mediated portion of participation in peer review process” (Ross-Hellauer, any open peer review process may or may not be 2017). publicly disclosed; Transparent review refers to complete openness to a distinct com-munity or the public. It allows a public community to watch peer 4.1 Open peer review complexities review unfold. Authors and the public know referees’ identities, and referees know authors’ identities; Since open peer review is a rather new concept, Author responses to referee comments are public. In literature explaining models, concepts, scenarios and transparent review the public can see manuscripts, details about it is neither abundant nor extensive. The reviews, and replies from authors and public reviewers following part of this chapter provides an insight into as well as the published articles.; Crowd-sourced the most interesting thoughts about open peer review review is a public review process in which any in their integral form without intervening into the community member may contribute to the article original text. Ross-Hellauer (2017) offered the full list review. In crowd-sourced review there is no limit to the of open peer review traits: “1.) Open identities: number of comments or reviews an article may receive. Authors and reviewers are aware of each other’s In some proposed implementations of crowd-sourced identity 2.) Open reports: Review reports are review, there is little editorial mediation of article published alongside the relevant article 3.) Open reviews. Rather, authors may simply submit papers to participation: The wider community to able to a pre-print server or other community for crowd- contribute to the review process 4.) Open interaction: sourced commentary; Pre-publication review occurs Direct reciprocal discussion between author(s) and prior to article publication, and typically occurs in a reviewers, and/or between reviewers, is allowed and public space such as a pre-print server; Synchronous encouraged 5.) Open pre-review manuscripts: review occurs at the same time as publication of the Manuscripts are made immediately available (e.g., via article. In the literature, synchronous review is pre-print servers like arXiv) in advance of any formal approached only theoretically, as part of a novel and peer review procedures 6.) Open final-version completely iterative publishing model.; Post- commenting: Review or commenting on final “version publication review occurs after an article is published, of record” publications 7.) Open platforms: Review is much like commentary on a blog post”. Wang, de-coupled from publishing in that it is facilitated by a Wolfram, Hoyt, Ingwersen, Pöschl, Smith and Bates different organizational entity than the venue of (2016) analysed the current open peer review models publication”. Open peer review comes in several and concluded that they aim at different levels flavours or scenarios, depending on the publisher and / of transparency and openness: “1.) Optional open peer or scientific journal. For instance, The Royal Society review, in which reviewers are encouraged, but are not proposed 4 different scenarios: “Author agrees to open required, to provide their identity in review reports; and peer review - referee agrees to open peer review: authors may choose, but are not required, to publish Signed referee report made public; Author does not review histories alongside the articles. In a published agree to open peer review - referee agrees to open peer review history, all versions of the manuscripts, review: Referee name only disclosed to author, referee reviewer reports, and author’s rebuttals with time- report is not made public; Author agrees to open peer stamps are accessible and linked to the article. 2.) review - referee does not agree to open peer review: Mandatory open peer review, in which the review Referee name not disclosed to author or made public, process is open to the public and review histories are referee report made public and Author does not agree accessible to all. Reviews may be conducted to open peer review - referee does not agree to open prepublication so that there is a clear delineation peer review: Referee name not disclosed to author, between the under review versions and the final referee report is not made public” (Open peer review in accepted version. Mandatory open peer review may Royal Society Open Science). Ford (2013) reviewed also be conducted post-publication, in which, articles characteristics of open peer review and offered the are published first and the review process is ongoing”. following extensive list of the implementations of open There are also other views on open peer review which peer review: “Signed review refers to submitted actually try to facilitate the implementation process reviews signed by the referee that are either published itself and make it better. For instance, one of the alongside articles at the time of publication or are harshest criticism of the peer review process is often signed when an author receives them; Disclosed review directed towards choice of reviewers especially when refers to a process in which referees and authors know journal editors are not sure whom to ask to be a each other’s’ identities during peer review process, reviewer because they could create a conflict with enabling them to engage in discussion or discourse; author(s). In such cases, some people are not suitable Editor-mediated review is a characteristic found in to be the reviewer and open peer review would allow most open peer review processes. Editor mediation is authors to request a different reviewer (Bali, 2015) any work done by a journal editor to facilitate open when not satisfied with the review.

______28th CECIIS, September 27-29, 2017, Varaždin, Croatia ______224 Proceedings of the Central European Conference on Information and Intelligent Systems

5 Research collected) to be in favour of open peer review becoming mainstream scholarly practice. The same survey showed that three out of four (76.2%) The research part of the paper is focused on scientific respondents reporting having taken part in open peer journal publishers and their acceptance and possible review. implementation of open peer review. The purpose of the research in this paper was to It must be noted that the development of peer review establish whether scientific journal publishers offer is an on-going process with not so many researches or publicly available documents (policies, author practical experiences published and explained. guidelines etc.) which explicitly describe a publisher’s In the last 15 years, open peer review or its views and stand on open peer review and its equivalents were experimentally implemented in very implementation. Content analysis and comparison few scientific journals. In 2004 the Nursing Research were applied as main research methods. The source of inaugurated open peer review for the first time. Three data about scientific publishers for analysis and years later, 32 papers were published accompanied comparison was the Top publishers list at with the corresponding reviews (Dougherty, 2004). http://www.scijournal.org/top-international-journal- Dougherty suggested the value of peer review itself publisher.shtml where 1600 journal publishers were and open peer review in hands of capable experts listed publishing anything from 1 journal to over 2500 possessing specialized knowledge in the are addressed journals. by paper manuscripts supporting the development of The research questions were: science (in journal’s discipline) with their assessments RQ1: Do scientific publishers support open peer and recommendations. review by publicly announcing their support in form of Journal Nature also started a trial in open peer a publicly available document(s)? review “to explore the interest of researchers in a RQ2 Is there a difference between big and small particular model of open peer review, whether as publishers (in number of scientific journal owned by a authors or as reviewers” (Nature’s Peer Review Trial, publisher) in accepting the open peer review? 2006). The result was that “despite the significant The main hypothesis of the research is that despite interest in the trial, only a small proportion of authors not so short existence of open peer review, it is still not opted to participate” (Nature’s Peer Review Trial, widely accepted and implemented. 2006). One of the most famous examples is journal During the period of June 14th till June 19th 2017 Atmospheric Chemistry and Physics which Web sites of the first 100 scientific publishers from the experimented with the two stage open peer review Top publishers list were visited, searched and browsed (Koop and Pöschl, 2006). In January 2012 Agricultural (read) for information about publicly available and Forest Meteorology began publishing PDFs of documents (policies, guidelines etc.) which contained editor-selected peer review reports alongside to the explanation about what type of peer review is applied published articles. Finally, Elsevier started a pilot study in journal(s) of a particular publisher. of open peer review in 5 of its journals: Agricultural and Forest Meteorology, Annals of Medicine and Table 1. The first 100 publishers on the Top Surgery, Engineering Fracture Mechanics, Journal of publishers list and number of journals they publish Hydrology: Regional Studies, International Journal of (information retrieved in June 2017 from Surgery (Mehmani and van Rossum, 2014). http://www.scijournal.org/top-international-journal- Mehmani and van Rossum (2014) reported that from publisher.shtml) the 1990s onwards, a number of scientific journals began to trial new approaches: BioMed Central asked reviewers to sign their reviews and publishes them Publisher # of journals alongside the author's response; F1000 (Faculty of published 1000) launched F1000 Research which publishes Elsevier 2571 review reports alongside with the submitted articles Springer-Verlag 2209 and eLIFE publishes parts of the post-review decision Taylor and Francis 1803 letters and the associated author responses (if they John Wiley and Sons 1604 agree). The Nature publishing reported in 2016 that three Sage Publications 742 journals: PeerJ, the BMJ and F1000Research embraced SciELO 684 open peer review in various different forms, while Walter de Gruyter 683 Nature Communications is experimenting with this type of peer review (Callaway, 2016). Ross-Hellauer, Redalyc 447 Deppe and Schmidtthe (2017) presented findings of an RMIT Publishing 415 online survey conducted in 2016 for the Inderscience Publishers 391 OpenAIRE2020 about attitudes of editors, authors and Hindawi Publishing Corporation 366 reviewers towards and levels of experience with open pee review. The findings showed that the majority of Project MUSE 361 respondents (total of 3,062 full responses were Cambridge University Press 329

______28th CECIIS, September 27-29, 2017, Varaždin, Croatia ______Proceedings of the Central European Conference on Information and Intelligent Systems 225

Oxford University Press (OUP) 310 Palgrave MacMillan 60 Revues.org 309 Frontiers Media 57 Emerald 308 University of Chicago Press, The 57 Wolters Kluwer - Lippincott 287 EDP Sciences 55 Williams and Wilkins Horizon Research Publishing 55 Biomed Central Ltd. 268 Duke University Press 54 African Journals Online 230 Science Alert 52 Scientific Research Publishing 214 SciTechnol 50 Sabinet Online Ltd 185 PePSIC 48 Medknow Publishers 177 American Chemical Society 47 Institute of Electrical and 174 (ACS) Electronics Engineers (IEEE) Science and Engineering 47 Informa Healthcare 166 Publishing Company Thieme Publishing Group 157 BMJ Publishing Group 46 SpringerOpen 150 Brepols Publishers 46 Brill Academic Publishers 146 OMICS Publishing Group 45 IGI Global 146 Annual Reviews 44 De Gruyter Open 135 Association for Computing 44 Dove Medical Press 129 Machinery (ACM) John Benjamins Publishing 44 MDPI 124 Company World Scientific 114 Sri Lanka Journals Online 44 Nature Publishing Group 110 Royal Society of Chemistry, The 42 Bentham Science 109 Canadian Center of Science and 41 Scientific and Academic 105 Education Publishing (SAP) OA Publishing London 40 Karger 104 Edinburgh University Press 39 Ă‰rudit 99 M.E. Sharpe 39 ISRN International Scholarly 96 Morgan and Claypool 38 Research Network Peeters Publishers 37 Intellect 95 University of California Press 36 IOS Press 93 American Society of Civil 35 Bangladesh Journals Online 87 Engineers (ASCE) Libertas Academica 87 Consejo Superior de 35 OECD - Organisation for 83 Investigaciones CientĂficas Economic Co-operation and Berghahn Journals 34 Development Co-Action Publishing 34 Mary Ann Liebert 81 Herbert Publications 34 ScopeMed 77 Future Medicine 33 Universidad Complutense de 75 Institution of Engineering and 33 Madrid Technology (IET) Academic Journals 73 MIT Press 33 Science and Education 72 Copernicus Publications 32 Publishing FrancoAngeli 70 Hogrefe and Huber Publishing 32 Group Institute of Physics (IOP) 70 ICE Publishing 32 American Psychological 68 Penn State University Press 32 Association (APA) PAGEPress 64 Maney Publishing 31 Akademiai Kiado RT 62 Multiscience 31 STM Journals 61 American Institute of Physics 30 (AIP) Nepal Journals Online 60

______28th CECIIS, September 27-29, 2017, Varaždin, Croatia ______226 Proceedings of the Central European Conference on Information and Intelligent Systems

Asian Network for Scientific 30 International Information (ANSInet) Journal of International Institute for 30 Surgery Science, Technology and Oxford Yes 1 Journal: Giga Education (IISTE) University Science OpenJournals Publishing 30 Press Tehran University of Medical 30 Biomed Yes 2 Journals: Sciences Central Ltd. BMC Infectious It must be said that with some publishers, journals Diseases, have an autonomy to decide which type of peer review Journal of will be applied. Consequentially, the content of Inflammation publishers’ Web sites was carefully analysed to find Springer Yes ? ? the proper and accurate information about the peer Open review process. Individual scientific journals were not MDPI Yes 1 Journal: Life analysed because the first 100 publishers from the Top Nature Yes 1 ? publishers’ list own the total of 20053 journals (Table Publishing 1.). However, such a significant number of journals Group presents a real possibility of implementation of open Palgrave Yes ? Books peer review (something that can be analysed in future MacMillan research). Publishers’ Web sites were analysed for BMJ Yes 2 Journals: BMJ publicly available documents explaining open peer Publishing Open, BMJ review acceptance and / or implementation in a journal Group or journals owned by a particular publisher. In spite of Copernicus Yes* ? ? possible deficiencies (some publishers tend to avoid Publications clarity about the peer review process in their journals * Open peer review is called Interactive Public Peer or give different names to the peer review types in Review order to achieve better marketing effect for their ? Information not available journals) it is the best approach for discovering the actual and current situation regarding the acceptance Three publishers stated they implement open peer and implementation of open peer review process at the review but didn’t provide the number of journals in major scientific publishers. which they implement open peer review and two publishers didn’t provide the titles of journals in which they implement open peer review. These results clearly 5.1 Research results indicate experimental nature of open peer review because of the small number of scientific publishers After the careful analysis of the Web sites of the first implementing this type of peer review. The remaining 100 publishers from the Top publishers list, the results publishers which Web sites were analysed in this are presented in the Table 2. research stated that they apply “standard peer review”, “classic peer review” or more specifically “single- Table 2. Publishers offering open peer review blind peer review” and “double-blind peer review”. Though modest, the results are significant because big Publisher Open # of Publication publishing houses have a substantial influence on peer journals type and title development of the scientific publishing market in review general and can lead other publishers into the new phase of development of peer review if they decide to Elsevier Yes 5 Journals: share their results of implementation of open peer Agricultural review publicly which might become a key issue and Forest regarding the improvement of peer review process. The Meteorology, Annals of results must contain transparent explanation of the Medicine and implementations and their good and bad sides. By doing so, the major scientific publishers will help other Surgery, publishers, journal editors and authors to decide Engineering whether to implement open peer review or not. Since Fracture many scientific journals publishers promote open Mechanics Journal of access on their Web pages, it is safe to assume that they Hydrology: will be more interested in open peer review in future if it demonstrates positive impact on quality of peer Regional review in general. Studies,

______28th CECIIS, September 27-29, 2017, Varaždin, Croatia ______Proceedings of the Central European Conference on Information and Intelligent Systems 227

6 Future research but alternatives should first be tested; and in testing, one is well-advised to manipulate one variable at a time: Here we are dealing with a change in medium After this initial research about the current state in (paper to electronic), a change in economic model implementation of open peer review, more detailed (subscription to author-side payment) and a change in research is planned to discover details about what quality control mechanism (peer review to open peer scientific publishers expect and want from open peer commentary)”. review. The focus of the future research will primarily be on publishers of the Croatian scientific journals, journal editors and authors. While small scientific communities are naturally part of the global scientific References community, they still have their particularities and are often not included in scope of experiments of the world An Introduction to Open Peer Review. (2014) major scientific publishers because of the size of their Scientific American. Retrieved from market. Being a big or small community does not https://blogs.scientificamerican.com/information- presume the outcome of implementation of open peer culture/an-introduction-to-open-peer-review/ review and could provide a valuable insight into effects of implementation of open peer review in the process Callaway, E. (2016). Open peer review finds more of transformation of the scientific information takers. Retrieved from publishing. Such research should be specially focused http://www.nature.com/news/open-peer-review- on reasons why some publishers or journals should or finds-more-takers-1.20969?WT.ec_id=NATURE- should not accept open peer review. 20161117&spMailingID=52782037&spUserID= MjA1NTQwOTk5MwS2&spJobID=1045580408 &spReportId=MTA0NTU4MDQwOAS2 7 Conclusion Cooper, M. L. (2009). Problems, Pitfalls, and Promise in the Peer-Review Process: Commentary on. There is no doubt in any researcher’s mind that the peer Perspectives on Psychological Science, 4(1), 84- review process should be improved. The question is – 90. in what direction? The literature shows that little has Csiszar, A. Peer review: Troubled from the start. changed in the peer review process in the last few Nature, 532(7599) Retrieved from decades. The latest development in this process is https://www.nature.com/news/peer-review- invention of open peer review, a process which should troubled-from-the-start-1.19763 be transparent and open to anyone interested in it. The research study in this paper showed that only few major Dougherty, M. (2007). Open Peer Review 3 Years scientific publishers have commenced an experimental Later. Nursing Research, 56(5), 295. implementation of open peer review in their journals. Eaton K.A., Rex Holland G., Giannobile W.V., The research study answered the both research Hancocks S., Robinson P.G. and Lynch C.D. questions. RQ1: Scientific publishers support open (2014). How is research publishing going to peer review in a small number of cases. This is not progress in the next 20 years?: transcription of encouraging but one should have in mind that the session for editors, associate editors, publishers system of scientific publishing has always been and others with an interest in scientific publishing changing very slowly as well as the system of scientific held at IADR meeting in Seattle on Wednesday, information in general, and that changes have not often 20 March 2013. Journal of dentistry, 42(3), 219- been immediately visible. This doesn’t mean that 228. changes shouldn’t be done but instead, it means that they should be better documented and presented and Ford, E. (2013). Defining and Characterizing Open the information about changes should be made publicly Peer Review: A Review of the Literature Journal available. RQ2: The research study showed that major of Scholarly Publishing. Journal of Scholarly publishers are more inclined towards experimenting Publishing, 44(4), 311-326. and acceptance of open peer review but are also very Harnad, S. (1997). Open Peer Commentary: A careful about extent and way of its implementation. Supplement, Not a Substitute, for Peer Review. Future research which will include direct contact with Retrieved from http://listserv.nasig.org/scripts/wa- scientific publishers of different sizes would NASIG.exe?A3=ind9706&L=SERIALST&E=0& presumably provide more answers. The main P=23967&B=--&T=text%2Fplain&header=1 hypothesis of the research that open peer review is still not widely accepted and implemented has been Harnad, S. (2000). The Invisible Hand of Peer confirmed. At this moment, open peer review is not Review. Exploit Interactive, 5 Retrieved from researched enough so there are still many uncertainties http://users.ecs.soton.ac.uk/harnad/nature2.html surrounding it. As Harnad (1997) pointed out: “Peer Iannone, C. (2013) Peer Review and Other Problems. review is imperfect; it can no doubt be improved upon, Academic Questions, 26(2), 137-140.

______28th CECIIS, September 27-29, 2017, Varaždin, Croatia ______228 Proceedings of the Central European Conference on Information and Intelligent Systems

Janodia M.D. (2017). Open Access Publishing and Open peer review in Royal Society Open Science. Peer Review: Problems and Solutions, Manipal Retrieved from Journal of Pharmaceutical Sciences, 3(1), 1. http://rsos.royalsocietypublishing.org/open-peer- review Koop, T.and Pöschl, U. (2006). An open,two-stage peer review journal. Nature. Retrieved from Overview of Open Access Publishing. Retrieved from http://www.nature.com/nature/peerreview/debate/ http://www.openoasis.org/index.php%3Foption%3 nature04988.html Dcom_content%26view%3Darticle%26id%3D358 %26Itemid%3D263 Lahiri, D. K. (2006). Discourse among referees and editors would help. Nature, 439(16), 784. Overview: Nature's peer review trial. Retrieved from http://www.nature.com/nature/peerreview/debate/ Lajtha, K. and Baveye, P. C. (2010). How should we nature05535.html. deal with the growing peer-review problem?, Biogeochemistry, 101, 1–3. Post-publication peer review. Retrieved from https://www.revolvy.com/topic/Post- Lee, C. (2012). Open peer review by a selected-papers publication%20peer%20review&item_type=topic network. Frontiers in computational neuroscience, 6. Retrieved from Ross-Hellauer, T. (2017). What is open peer review? http://journal.frontiersin.org/article/10.3389/fncom A systematic review. F1000Research 2017, 6, .2012.00001/full 588. Lipworth, W. L., Kerridge, I. H., Carter, S. M. and Ross-Hellauer, T., Deppe, A. and Schmidt, B. (2017). Little, M. (2011). Journal peer review in context: OpenAIRE survey on open peer review: Attitudes A qualitative study of the social and subjective and experience amongst editors, authors and dimensions of manuscript review in biomedical reviewers. Retrieved from publishing, Social Science & Medicine, 72, 1056- https://zenodo.org/record/570864 1063. Sandewall, E. (2012). Maintaining live discussion in Masic, I. (2012). Plagiarism in scientific publishing. two-stage open peer review. Frontiers in Acta Informatica Medica, 20(4), 208-213. computational neuroscience, 6, 1-11. Mehmani, B. and van Rossum, J. (2014). Elsevier Smith, R. (2006). Peer review: a flawed process at the trials publishing peer review reports as articles. heart of science and journals. Journal of the Royal Retrieved from Society of Medicine, 99, 178-182. https://www.elsevier.com/reviewers- Tattersall, A. (2015). For what it’s worth – open peer update/story/peer-review/elsevier-pilot-trials- review landscape. Online Information Review, publishing-peer-review-reports-as-articles 39(5), 649-663. Meruane, O.S., González Vergara, C. and Pina- Wang, P., Hoyt, J., Poschl, U., Wolfram, D., Stranger, Á. (2016). What We Still Don't Know Ingwersen, P., Smith, R. and Bates, M. (2016). About Peer Review. Journal of Scholarly The last frontier in open science: Will open peer Publishing, 47(2), 180-212. review transform scientific and scholarly Nassi-Calò, L. Adoption of open peer review is publishing?. Proceedings of the Association for increasing. Retrieved from Information Science and Technology, 53(1), 1-4. http://blog.scielo.org/en/2017/01/10/adoption-of- Watkinson, A., Jamali, H. R., Herman, E. and Allard, open-peer-review-is- S.L. (2015). Peer review: still king in the digital increasing/#.WUO7leklGRQn age. Learned publishing, 28, 15-21. Nature’s Peer Review Trial (2006). Nature. Retrieved Wheeler, B. (2011). The ontology of the scholarly from journal and the place of peer review. Journal of http://www.nature.com/nature/peerreview/debate/ scholarly publishing, 42(3), 307-322. nature05535.html

______28th CECIIS, September 27-29, 2017, Varaždin, Croatia