Evidence Based Library and Information Practice 2019, 14.3

Evidence Based Library and Information Practice

Evidence Summary

Publons Peer Evaluation Metrics are not Reliable Measures of Quality or Impact

A Review of: Ortega, J. L. (2019). Exploratory analysis of Publons metrics and their relationship with bibliometric and altmetric impact. Aslib Journal of Information Management, 71(1), 124– 136. https://doi.org/10.1108/AJIM-06-2018-0153

Reviewed by: Scott Goldstein Web Librarian Appalachian State University Libraries Boone, North Carolina, United States of America Email: [email protected]

Received: 1 May 2019 Accepted: 14 July 2019

2019 Goldstein. This is an Open Access article distributed under the terms of the Creative Commons‐ Attribution‐Noncommercial‐Share Alike License 4.0 International (http://creativecommons.org/licenses/by-nc- sa/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one.

DOI: 10.18438/eblip29579

Abstract Methods – Author extracted article data from Publons and joined them (using the DOI) with Objective – To analyze the relationship data from three altmetric providers: between scholars’ qualitative opinion of Altmetric.com, PlumX, and Crossref Event publications using Publons metrics and Data. When providers gave discrepant results bibliometric and altmetric impact measures. for the same metric, the maximum value was used. Publons data are described, and Design – Comparative, quantitative data set correlations are calculated between Publons analysis. metrics and altmetric and bibliometric indicators. Setting – Maximally exhaustive set of research articles retrievable from Publons. Main Results – In terms of coverage, Publons is biased in favour of life sciences and subject Subjects – 45,819 articles retrieved from areas associated with health and medical Publons in January 2018. sciences. Open access publishers are also over- represented. Articles reviewed in Publons 153

Evidence Based Library and Information Practice 2019, 14.3

overwhelmingly have one or two pre- altmetric providers and found that coverage of publication reviews and only one post- publications between them varies widely. publication review. Furthermore, the metrics of significance and quality (rated on a 1 to 10 This commentary relies on Perryman’s (2009) scale) are almost identically distributed, critical appraisal tool for bibliometric studies. suggesting that users may not distinguish The study fares very well when evaluated between them. Pearson correlations between against it. For instance, the author clearly Publons metrics and bibliometric and altmetric states the research questions after a literature indicators are very weak and not significant. review that motivates the need for a more in- depth look at Publons. The data are analyzed Conclusion – The biases in Publons coverage using appropriate statistical methods given the with respect to discipline and publisher research objectives, and the results are support earlier research and suggest that the graphically displayed in an appropriate willingness to publish one’s reviews differs manner given the types of analyses performed. according to research area. Publons metrics are The correlations between all the metrics are problematic as research quality indicators. presented in a correlation matrix, which, Most publications have only a single post- although missing a colour legend, is an publication review, and the absence of any excellent way to visualize this type of data. significant disparity between the scores of Limitations of the data sources are also significance and quality suggest the constructs considered. Sufficient, though perhaps are being conflated when in fact they should minimal, detail was provided on how the be measuring different things. The correlation author scraped the data from Publons; it analysis indicates that peer evaluation in would be difficult for a novice researcher in Publons is not a measure of a work’s quality this area to replicate the study. It is not exactly and impact. clear whether scraping the website violates Publons’ terms of service, which explicitly Commentary state that “The correct way to access these data is via our API.” The study is the first in-depth examination of articles in Publons, a web platform that allows This study highlights a couple of implications users to make public their peer-review reports for information professionals. Peer evaluations for journals as well as rate articles on quality of scholarship outside of traditional (double-) and relevance. Previous studies looking at blind peer review (pre- and post-publication Publons have focused on who uses the service review and open peer review as adopted by rather than what articles are reviewed. This some journals during the refereeing process) is study sheds more light on two questions that an emerging practice. Librarians should have been asked in the literature. First, what is familiarize themselves with these new the relationship between traditional methods of review and be in a position to bibliometric indicators and peer subjective inform scholars of their opportunities and valuations (interpreted more broadly than challenges (see Ford, 2016). Information refereed peer review as it encompasses other professionals may also take away from this pre- and post-publication reviews)? In a study a more measured view of the role peer comparable study focusing on Faculty of 1,000 review plays in the assessment of scholarly (F1000) recommendations, Waltman and literature. Although this study suggests Costas (2014) found only a weak correlation Publons metrics are not suitable quality between peer evaluations and citations, indicators for reasons that are specific to the suggesting that the measures may capture Publons platform, it would be wise to different types of impact. Second, in what remember that peer review is not an infallible ways do the data providers used in measure of quality, let alone impact. An scientometric analysis provide discrepant evidence-based critical appraisal of a scholarly data? Zahedi and Costas (2018), for instance, work does not simply consist of checking the conducted the most exhaustive comparison of peer review status. 154

Evidence Based Library and Information Practice 2019, 14.3

References Waltman, L., & Costas, R. (2014). F1000 recommendations as a potential new Ford, E. (2016). Opening review in LIS data source for research evaluation: A journals: A status report. Journal of comparison with citations. Journal of Librarianship and Scholarly the Association for Information Science Communication, 4(General Issue), and Technology, 65(3), 433–445. eP2148. https://doi.org/10.1002/asi.23040 https://doi.org/10.7710/2162-3309.2148 Zahedi, Z., & Costas, R. (2018). General Perryman, C. (2009). Evaluation tool for discussion of data quality challenges in bibliometric studies. Retrieved from social media metrics: Extensive https://www.dropbox.com/l/scl/AAAL comparison of four major altmetric 7LUZpLE90FxFnBv5HcnOZ0CtLh6RQ data aggregators. PLoS ONE, 13(5), rs e0197326. https://doi.org/10.1371/journal.pone.01 97326

155