Multidimensional Journal Evaluation of PLOS ONE
Total Page:16
File Type:pdf, Size:1020Kb
DE GRUYTER DOI 10.1515/libri-2013-0021 Libri 2013; 63(4):259−271 Winner of LIBRI Best Student Paper Award 2013 Christel Fein Multidimensional Journal Evaluation of PLOS ONE Abstract: PLOS ONE (formerly PLoS ONE) is an international sary to evaluate beyond the impact factor to incorporate open access online journal published by the Public Library various quantitative informetric indicators for its several of Science. The periodical covers all science and medicine dimensions into the assessment of a scientific journal categories and has published as many as 28,852 documents (Moed 2005). This paper focuses on the evaluation of the from 2007 to 2011. PLOS ONE will be used to show the range scientific online journal PLOS ONE (formerly PLoS ONE), of journal metrics and informetric methods regarding va- investigating the research question how its standing in lidity, practicability and informative value. To assess this the scientific community is evaluated and whether there data as specifically as possible and to address all relevant are different dimensions each having specific indicators of factors, the evaluation is split into five dimensions, each journal evaluation. Librarians, publishers, editors, schol- of which involves distinct metrics. The five dimensions are arly authors, and the readership are among the stakehold- journal output, journal content, journal perception, journal ers in journal evaluation, but all have diverse preferences citations and journal management. Each of them is pointed in employing methods of journal evaluation (Haustein out in the process of the analyses, and all significant evalu- 2012), and it is of interest which indicators of which di- ation results are presented. The results show that PLOS ONE mension they consider as most significant. has experienced an enormous development. Because of a PLOS ONE is a peer-reviewed open access journal pub- relatively low rejection rate of 31%, its openness towards lished by the Public Library of Science since 2006. A sub- a multitude of different research areas, an internationally stantial characteristic of the journal is its thematic scope large peer review community, and its open access, a plu- which encompasses studies from any field of research con- rality of documents can be published in comparison with a cerning medicine as well as the broader area of science. A print-journal or other online periodicals. The results of the further distinctiveness is the journal’s management with evaluation indicate that PLOS ONE should be assessed from regards to the publication process since each submission numerous perspectives because there are a variety of indi- passes an internal and external pre-publication peer re- cators beyond the impact factor that can be made use of in view procedure and is hence not instantly rejected based order to evaluate exhaustively the standing of the journal as on a potential thematic niche area or an insufficiency of well as its prestige and impact. perceived significance. Christel Fein, Bachelor’s Degree student since October 2010, Institu- Methods te of Linguistics and Information Science, Heinrich-Heine-University, Duesseldorf, Germany. Email: [email protected] All 28,852 documents published in PLOS ONE during the This paper was prepared in October 2012 as part of the Seminar 5-year-period between 2007 and 2011 have been extracted on Empirical Information Science at the Institute of Linguistics from Web of Science®. This data provides the basis for the and Information Science, Heinrich-Heine-University, Duesseldorf, evaluation. The data concerning the conducted evalua- Germany. tion has been collected as well as analysed multidimen- sionally, to demonstrate more fully the complex structures and aspects of the impact, prestige and position of PLOS ONE, and to assess the information and data obtained as Introduction specifically as possible. Based on Juchem, Schlögl, and Stock (2006) and Haustein (2012), a framework of five In order to make an appropriate and exhaustive analysis dimensions of journal evaluation has been applied, in of the influence and impact of a scientific periodical, it is which each contains several metrics to analyse scientific not sufficient to use one single metric. Instead, it is neces- periodicals from various perspectives, i.e. journal output, 260 Christel Fein, Multidimensional Journal Evaluation of PLOS ONE DE GRUYTER journal content, journal perception, journal citations, and tributing authors, institutions as well as countries can be journal management. analysed. Concerning the names of the contributing au- In the following paper, each section focuses on one of thors, it is quite difficult on account of the homonymy and the above-mentioned dimensions in detail and describes synonymy problem to explicitly determine individual au- the relevant methods. Findings are presented in the Re- thors automatically (Smalheiser and Torvik 2009). Hence, sult and Discussion section. The first section deals with the investigation result regarding the contributing authors journal output, a dimension which assesses the publica- limits itself to the average number of authors per docu- tion output of the journal. Subsequently, an analysis of ment as well as document type, and to contributing coun- the content of PLOS ONE is carried out. The dimension tries, which can be determined without any problems be- of the content of a journal involves the thematic scope of cause of the fact that country names are controlled terms the publications within the periodical. To draw inferences in the Web of Science® database. from the reader perception of the journal, the dimension of journal perception gives some indication of the prestige and the impact of PLOS ONE within the scholarly commu- nity. Hence, it includes an assessment of the readership. Journal Content Following this, the findings of the dimension dealing with This dimension includes evaluations of the thematic field journal citations are depicted, and the paper shows how of the journal and the topical focus during the examined publications are used in PLOS ONE for formal scholarly time period. In general, the content of a journal is influ- communication via citations. Finally, a section on journal enced significantly by the editors and the peer reviewers. management addresses issues concerned with the man- agement of the periodical and investigates important facts Too often a journal’s decision to publish a paper is dominated relating to editors, publishers, the review process, and by what the editors think is interesting and will gain greater publication history (Haustein 2012). Because of space lim- readership — both of which are subjective judgments and lead itations of this work, the selection of the discussed data is to decisions which are frustrating and delay the publication of your work. PLOS ONE will rigorously peer-review your narrowed down to major findings. Completing this work, a submissions and publish all papers that are judged to be conclusion summarises the conducted evaluation consid- technically sound. Judgments about the importance of any ering all dimensions, and a final valuation of the impact of particular paper are then made after publication by the the scientific journal PLOS ONE is made. readership. (PLOS ONE 2013) Because PLOS ONE has an international peer review pan- el, and only rejects less than one third of all documents Journal Output submitted, it generally does not exclude manuscripts on the basis of content. One characteristic feature of the jour- This section focuses on the assessment of the scientific nal is that it does not limit itself to a certain scientific field journal PLOS ONE with regards to its publication output. and permits publication of research on topics outside and The data set under evaluation has been limited to the Web between conventional science branches. of Science® coverage of the journal’s publications be- For analysing and illustrating PLOS ONE’s journal tween 2007 and 2011. Therefore, 28,852 documents have content, the documents’ topics have been extracted from been analysed. As Tenopir and King (2009) described it, the Web of Science® dataset and analysed with regards the electronic publishing and the notable growth of sci- to their frequency and change from 2007 to 2011. By com- entific communication have contributed to an increase of paring the content across time, specific words can be re- journal output and single indicator scores. For the pur- vealed which have been the predominant subjects in the poses of this evaluation, the output of PLOS ONE has been documents. Furthermore, progress and the emergence of analysed with respect to the frequency of publication, the research fields can be observed and described by analys- number of issues published within one year, and the fre- ing the content (Chen et al. 2008). The assessment of the quency of publication of various document types, as well document titles was conducted with the free online tool as the average publication length. ‘Wordle.’ A word cloud provides an illustrative impression Furthermore, the output of the journal is influenced of the major subjects of PLOS ONE’s publications in which by the contributing authors as the actual producers of the the font size of a word correlates with its frequency of oc- publication output. The evaluation can be conducted on currence (Peters 2009). Hence, a more frequently used a micro-, meso-, and macro-level, meaning that the con- word has a larger depiction in the word cloud. DE GRUYTER Christel