DE GRUYTER DOI 10.1515/libri-2013-0021 Libri 2013; 63(4):259−271

Winner of LIBRI Best Student Paper Award 2013

Christel Fein Multidimensional Journal Evaluation of PLOS ONE

Abstract: PLOS ONE (formerly PLoS ONE) is an international sary to evaluate beyond the to incorporate online journal published by the various quantitative informetric indicators for its several of . The periodical covers all science and medicine dimensions into the of a categories and has published as many as 28,852 documents (Moed 2005). This paper focuses on the evaluation of the from 2007 to 2011. PLOS ONE will be used to show the range scientific online journalPLOS ONE (formerly PLoS ONE), of journal metrics and informetric regarding va- investigating the question how its standing in lidity, practicability and informative value. To assess this the is evaluated and whether there data as specifically as possible and to address all relevant are different dimensions each having specific indicators of factors, the evaluation is split into five dimensions, each journal evaluation. , publishers, editors, schol- of which involves distinct metrics. The five dimensions are arly authors, and the readership are among the stakehold- journal output, journal content, journal perception, journal ers in journal evaluation, but all have diverse preferences and journal management. Each of them is pointed in employing methods of journal evaluation (Haustein out in the process of the analyses, and all significant evalu- 2012), and it is of interest which indicators of which di- ation results are presented. The results show that PLOS ONE mension they consider as most significant. has experienced an enormous . Because of a PLOS ONE is a peer-reviewed open access journal pub- relatively low rejection rate of 31%, its towards lished by the Public Library of Science since 2006. A sub- a multitude of different research areas, an internationally stantial characteristic of the journal is its thematic scope large community, and its open access, a plu- which encompasses studies from any field of research con- rality of documents can be published in comparison with a cerning medicine as well as the broader area of science. A print-journal or other online periodicals. The results of the further distinctiveness is the journal’s management with evaluation indicate that PLOS ONE should be assessed from regards to the publication process since each submission numerous perspectives because there are a variety of indi- passes an internal and external pre-publication peer re- cators beyond the impact factor that can be made use of in view procedure and is hence not instantly rejected based to evaluate exhaustively the standing of the journal as on a potential thematic niche area or an insufficiency of well as its prestige and impact. perceived significance.

 Christel Fein, Bachelor’s Degree student since October 2010, Institu- Methods te of and Information Science, Heinrich-Heine-, Duesseldorf, Germany. Email: [email protected] All 28,852 documents published in PLOS ONE during the This paper was prepared in October 2012 as part of the Seminar 5-year-period between 2007 and 2011 have been extracted on Empirical Information Science at the Institute of Linguistics from ®. This data provides the basis for the and Information Science, Heinrich-Heine-University, Duesseldorf, evaluation. The data concerning the conducted evalua- Germany. tion has been collected as well as analysed multidimen- sionally, to demonstrate more fully the complex structures and aspects of the impact, prestige and position of PLOS ONE, and to assess the information and data obtained as Introduction specifically as possible. Based on Juchem, Schlögl, and Stock (2006) and Haustein (2012), a framework of five In order to make an appropriate and exhaustive dimensions of journal evaluation has been applied, in of the influence and impact of a scientific periodical, it is which each contains several metrics to analyse scientific not sufficient to use one single metric. Instead, it is neces- periodicals from various perspectives, i.e. journal output, 260 Christel Fein, Multidimensional Journal Evaluation of PLOS ONE journal content, journal perception, journal citations, and tributing authors, institutions as well as countries can be journal management. analysed. Concerning the names of the contributing au- In the following paper, each section focuses on one of thors, it is quite difficult on account of the homonymy and the above-mentioned dimensions in detail and describes synonymy problem to explicitly determine individual au- the relevant methods. Findings are presented in the Re- thors automatically (Smalheiser and Torvik 2009). Hence, sult and Discussion section. The first section deals with the investigation result regarding the contributing authors journal output, a dimension which assesses the publica- limits itself to the average number of authors per docu- tion output of the journal. Subsequently, an analysis of ment as well as document type, and to contributing coun- the content of PLOS ONE is carried out. The dimension tries, which can be determined without any problems be- of the content of a journal involves the thematic scope of cause of the fact that country names are controlled terms the publications within the periodical. To draw inferences in the Web of Science® . from the reader perception of the journal, the dimension of journal perception gives some indication of the prestige and the impact of PLOS ONE within the scholarly commu- nity. Hence, it includes an assessment of the readership. Journal Content Following this, the findings of the dimension dealing with This dimension includes evaluations of the thematic field journal citations are depicted, and the paper shows how of the journal and the topical focus during the examined publications are used in PLOS ONE for formal scholarly time period. In general, the content of a journal is influ- communication via citations. Finally, a section on journal enced significantly by the editors and the peer reviewers. management addresses issues concerned with the man- agement of the periodical and investigates important facts Too often a journal’s decision to publish a paper is dominated relating to editors, publishers, the review process, and by what the editors think is interesting and will gain greater publication history (Haustein 2012). Because of space lim- readership — both of which are subjective judgments and lead itations of this work, the selection of the discussed data is to decisions which are frustrating and delay the publication of your work. PLOS ONE will rigorously peer-review your narrowed down to major findings. Completing this work, a submissions and publish all papers that are judged to be conclusion summarises the conducted evaluation consid- technically sound. Judgments about the importance of any ering all dimensions, and a final valuation of the impact of particular paper are then made after publication by the the scientific journal PLOS ONE is made. readership. (PLOS ONE 2013)

Because PLOS ONE has an international peer review pan- el, and only rejects less than one third of all documents Journal Output submitted, it generally does not exclude manuscripts on the basis of content. One characteristic feature of the jour- This section focuses on the assessment of the scientific nal is that it does not limit itself to a certain scientific field journal PLOS ONE with regards to its publication output. and permits publication of research on topics outside and The data set under evaluation has been limited to the Web between conventional science branches. of Science® coverage of the journal’s publications be- For analysing and illustrating PLOS ONE’s journal tween 2007 and 2011. Therefore, 28,852 documents have content, the documents’ topics have been extracted from been analysed. As Tenopir and King (2009) described it, the Web of Science® dataset and analysed with regards the electronic and the notable growth of sci- to their frequency and change from 2007 to 2011. By com- entific communication have contributed to an increase of paring the content across time, specific words can be re- journal output and single indicator scores. For the pur- vealed which have been the predominant subjects in the poses of this evaluation, the output of PLOS ONE has been documents. Furthermore, progress and the emergence of analysed with respect to the frequency of publication, the research fields can be observed and described by analys- number of issues published within one year, and the fre- ing the content (Chen et al. 2008). The assessment of the quency of publication of various document types, as well document titles was conducted with the free online tool as the average publication length. ‘Wordle.’ A word cloud provides an illustrative impression Furthermore, the output of the journal is influenced of the major subjects of PLOS ONE’s publications in which by the contributing authors as the actual producers of the the font size of a word correlates with its frequency of oc- publication output. The evaluation can be conducted on currence (Peters 2009). Hence, a more frequently used a micro-, meso-, and macro-level, meaning that the con- word has a larger depiction in the word cloud. DE GRUYTER Christel Fein, Multidimensional Journal Evaluation of PLOS ONE 261

However, one has to be aware of the limitations. articles in a journal receive in a given year divided by the Uncontrolled vocabulary involves the risk of ambigui- number of articles published (Haustein 2012). The higher ties. Furthermore, there is no isolation of synonyms and the immediacy index, the higher the speed of im- homonyms and a control for semantic relations is lacking pact. (Haustein 2012). The analysis of terms is thus to be un- Weighted journal indicators have been analysed in or- derstood as a preliminary assessment of topics published der not to solely limit the to the number of in PLOS ONE over time. In addition, using the ‘analyse re- received citations. The score is subject to the sults’ function of the , the journal’s size and is a metric for the absolute weighted im- results for subject categories in the journal covering 2007 pact (Haustein 2012). According to West, Bergstrom, and to 2011 were determined. Bergstrom (2010), the Eigenfactor score takes into consid- eration a citation timeframe of one year and a publication timeframe of five years, and it uses a damping factor of 0.85 to weight citations. As Bergstrom (2007) states, the Journal Citations Eigenfactor is the amount of time an arbitrary reader in- vests in a specific periodical based on the probability The dimension of journal evaluation is concerned with of being accessed through the list of references of other the formal aspects of scientific communication, which documents. In this context, the interconnectedness of the becomes recognisable through citations. A citation is a citation network is also relevant. A size-independent indi- transmission of information from a cited document to cator is the article influence score (Bergstrom 2007). another citing document, which is there noted as a refer- ence, whereas a reference is the bibliographic information [It can be described as an average Eigenfactor per article and it in a document that reveals where the author has gained is] defined as the sum of all weighted citations received in year the information (Stock 2007). Depending on the number y divided by the number of documents published in [the journal of citations, the scholarly influence can be determined, during the five preceding years]. (Haustein 2012, 267) generally considering that a large number of citations of a publication has a greater impact (Haustein 2012). The An illustration of these indicators concerning PLOS ONE evaluation of this dimension investigates the question of follows in the “Results and Discussion” section. how often articles from PLOS ONE are cited, how the per- ception of the journal has developed over the years, which countries cite the journal the most, which other journals cite PLOS ONE the most, and which document types are Journal Perception mainly cited. Journal perception is a complex analytic field that is es- Within the evaluation of journal content, typical jour- sential for a multidimensional assessment of the journal nal indicators have also been assessed. They include the PLOS ONE. The focus of this dimension lies on the behav- impact factor, the immediacy index, the Eigenfactor, and iour of the readership and how PLOS ONE is perceived by article influence. The data were obtained from Journal Ci- them. The following questions will be answered in this tation Report (JCR).The impact factor is the most common section: How often is the journal being read (number of used informetric indicator, even though it is also contro- downloads), how does the number of downloads corre- versial because of its problems regarding its construc- late to the number of citations, and what documents are tion and methodology. However, Garfield (1972) already downloaded but not cited? highlighted that it is a basic indicator which means that PLOS ONE is an online journal and the entire HTML it is a size-independent measure to compare the impact usage data is available online since 2009 as article-level of a journal. “[It illustrates] citations received in one year metrics including the amount of downloads of docu- for documents published in the two previous years.” ments, usage of bookmarking sites, requests, notes, , (Haustein 2012, 233). The impact factor will help evaluat- user comments, etc. These transparent global usage sta- ing a journal’s relative significance, especially when com- tistics have been analysed to address how often docu- paring it to other journals. ments from PLOS ONE have been downloaded and how The immediacy index is a mean citation rate, similar downloads compare to citations. Pearson’s correlation to the impact factor, which is calculated annually by the coefficient has been employed to measure the linear de- Institute for Scientific Information (ISI) for those journals pendence between both data sets. The metric can range which it indexes. It refers to the number of citations the from -1 to 1. A value of 1 suggests a positive linear equation 262 Christel Fein, Multidimensional Journal Evaluation of PLOS ONE DE GRUYTER which describes the relationship between both data sets scientific journal globally. With a publication number of optimally, with all data points lying on a line for which 6,723 documents, PLOS ONE became the largest scientific the one increases as the other increases (Hartung 1999). If journal in the world in 2010 (Morrison 2011). 2011 was an the value is 0, it signifies that there is no linear correlation outstanding productive year recording a growth of 105% between the variables. compared to the preceding year with a publication of 13,785 documents. According to Konkeil (2011), roughly 1 in 60 of all articles indexed by PubMed for 2011were pub- lished in PLOS ONE. Journal Management The rising trend in the number of publications can be explained by the fact that PLOS ONE offers researchers a This section focuses on different aspects related to the relatively high probability of acceptance of an article re- management of the journal by the publishers and editors flected in a rejection rate of just 31% of all submissions and the peer-review process, which influence the reputa- (PLOS ONE 2013) in comparison to other journals such tion of the journal as well as its selection by authors and as PLoS Medicine. In addition, a further positive aspect the readership (Haustein 2012). Findings on the composi- of PLOS ONE that influences authors as the producers of tion of the editorial board will be illustrated. The editorial output is the relatively short duration of the review pro- board plays an important part in cess lasting averagely 117.8 days from submitting a docu- since it shapes the content of a journal and acts as a gate- ment until its publication. The relatively low number of keeper (Braun 2004). To be able to make an assessment of publications in the summer months of June and July may the internationality of the editorial board, which stands be related to the summer break prevailing in all science for the quality of the management and the optimisation disciplines when mainly conferences take place and con- of the scientific quality of the journal, the names of the sequently not much is published. August, September members of the editorial boards have been extracted from and October are then again stronger publication months, PLOS ONE’s website (PLOS Editorial Board 2013) and were which can – amongst other things – be influenced by the then analysed with regards to the author’s nationality. scientific input from the conferences. Moreover, the review process and the time from submis- In the following paragraph, the findings with regards sion to publication are analysed and aspects concerning to the frequency of various document types are described. the publishing market will be addressed. Each document type in a scientific journal serves another purpose. In PLOS ONE, three different document types were published from 2007 to 2011, namely review articles, Results and Discussion editorial material and research articles. While 121 review articles summarised previous publications in order to out- The growth of scientific communication and scientific out- line the field of research and to demonstrate the greater put correlates to a great extent with the progress in the context of single findings, 5 editorials, which were gener- field of research and development (Haustein 2012. In the ally written by the editors of PLOS ONE, served as an in- five years underlying this work for analysis, the journal troduction to a topic as well as a depiction of the nexus. PLOS ONE published a total of 28,852 documents in 60 is- Research articles are the most frequently published docu- sues, which makes 12 issues per year. Having a closer look ment type in the periodical with a total of 28,726 publi- at Figure 1 and the number of publications per issue in di- cations from 2007 to 2011. This document type alone ac- rect comparison from 2007 to 2011 it is immediately strik- counts for about 99.6% of all publications of the journal ing that 2011 stands out as an above-average productive in the period from 2007 to 2011. It primarily serves the dis- year. Even though the time span under analysis has been semination and announcement of latest scientific knowl- limited to five years, a general increase of the publication edge through the presentation and discussion of new re- output by . 1026% from 2007 to 2011 can be noticed search results (Haustein 2012). for all publications. In 2007, PLOS ONE published 1,223 Through this strong unilateral result of the distribu- documents and in 2011 13,785. In 2008, it published 2,716 tion of document types, the focus and the aim of PLOS documents, more than twice as many as in the previous ONE becomes evident, namely to fulfil especially the in- year, presumably because of increasing awareness of the formation needs of readers who want to learn about cur- journal, making it the largest open access journal glob- rent research and trends, and provide researchers with a ally (Konkeil 2011). The following year, 4,405 documents platform to share their research with the scientific com- were published and the journal became the third largest munity. Furthermore, PLOS ONE is an online-only publi- DE GRUYTER Christel Fein, Multidimensional Journal Evaluation of PLOS ONE 263

Figure 1: Direct comparison of the number of publications per issue from 2007 to 2011 cation which results in the fact that more documents than evaluation. The number has increased annually, except in a print journal could contain can be published. 2009. In 2008, an increase in the number of authors per Moreover, the number of pages of a publication can publication could be recorded compared to 2007. This can also be measured empirically and attributed to journal be an effect of the increasing popularity of the journal. output. The 28,852 analysed documents published in Striking, however, is the variation of the number of au- PLOS ONE between 2007 and 2011 have an average docu- thors depending on the document type. Editorial material ment length of 9.95 pages. The longest number of pages stands out with a number of averagely only 3.2 authors per was a 55-page article which dealt with evolutionary bi- document. The editorial is written by editors or authors ology and had the title “New Information on the Cranial of PLOS ONE, a major collaboration of different authors is Anatomy of Acrocanthosaurus Atokensis and Its Implica- thus not prevailing. tions for the Phylogeny of Allosauroidea.” The shortest On a macro-level, the evaluation of the countries published documents encompassed 2 pages. Variations in contributing to PLOS ONE has revealed that researchers the length of documents may be because of the different across the world contribute to the journal, which thus has topics covered. an international focus and is not limited to its publishing PLOS ONE is a periodical which is interested in re- country of origin, the United States. search reports from the entire field of science. When ana- However, the United States was the country with the lysing the document length per document type, it becomes most contributing authors as can be seen in Figure 2. evident that the documents vary in length depending on The United States had three times as many publications the respective document type. While research articles as the second-placed England. Germany is in third place comprise 9.9 pages on average, editorials average only 5.6 in the international ranking in terms of the contributing pages. However, reviews are the longest document type institutions. Although Germany is not an English-speak- with an average of 11.7 pages, although they only account ing country, it is among the world’s leading countries in for ca. 0.4% of all publications. These can be the field of medicine and biotechnology research, which based on the fact that each document type pursues a dif- may explain the third position. Moreover, it is noticeable ferent purpose (see above). that 14 of the top 25 countries were European countries The 28,852 documents computed from the Web of Sci- whereas 7 were Asian. Overall the distribution of papers ence® data set were examined with respect to the number per country reflected the common output per country in of authors. It becomes evident that an average of 6.77 au- science and medicine, although the United States was ex- thors was involved per document in the five years under ceptionally strong. 264 Christel Fein, Multidimensional Journal Evaluation of PLOS ONE DE GRUYTER

Figure 2: Top 25 of contributing countries

The analysis of PLOS ONE title terms found a corre- eas of science equally: , genetics and molecu- spondence of the main focuses of the subject categories lar , agricultural science and science as well as from the website PLOS ONE (namely biochemistry, bi- medicine. Other PLOS journals like PLOS Medicine, PLOS ology, genetics and , infectious diseases, Genetics, PLOS Biology, PLOS Pathogens, are by contrast neuro science). It became obvious that, in the course of limited to specific disciplines and do not encompass the en- the years, initial hot topics were still treated consistently tire range of themes that are published by PLOS ONE. The in 2011 (compare Figure 3 to Figure 4). However, it is strik- immense scientific output is also because of that aspect. ing that the term ‘’ increasingly became the subject The number of publications has increased stead- matter of publications. Also, publications dealing with ily over the years 2007 to 2011. However, this is not au- ‘cells’ and ‘study’ gained further importance. A decrease tomatically associated with a simultaneous increase in of articles on the subject of ‘malaria’ and ‘evolution’ was, the number of citations. From 2007 to 2009, the number however, evident. of citations has increased, while from 2009 to 2011 it has From the results based on subject categories, it became decreased (see Table 4). In 2007, each publication was apparent that articles in PLOS ONE cover the following ar- cited 21.3 times on average. As to be expected because of DE GRUYTER Christel Fein, Multidimensional Journal Evaluation of PLOS ONE 265

Figure 3: Tag cloud generated with Wordle considering the titles of the articles published in PLoS ONE in 2007

Figure 4: Tag cloud generated with Wordle considering the titles of articles published in PLoS One in 2011 citation delay, this number has decreased continuously. On the question in which country PLOS ONE is most In 2008, a publication was cited 17.2 times, in 2009 10.8 perceived it was found that the number of citations from times and in 2010 only 5.2 times. In 2011, there were on the United States has been very high for the entire five- average just 1.1 citations per publications. However, this year period. This could result from the fact that the major- can be because of the fact that publications of the strong ity of citing PLOS ONE are located in the United output years 2010 and 2011 need a certain amount of time States. Besides, the output of the journal is also very high until they are cited. in the United States. From the first section dealing with 266 Christel Fein, Multidimensional Journal Evaluation of PLOS ONE DE GRUYTER

Figure 5: Top 10 countries according to number of citations to papers published in PLoS ONE 2007 to 2011 journal output it has become clear that the United States PLOS ONE. The large number of articles was accompanied comes in on the top of the publishing countries. In com- by a high number of citations. As shown in Table 2, arti- parison to other countries, the number of citations from cles have been cited 169,791 times which makes an average the United States is more than three times as high with a number of citations per article of 5.9. Considering the rela- number of 10,808 citations compared to England coming tion of publications and citations for reviews, it was found in second place with a number of 2,758 citations in 2007 out that the average number of citations is 4.6. Although (see Figure 5 and Table 1). there are only 121 reviews, they have been cited 559 times. One striking aspect is that the shape of the curve re- Therefore, the relation of the number of publications and lating to the number of citations reflects the shape of the the number of citations is essential for calculating the av- curve relating the number of citations from the United erage number of citations. With 5 publications that have States and all other countries except Germany. In Ger- been cited 17 times, the editorial material has an average many, the number of citations has increased to 10,807 in number of citations of 3.4. In the synopsis, it can be found 2011 and hence it is not declining but even surpasses the that articles are still perceived best. However, the other number of citations from the United States which only has document types are also relatively well perceived. a total of 7,028 citations (see Figure 5). Furthermore, an evaluation was conducted to deter- Within the scope of the evaluation of journal cita- mine which other scientific journals cite PLOS ONE. 2010 tions, an analysis of which of publication with re- was the year in which PLOS ONE was on average cited the gards to the document type was perceived most has also most by other journals. In the top 10 of the PLOS ONE cit- been made. As was pointed out in the first section, 28,726 ing journals is just one other PLoS journal, PLoS Genet- articles, 5 editorials und 121 reviews were published in ics, in eighth place with 451 citations, more than twice as DE GRUYTER Christel Fein, Multidimensional Journal Evaluation of PLOS ONE 267

Table 1: Number of citations from countries from 2007 to 2011

Ranking Countries/Territories 2007 2008 2009 2010 2011 1 USA 10808 18488 19185 14323 7028 2 England 2758 4593 4412 3339 1562 3 Germany 2660 4181 4230 2930 10857 4 France 2105 3219 3277 2287 1081 5 China 1610 3073 3263 2833 2157 6 Canada 1591 2613 2626 1957 919 7 Japan 1293 1931 2019 1482 783 8 Italy 1280 2298 2277 1637 794 9 Australia 1068 1922 1938 1542 835 10 Netherlands 1017 1846 1869 1355 670 11 Spain 997 1898 1846 1347 680 12 Switzerland 795 1426 1406 1021 473 13 SWEDEN 608 1051 1080 837 456 14 Scotland 501 773 794 629 254 15 Belgium 485 843 880 700 364 16 Denmark 407 720 611 490 265 17 South Korea 390 739 858 568 282 18 Brazil 389 654 676 498 284 19 Israel 370 601 593 373 191 20 India 363 643 676 555 310 21 Austria 314 523 538 368 199 22 Norway 252 394 450 328 163 23 South 249 525 476 339 163 24 Taiwan 249 451 450 343 289 25 Finland 248 497 481 329 195

Table 2: Number of publications, citations and average rates per Document type

Document type Publications Citations Citation rate (Average References Reference rate number of citations per document type) Article 28726 169791 5,9 1345475 46,8 Editorial Material 5 17 3,4 160 32,0 Review 121 559 4,6 10429 86,2 Total 28852 170367 13,9 1356064 165,0

many as in the preceding year’s 213 citations. The journal In addition, the immediacy index has been analysed, citing PLOS ONE most frequently is the journal Proceed- the results of which can be seen in Table 3. Since PLOS ings of the National of of the United ONE publishes twelve issues per year, a higher number States of America (PNAS), which is among the most-cited of citations can add up for a current year (Tomer 1986). interdisciplinary scientific periodicals on a global scale It becomes evident that the immediacy index reached its (PNAS 2013). peak in 2009 with a value of 0.582. In the following year The impact factor of PLOS ONE was 4.351 in 2009. It the immediacy index was 0.515 and in 2011 0.437. From increased to 4.411 in 2011 and decreased to 4.092 in 2011 2009 to 2011 the number of publications has increased im- (see Table 3). Generally, journals with a narrow thematic mensely, especially from 2010 to 2011. field have a smaller impact factor than periodicals cover- Considering the data of JCR® (Journal Citation Re- ing a broader scientific area. The higher the impact fac- ports®), the Eigenfactor score for PLOS ONE was 0.16 in tor, the higher the prestige and significance a journal has. 2009. It doubled to 0.32 in 2010. In 2011, it increased to 0.50 However, the impact factor is not an indicator for quality. (see Table 3). As stated before, the Eigenfactor depends on 268 Christel Fein, Multidimensional Journal Evaluation of PLOS ONE DE GRUYTER

Table 3: Overview of journal indicators for PLoS ONE

Publication Year Total cites Impact factor Immediacy index Eigenfactor Article influence 2009 20466 4.351 0.582 0.16359 1.921 2010 42795 4.411 0.515 0.31957 1.941 2011 75544 4.092 0.437 0.50216 1.797

Table 4: Overview of the number of publications, downloads and citations and their relation

Publication Year Publications Downloads Downloads per Citations Citation Rate (Average publication number of citations per publication) 2007 1223 4742508 3877.77 26063 21.311 2008 2717 8409898 3095.29 46698 17.187 2009 4404 11036484 2506.01 47643 10.818 2010 6723 12325505 1833.33 34768 5.172 2011 13785 15171181 1100.56 15195 1.102

the size of the journal and since PLOS ONE’s number of The evaluation of the 25 most downloaded documents publications has increased in 2011, the Eigenfactor score in comparison to the number of citations shows that huge has increased as well. It becomes evident that the article variations are possible between downloading and actual influence score correlates with the impact factor. In 2009, scientific use via citations. The document that was down- the article influence score in PLOS ONE was 1.92. In the loaded the most was only cited two times in Web of Sci- following year it slightly increased to 1.94, but in 2011 the ence® whereas the document being at the 24th position article influence score decreased to 1.79. The high increase in the download rankings was cited 145 times. The most in the number of publications in the journal also plays a downloaded article was about fellatio by fruit bats which role and has to be taken into consideration when looking is said to prolong copulation time. Many people will have at the article influence scores from 2009 to 2011. downloaded this article out of curiosity. One could con- Analysing PLOS ONE’s article-level metrics, it becomes clude that the title of a publication has an immense im- apparent that the number of downloads has increased from pact on the behaviour of the readership. However, while it 4,742,508 in 2007 to 15,171,181 in 2011 (see Table 4). It is might be an interesting topic to read, further scientific re- striking, however, that the average number of downloads search might be not as great. The article which was down- per publication has steadily decreased from an average of loaded 45,545 times and cited 145 times is about how a low 3877.77 downloads in 2007 to 1100.56 downloads in 2011. dose of dietary resveratrol partially mimics caloric restric- This trend could be a result of the large number of publica- tions and retards parameters in mice. The content tions which also have enormously increased over the years. of this publication is subject to a scientific discipline that When comparing the download numbers in relation has large funds and strong investors since research and to the number of citations it becomes evident that the the discovery of remedies against aging are of utmost number of downloads in 2007 was 182 times as high as the importance to the pharmaceutical- and cosmetics indus- number of citations in the same year (see Table 4). In the tries. following year, the relation is nearly identical. However, PLOS ONE is a unique journal since it is among those in 2009 the number of downloads is 232 times as high as journals which buck the global market’s trend for scholar- the amount of citations. In 2010, the number of publica- ly journal publishing to be operated by commercial com- tions rapidly increases. The number of downloads, with a panies like : total of 12,325,505, is 355 times as many as the number of citations. In 2011, the number of downloads was 998 times PLoS is a non-profit organisation of scientists and physicians committed to making the world’s scientific and medical as many as the number of citations. Pearson’s correlation literature a freely accessible public resource. (PLOS ONE 2013) coefficient regarding the citation and download data has a value of 0.34. While the cumulative number of downloads The Public Library of Science has been publishing the increased over the years, the number of citations has de- periodical PLOS ONE as an open access peer-reviewed creased since 2009. DE GRUYTER Christel Fein, Multidimensional Journal Evaluation of PLOS ONE 269 scientific journal since December 2006. In 2004, Yue and readership. PLOS ONE, however, thoroughly peer-reviews Wilson defined journal age as the number of years since any submission. Then, all documents that are assessed the first publication regardless of title change and hence, to be formally flawless are accepted. An evaluation on PLOS ONE has a publication history of six years. The first the significance of a document is made by the reviewers journal that was published by the Public Library of Sci- (Bingham 2012). ence was PLOS Biology in October 2003, whereas PLOS Ne- glected Tropical Diseases commenced publication as late [When a document has passed the quality check, it is sent to a] member of the editorial board, who takes responsibility as the as October 2007. academic editor for the submission. (PLOS ONE 2013) The scope of a journal’s influence is of great signifi- cance in authors’ decisions in choosing an appropriate The latter is accountable for carrying out the peer-review journal for their respective publications. For readers it process and for accepting or rejecting a (PLOS is also necessary to be aware of the thematic field of the ONE 2013). 95.8% of the documents accepted for publica- journal. PLOS ONE tries to attract a large group of readers tion were sent for reassessment by external experts. 4.2% and authors in its self-portrait regarding its scope. “[PLOS of documents are reconsidered by the editors themselves. ONE] features reports of original research from all disci- PLOS ONE publishes approximately 69% of all submis- plines within science and medicine” (PLOS ONE 2013). sions, after review by, on average, 2.8 experts (PLOS ONE Because PLOS ONE does not exclude papers on the basis 2013). Only 31% of the submitted documents are rejected. of subject field, the journal allows for the discovery of the PLOS ONE’s acceptance rate hence is 69% and thereby es- relations between papers whether within or between dis- sentially higher than any other PLoS journal. ciplines (PLOS ONE 2013). An analysis of the 50 most cited documents shows Within the realm of the journal’s scope, it can be high- that it takes an average of 117.88 days until a document lighted that PLOS ONE received the Publishing Innovation is published. The publication delay, i.e. the time between Award of the Association for Learned and Professional submission and publication (Amat 2008), takes only little Society Publishers in 2009 (Binfield 2009). The prize is time compared to other journals. As can be seen in Table awarded as a sign of appreciation for a ground-breaking 5, from submission to acceptance 82.6 days pass on aver- approach to any facet of publication. age. From acceptance to publication only 35.28 days pass. PLoS ONE from the Public Library of Science […] combines the PLOS ONE is an open access journal. According to traditional values of the journal with innovative online features Linde and Stock’s (2011) classification of open access, to create an inclusive and efficient publication channel. It is PLOS ONE mainly follows the silver road, meaning that au- bold and successful and shaping the future of publishing. thors pay a publication fee which partly covers expenses (Binfield 2009) including peer review, journal production, online hosting and archiving, so that the journal can continue to provide The findings demonstrate that the journal is managed in- open access. The Creative Attribution Licence ternationally. As can be seen in Figure 6, most editors orig- is applied to all publications in the periodical. As a con- inate from the United States (1,336 editors), followed by sequence, all authors retain of the with significantly fewer editors (260 edi- for their respective publications. However, any download, tors). Germany is in third position with 192 editors. France reuse, reprint, alteration, distribution, and/or replication has 175 editors, followed by Canada with 143 editors, Aus- of articles is permitted as long as the original authors and tralia with 131 editors, Italy with 113 editors, Spain with sources are cited (Linde and Stock 2011). 101 editors, China with 59 editors, and finally Brazil with An immediate and free provision of a document is 58 editors. Zsindely, Schubert, and Braun (1982) state that guaranteed because of the author-pays model. PLOS a minimum of five countries must be represented by mem- ONE charges authors US $1,350 to publish a document, bers of the editorial board for a journal to be international. an amount which has not been raised since August 2009 In the first four years following its launch, the periodical (PLOS Publish 2013). It was not until 2010 that PLOS ONE employed over 35,000 external peer reviewers (Laloup covered its operational costs without making a loss (Jer- 2010). With over 3,000 academics from 44 countries rep- ram 2011). The immense journal output and increase of resented on the editorial board, a clear international influ- number of publications can be regarded as the reason for ence and attraction can be observed. the surplus. It is rather often the case that a journal’s decision to publish a manuscript is subject to the editor’s attitude and his assessment whether the document will gain great 270 Christel Fein, Multidimensional Journal Evaluation of PLOS ONE DE GRUYTER

Table 5: Duration from submission to publication.

Ø Submission – Acceptance Ø Acceptance – Publication Ø Submission – Publication 82.6 days 35.28 days 117.88 days

Figure 6: Internationality of the editorial board of PLOS ONE

Conclusion ONE and different informetric indicators have been ap- plied. In the context of this empirical study, 28,852 publications In the course of this study it has become apparent that in PLOS ONE from 2007 to 2011 have been the subject of PLOS ONE is not a traditional journal. It has experienced the analysis of the scientific periodical. It was shown that an enormous development, taking into account that it is a it does not suffice to make use of only one single metric to relatively recent periodical and has increasingly high pub- be able to do an adequate and entire investigation of the lication numbers. This trend is due to the fact that PLOS influence of this scientific journal. For the assessment of ONE is built on several conceptually different ideas com- the prestige, scope, effect and impact of PLOS ONE, it is pared to traditional peer-reviewed scientific publishing in rather necessary to employ several quantitative, informet- that it does not use the perceived importance of a paper as ric indicators for various starting points as well as aspects a criterion for acceptance or rejection. PLOS ONE leaves it and consequently to evaluate multidimensionally. to the scientific community to ascertain importance since For the evaluation of PLOS ONE, five dimensions papers will be made available for community-based open have been used within which aspects and questions were peer review containing online commentary, discussion, subsumed and then examined empirically. These dimen- and rating by more than 3,000 editors from the editorial sions are journal output, relating to the actual publication board and over 35,000 external peer reviewers (MacCal- output, journal content, involving the thematic scope of lum 2006). Furthermore, PLOS ONE is open to documents PLOS ONE, journal citations, assessing the received cita- from all scientific disciplines. Traditionally, journals limit tions and therewith the impact of scholarly communica- the thematic extent of papers too strictly. It does not re- tion, journal perception and usage, implying an analysis strict itself to a specific scientific area in an effort to facili- of the readership, as well as journal management, cover- tate publications on topics outside or between traditional ing all aspects relating to the management of PLOS ONE science areas. Because of its as an online-only pub- by publishers and editors. In this paper, each section lication the journal can publish more documents than a has considered one of the five facets of evaluating PLOS periodical in print. A further feature of PLOS ONE is the DE GRUYTER Christel Fein, Multidimensional Journal Evaluation of PLOS ONE 271 author-pays model, facilitating the provision of all docu- Linde, F., and W. G. Stock. 2011. Information Markets. A Strategic ments as open access quickly after publication. Guideline for the I-Commerce. Berlin, Germany: De Gruyter Saur. This analysis has revealed that there are numerous MacCallum C. J. 2006. “ONE for All: The Next Step for PLoS.” PLOS perspectives from which PLOS ONE can be assessed and Biology 4 (11): e401. doi:10.1371/journal.pbio.0040401 there are a variety of indicators beyond the impact fac- Moed, H. F. 2005. Citation Analysis in Research. Dordrecht, tor that can be made use of in order to fully evaluate the Netherlands: Springer. standing of the journal as well as its prestige and impact. Morrison, H. 2011. “PLoS ONE now World’s Largest Journal.” Accessed September 4, 2012. http://poeticeconomics.blogspot. de/2011/01/-one-now-worlds-largest-journal.html. Peters, I. 2009. Folksonomies: Indexing and Retrieval in Web 2.0. Berlin, Germany: De Gruyter Saur. References PLOS Editorial Board. 2013. “PLOS ONE Editorial Board.” Accessed October 25, 2013. http://www.plosone.org/static/edboard. Amat, C. B. 2008. “Editorial and Publication Delay of Papers action. Submitted to 14 Selected Food Research Journals: Influence of PLOS ONE. 2013. “PLOS ONE Journal Information.” Accessed October Online Posting.” 74 (3): 379–389. doi: 10.1007/ 14, 2013. http://www.plosone.org/static/information.action. s11192-007-1823-8 PLS Publish. 2013. “Pricing Policy: Publication Fees.” Accessed Bergstrom, C. T. 2007. “Eigenfactor: Measuring the Value and October 14, 2013. http://www.plos.org/publish/pricing-policy/ prestige of Scholarly Journals.” College and Research Libraries publication-fees/?__utma=261502610.115420522.13472002 News 68 (5): 314–316. 70.1348831478.1348846096.8&__utmb=261502610.1.10.13 Binfield, P. 2009. “PLoS ONE Wins ALPSP Award for Publishing 48846096&__utmc=261502610&__utmx=-&__utmz=26150 Innovation 2009 – ‘Bold and Successful and Shaping the Future 2610.1347812223.4.4.utmcsr=en..org|utmccn=(ref of Publishing.’” PLOS blogs. Accessed September 7, 2012. erral)|utmcmd=referral|utmcct=/wiki/PLOS_ONE&__utmv=- http://blogs.plos.org/everyone/2009/09/14/plos-one-wins- &__utmk=67714578. alpsp-award-for-publishing-innovation-2009./. Proceedings of the National Academy of Sciences of the United States Bingham, M. 2012. “Rejected without Review: Here’s Why it of America (PNAS). 2013. “About PNAS.” Accessed October 14, Happens.” Accessed September 7, 2012. http://www. 2013. http://www.pnas.org/site/aboutpnas/index.xhtml. maxbingham.com/blog/2012/03/rejected-without-review- Smalheiser, N. ., and V. I. Torvik. 2009. “Author Name heres-why-it-happens. Disambiguation.” In Annual Review of Information Science Braun, T. 2004. “Keeping the Gates of Science Journals: Gate and Technology, 43, edited by B. Cronin, 1–43. Bingley, UK: Keeping Indicators of National Performance in the Sciences.” Emerald. doi: 10.1002/aris.2009.1440430113 In Handbook of Quantitative Science and Technology Research: Stock, W. G. 2007. Information Retrieval: Informationen Suchen und The Use of Publication and Statistics in Studies of Finden. Munich, Germany: Oldenbourg. S & T Systems, edited by H.F. Moed, W. Glänzel, and U. Tenopir, C., and D. W. King. 2009. “The Growth of Journals Schmoch, 95–114. Dordrecht, Netherlands: Kluwer ,” In The Future of the , edited by B. Publishers. Cope and A. Phillips, 105–123. Oxford, UK: Chandos Publishing. Chen, C. M., I. Y. Song, X. J. Yuan, and J. Zhang. 2008. “The Thematic Tomer, C. 1986. “A Statistical Assessment of 2 Measures of and Citation Landscape of Data and Knowledge Citation – The Impact Factor and the Immediacy Index.” (1985–2007).” Data and Knowledge Engineering 67 (2): 234– Information Processing and Management 22 (3): 251–258. doi: 259. doi: 10.1016/j.datak.2008.05.004 10.1016/0306-4573(86)90057-9 Garfield, E. 1972. “Citation Analysis as a Tool in Journal Evaluation. West, J. D., T. C. Bergstrom, and C. T. Bergstrom. 2010. “The Journals Can Be Ranked by Frequency and Impact of Citations Eigenfactor Metrics ™: A Network Approach to Assessing for Studies.” Science 178 (4060): 471–479. Scholarly Journals.” College and Research Libraries 71 (3): Hartung, J. 1999. Statistik. Munich, Germany: Oldenbourg. 236–244. Haustein, S. 2012. Multidimensional Journal Evaluation. Analyzing Yue, W. P., and C. S. Wilson. 2004. “Measuring the Citation Impact of Scientific Periodicals beyond the Impact Factor. Berlin, Research Journals in Clinical : A Structural Equation Germany: De Gruyter Saur. Modelling Analysis.” Scientometrics 60 (3): 317–332. doi: Jerram, P. 2011. “PLoS Progress Update.” PLOS blogs. Accessed 10.1023/B:SCIE.0000034377.93437.18 September 7, 2012. http://blogs.plos.org/plos/2011/07/2010- Zsindely, S., A. Schubert, and T. Braun. 1982. “Editorial Gatekeeping plos-progress-update/. Patterns in International Science Journals. A New Science Juchem, K., C. Schlögl, and W. G. Stock. 2006. “Dimensionen Indicator.” Scientometrics 4 (1): 57–68. doi: 10.1007/ der Zeitschriftenszientometrie am Beispiel von “Buch und BF02098006 Bibliothek”.” Information: Wissenschaft & Praxis 57 (1): 31–37. Konkeil, S. 2011. “PLOS ONE: Five Years, Many Milestones.” Editorial history Accessed September 4, 2012. http://blogs.plos.org/ received 29 June 2013 everyone/2011/12/20/plos-one-five-years-many-milestones. accepted 6 August 2013 Laloup, J. 2010. “Thanking our Peer Reviewers.” PLOS blogs. Accessed September 16, 2012. http://blogs.plos.org/ everyone/2010/12/29/thanking-our-peer-reviewers-2/.