Game of Tenure: the Role of “Hidden” Citations on Researchers' Ranking In
Total Page：16
File Type：pdf, Size：1020Kb
1Game of Tenure: the role of “hidden” citations on researchers’ ranking in 2Ecology 3Running title: The role of hidden citations in Ecology 4 Ana Benítez-López1,2,* and Luca Santini1* 51Department of Environmental Science, Institute for Wetland and Water Research, Radboud 6University, P.O. Box 9010, NL-6500 GL, Nijmegen, the Netherlands. 72Integrative Ecology Group, Estación Biológica de Doñana, CSIC, 41092, Sevilla, Spain 8*Corresponding author: Ana Benítez ( [email protected] ; [email protected] ) and 9Luca Santini ( [email protected] ) . Both authors contributed equally to the 10manuscript. 11 12ORCID: 13Ana Benítez-López: 0000-0002-6432-1837 14Luca Santini: 0000-0002-5418-3688 15 16 17Data availability 18A more detailed description of the model algorithm, the code of the simulation and all 19estimated parameters are made available as part of the Supplementary Materials (Appendix 20S1, https://figshare.com/s/4c77aa0df498e87ec0ca). 21 22Abstract 23Aim 24Field ecologists and macroecologists often compete for the same grants and academic 25positions, with the former producing original data that the latter generally use for model 26parameterization. Original data are usually cited only in the supplementary materials thereby 27not counting formally as citations, creating an unfair system where field ecologists are 28routinely under-acknowledged and possibly disadvantaged in the race for funding and 29positions. Here we explore how the performance of authors contributing ecological data 30would change if all the citations to their work would be accounted for by bibliometric 31indicators. 32 33Location 34Worldwide 35 36Time period 372008-2017 38 39Major taxa studied 40Homo sapiens academiae 41 42Methods 43We collected the track record of >2300 authors from Google Scholar and citation data from 44600 papers published in 40 ecology journals, including field-based, conservation, general 45ecology and macroecology studies. Then we parameterize a simulation that mimics the 46current publishing system for ecologists and assessed author rankings based on number of 47citations, H-Index, Impact Factor and number of publications under a scenario where 48supplementary citations count. 49 50Results 51We found weak evidence for field ecologists being lower ranked than macroecologists or 52general ecologists, with publication rate being the main predictor of author performance. 53Accounting for supplementary citations in bibliometrics did not substantially change the 54current ranking dynamics. 55 56Main conclusions 57Current ranking dynamics are largely unaffected by supplementary citations as they are 10 58times less than the number of main text citations. This is exacerbated by the common practice 59of citing datasets assembled by previous papers instead than the original articles. 60Nonetheless, researcher performance evaluations should include criteria that better capture 61authors’ contribution of new, publicly available, data. This could encourage field ecologists 62to collect and store new data in a systematic manner, thereby mitigating the data patchiness 63and bias in macroecology studies, and further accelerating the advancement of Ecology. 64 65Keywords: Bibliometrics, Citation analysis, Ecology, H-Index, Journal impact factor, 66Publish or perish, Scientometrics, Supplementary materials 67 68Introduction 69 70The last century has seen an exponential growth of scientific productivity (Larsen & von Ins, 712010; Bornmann & Mutz, 2015), and nowadays, several million papers are published every 72year in about 10,000 scientific journals. Science has also become increasingly competitive, 73and the second half of the last century has been characterized by a radical shift in academic 74practice. The widely known “publish or perish” paradigm (Garfield, 1996) is more relevant 75than ever, with authors under the constant pressure to produce papers in order to succeed in 76an increasingly competitive academic environment (Powell, 2015). 77 Nowadays more researchers compete with each other for declining research grants, 78funding, and academic positions, resulting in a pyramid where for any given number of PhD 79students, only a limited number of postdoc positions and even less tenure track positions or 80professorships are available (Cyranosk et al., 2009; Powell, 2015). In turn, hiring or funding 81committees can hardly evaluate the full scientific production of researchers or it becomes 82increasingly harder to rank increasingly specialized researchers applying for a broadly 83described position, rendering the evaluation by a general committee prone to subjectivity and 84hardly reproducible (Pier et al., 2018; Forscher et al., 2019). This has led to an increased 85reliance by evaluation committees on quantitative metrics for ranking researchers (Wouters, 862014). A multitude of indicators have been proposed, but the most commonly used are the 87number of publications, the total number of citations (Reich, 2013), and the H-index which 88combines the previous two, and corresponds to the number of papers (h) that have each been 89cited at least h times (Hirsch, 2005). Journals are also ranked according to several different 90metrics (Bradshaw & Brook, 2016), but unarguably the most commonly used is the impact 91factor (IF), which measures the average number of citations received by a journal in the 92previous two years (Garfield, 1955). Nowadays, it is common to evaluate the quality of 93papers by the IF of the journal in which they are published, thus the IF today has the power to 94influence decisions made by university hiring committees and funding agencies (Callaway, 952016; McKiernan et al., 2019). 96 Ecology is a relatively young science, which started around the end of the 19th century 97and since then it has rapidly diversified (Benson, 2000). Ecology can be studied at multiple 98organization levels, from individuals to populations, communities, ecosystems or landscapes. 99Further, it can be studied at different spatial, temporal or taxonomic scales, which generally 100exceeds the scale at which field studies can be conducted (Estes et al., 2018). This has led to 101the emergence of disciplines that rely on large quantities of secondary data (data originally 102collected for other research), which we collectively consider as macroecology in this study 103(McGill, 2019). Over the past years the publication of data papers, meta-analyses and 104synthetic analyses in Ecology has skyrocketed (Carmel et al., 2013). Large-scale synthetic 105analyses and big data approaches are instrumental to advance ecology because they identify 106general patterns that escape the idiosyncrasies of local scale studies (Currie, 2019; McGill, 1072019). However, they rely heavily on the availability of high quality, systematically collected 108primary data, whose original sources are commonly cited in the supplementary materials on 109the paper due to journal policies regarding restricted word count and space (Fox et al., 2016). 110This has created an unfair system where citations to primary data in field-based research 111articles are published predominantly in the supplementary material and are systematically 112undercounted or completely ignored (Seeber, 2008), simply because they are invisible to 113search engines such as PubMed, Scopus or Web of Science (Fig. 1). 114 Traditionally ecologists have had few incentives to share their data (Reichman et al., 1152011). It is common that authors are willing to exchange their data for coauthorship in order 116to compensate for the unbalanced credit to their work, which in turn could eventually lead to 117an inflation on the number of citations that an author can get, with respect to the amount of 118data shared, for being included in data papers (e.g. TRY and PREDICTS databases, Kattge et 119al., 2011; Hudson et al., 2014). Further, some databases are not fully accessible, with co- 120authorship being a precondition for the use of the data (e.g. TRY, MoveBANK). This 121problem trickles down from journals ending up with lower impact factors than they should, to 122individual scientists getting fewer citations and lower h-indices than deserved. 123 It has been argued that the role of empirical field research has faded appreciably in the 124past decades (Noss, 1996; Tewksbury et al., 2014; Ríos-Saldaña et al., 2018), and this has 125inevitably generated contrasts between authors focusing at different scales of analysis 126(Gaston & Blackburn, 1999; Ferreira et al., 2016; Ríos-Saldaña et al., 2018), but who might 127compete for the same positions or grants in the near future (but see Arnold, 2003). In 128addition, studies that include primary data collection are generally far more expensive than 129studies that analyze published data, which makes it harder, for example, to find sufficient 130funding to ensure a field-based PhD over a computer-based PhD. Furthermore, high-impact 131journals tend to favour large-scale analyses that answer big, general questions (Fitzsimmons 132& Skevington, 2010). This exacerbates the inequalities between field-ecologists (primary data 133providers) and macroecologists, who use the data produced by the former in large, big data 134analyses (Fig. 1). One solution that has been proposed to make the system fairer would be to 135make supplementary citations count for author's performance metrics and for journals’ 136ranking metrics (Seeber, 2008; Weiss et al., 2010; Pop & Salzberg, 2015; McDannell, 2018). 137 In this study we perform a bibliometric analysis to explore