Rethink Impact Factors: Find New Ways to Judge a Journal

COMMENT MEDICINE How does the NEUROSCIENCE How evolution STATISTICS This time with CONSERVATION Find ways to test human microbiome work shaped our biology for feeling: P values, confidence toolkit that nudges people to as an ecosystem? p.623 social living p.627 and significance pitfalls p.628 protect the planet p.630

Rethink impact factors: find new ways to judge a journal A broader, more-transparent suite of metrics will improve science publishing, urge Paul Wouters and colleagues, together with 18 co-signatories.

lobal efforts are afoot to create a to hammer out a broader suite of journal of this change: expansion of indicators to constructive role for journal metrics indicators, and other ways to judge a jour- cover all functions of scholarly journals, a in scholarly publishing and to dis- nal’s qualities. It is a challenging task: our set of principles to govern their use and the Gplace the dominance of impact factors in the interests vary and often conflict, and change creation of a governing body to maintain assessment of research. To this end, a group requires a concerted effort across publishing, these standards and their relevance. of bibliometric and evaluation specialists, academia, funding agencies, policymakers Our proposal stems from a 2017 workshop scientists, publishers, scientific societies and and providers of bibliometric data. held in Leiden, the Netherlands. It was co-

ILLUSTRATION BY DAVID PARKINS DAVID BY ILLUSTRATION research-analytics providers are working Here we call for the essential elements organized by the Centre for Science and

COMMENT

Technology Studies at Leiden University The prominence of the JIF in research (where P.W., S.d.R. and L.W. work), Clarivate KEY FUNCTIONS evaluation and the subsequent flourishing of Analytics (the company that produces abuses (such as stuffing reference lists with the annual Journal Citation Reports) and What’s a journal for? journal self-citations) and even fraud (such as Europe’s life-sciences organization, EMBO1. the emergence of a cottage industry of ques- More than two dozen professionals from ●●Registering. Through publishing, tionable journals touting fake impact fac- across the scholarly ecosystem participated journals associate the intellectual tors) are particularly distressing examples4,7. (see also go.nature.com/2wfeyjc). claims in a piece of work with a date The San Francisco Declaration on Research We delineated the key functions of jour- and authorship, which can be used to Assessment (DORA), which critiques the use nals, which remain largely unchanged since establish priority. of the JIF as a surrogate measure of quality their inception more than 350 years ago. ●●Curating. Through editorial and for individual research articles or research- These are to register claims to original work, other review, work is selected and ers, has now been signed by 1,356 institu- to curate the research record (including issu- placed in a collection; this collection tions and more than 14,000 individuals. The ing corrections and retractions), to organize signals associations and delineates the Leiden Manifesto, which formulated more critical review and to disseminate and archive theoretical and methodological scope general principles for evaluation8, has been scholarship (see ‘What’s a journal for?’) of a scholarly domain. translated into 23 languages (see go.nature. The creation of new indicators is par- ●●Evaluating. Through peer review, com/2hv7eq3). Despite those initiatives, the ticularly important given that journals are works are evaluated according to influence of the JIF is still dominant. evolving rapidly and are becoming platforms several criteria (such as quality To prevent such abuse, we propose that for disseminating data, methods and other and novelty), and authors receive any use of indicators meet four criteria: digital objects. The Journal Impact Factor feedback from their peers. Through Justified. Journal indicators should have (JIF) is based on citations, as are most other publishing, the journal certifies only a minor and explicitly defined role in indicators in common use. These capture that the work has been evaluated; assessing the research done by individuals only limited aspects of a journal’s function. the journal continues to perform or institutions9. A more nuanced set of indicators would evaluative functions by issuing Contextualized. In addition to numerical show how a journal performs across all corrections and retractions. statistics, indicators should report statistical functions. Indicators around curating, for ●●Disseminating. By making the work distributions (for example, of article citation example, might consider the expertise and public, a journal formally distributes it counts), as has been done in the Journal Cita- diversity of the editorial board as well as the to a specialist community; with open tion Reports since 2018 (ref. 10). Differences acceptance rate of submitted papers and the access and other communication across disciplines should be considered. transparency of acceptance criteria. Indi- tools, the journal makes the work Informed. Professional societies and cators around data (such as data citations available to broader communities. relevant specialists should help to foster or reporting standards) will become more ●●Archiving. By associating work with literacy and knowledge about indicators. important with the advance of open science adequate metadata and making it For example, a PhD training course could and independent analysis. Indicators around available online and to indexes and include a role-playing game to demonstrate evaluating research might consider transpar- aggregators, the journal contributes the use and abuse of journal indicators in ency of the process, as well as the number to the permanent scholarly record career assessment. and diversity of peer reviewers and their and facilitates discovery. P.W. et al. Responsible. All stakeholders need to be timeliness. alert to how the use of indicators affects the behaviour of researchers and other stakehold- CLEAR CRITERIA FIT FOR PURPOSE ers. Irresponsible uses should be called out. Having more indicators does not equate The Journal Citation Reports, presenting to having better ones. We must also ensure the JIF and other journal indicators, were GOOD GOVERNANCE that new indicators are constructed and used conceived in 1975 as a summary of journals’ All stakeholders in the system share responsibly2,3. Improved indicators should citation activity in the Science Citation Index responsibility for the appropriate construc- be: valid (reflecting the concept meas- (now owned by Clarivate Analytics in Phil- tion and use of indicators, but in different ured); understandable; transparent (data adelphia, Pennsylvania). It was specifically ways. We therefore suggest the creation of an underlying criteria should be released, with intended to support librarians who wanted inclusive governing organization that would clearly explained limitations and degrees of to evaluate their collections and researchers focus on journal indicators. uncertainty); fair (systematic bias should who wished to choose appropriate publica- The governing body could propose new be avoided); adaptive (updated when bias, tion venues, as well as to provide insights for indicators to address the various functions abuse or other weaknesses become appar- scholars, policymakers and research evalua- of scholarly journals, make recommenda- ent); and reproducible (those who use the tors. Its inventors never expected the broad tions on their responsible use and develop indicator should be able to reproduce it). use and rampant misuse that developed4 (see standards. It could also create educational We think that these criteria will apply also go.nature.com/30teuoq). material (such as training in the ethics of even as research publishing changes. For Indicators, once adopted for any type of indicator development and use) and serve as example, we can imagine a future in which evaluation, have a tendency to warp prac- a place for people to publicize questionable the record of scholarly work includes, and tice5. Destructive ‘thinking with indicators’ uses of and good practices concerning indi- credit is attributed for, units smaller than (that is, choosing research questions that cators. For example, it could help to protect an individual publication. The principles are likely to generate favourable metrics, researchers against ‘predatory journals’ — above could apply to any unit of scholarly rather than selecting topics for interest and typically low-quality publications that do not work that is being tracked. One existing importance) is becoming a driving force conduct peer review or curate information example is citation of the individual data of research activities themselves. It dis- as promised, and that exist only for financial sets behind a specific figure in a paper (as courages work that will not count towards gain. The body could also give guidance on implemented, for instance, by EMBO’s that indicator. Incentives to optimize a open-access publishing and data sharing. SourceData initiative), which can include a single indicator can distort how research The organization of the governing subset of a publication’s authors. is planned, executed and communicated6. body could mirror successful examples

in scholarly publishing, such as the non-profit organizations Crossref and ORCID, which provide unique identifi- ers for articles and authors, respectively. It would be international in composition and would liaise among various stake- holder groups, including coordinating STEVE GSCHMEISSNER/SPL STEVE with various relevant initiatives, such as DORA, the UK Forum for Respon- sible Research Metrics and the Com- mittee on Publication Ethics. Members could include individuals from across the scholarly communication system, drawn from scholarly societies, com- mercial and non-profit publishers, higher-education institutes, research funders, government and elsewhere. We invite all interested stakeholders to contact us to join this initiative. On the basis of these responses, we aim to launch the governing body at a second workshop in 2020. Critics will counter that any incen- tive system will be vulnerable to gaming. However, we hope that the principles articulated here serve to work against pathologies and hijacking of our goals. Also, gaming multiple indicators would be much more difficult than gaming today’s homogeneous metrics. Scientific publishing is taking on new functions and becoming more open to the public. A new generation of journal indicators must support the diverse roles of publishers and incentivize good performance. ■ Coloured scanning micrograph of a community of bacteria from the nose. Paul Wouters is professor of scientometrics and dean of the Faculty of Social and Behavioural Sciences at Leiden University, the Netherlands. Cassidy What’s next for the R. Sugimoto, Vincent Larivière, Marie E. McVeigh, Bernd Pulverer, Sarah de Rijcke, Ludo Waltman. human microbiome? e-mail: [email protected]

1. Sugimoto, C. et al. F1000Research https:// Medical insights will flow from dissecting doi.org/10.7490/f1000research.1116751.1 host–microbe interactions using ecological and (2019). 2. Wilsdon, J. et al. The Metric Tide: Report of evolutionary approaches, argues Lita Proctor. the Independent Review of the Role of Metrics in Research Assessment and Management (2015). 3. Gingras, Y. Bibliometrics and Research ver the past decade, more than comes from the United States. Some 20% of Evaluation: Uses and Abuses (MIT Press, 2016). 4. Larivière, V. & Sugimoto, C. R. in Springer US$1.7 billion has been spent on this has gone to two phases of the Human Handbook of Science and Technology Indicators human microbiome research. Microbiome Project (HMP), which is (eds Glänzel, W., Moed, H. F., Schmoch, U. & OMajor projects are under way in the United creating the research resources needed for Thelwall, M.) (Springer, in the press). States, the European Union, China, Canada, studying the human microbiome (see ‘Big 5. Campbell, D. T. Eval. Program Plan. 2, 67–90 4 (1979). Ireland, South Korea and Japan. spend’). A review of what that decade of 6. Müller, R. & de Rijcke, S. Res. Eval. 26, This investment has confirmed the investment in human microbiome research 157–168 (2017). importance of the microbiome to human has achieved was published in February (see 7. Fong, E. A. & Wilhite, A. W. PLoS ONE 12, e0187394 (2017). health and development. It is now known, ‘Big wins’). And findings from the second 8. Hicks, D., Wouters, P. Waltman, L., for instance, that newborns receive essential phase of the HMP are published in this week’s de Rijcke, S. & Rafols, I. Nature 520, 429–431 microorganisms from their mothers1. More- Nature (see pages 641, 655 and 662)5–8. (2015). 9. McKiernan, E. C. et al. PeerJ Preprints 7, over, the sugars in breast milk that infants In my view, most of the research so far e27638v2 (2019). cannot digest nourish babies’ developing has placed too much emphasis on catalogu- 10. Larivière, V. et al. Preprint at bioRxiv https:// microbiomes2, which in turn shape their ing species names. We’ve been character- doi.org/10.1101/062109 (2016). immune systems3. izing the human microbiome as if it were A list of co-signatories accompanies this Now is a good moment for reflection. The a relatively fixed property to be mapped Comment online (see go.nature.com/2wfeyjc). biggest investment made (around $1 billion) and manipulated — one that is separate