The Leiden Manifesto in the Making

THE LEIDEN MANIFESTO IN THE MAKING FULL REPORT OF THE PLENARY SESSION AT THE 2014 STI CONFERENCE IN LEIDEN ON QUALITY STANDARDS FOR EVALUATION: ANY CHANCE OF A DREAM COME TRUE?

ISMAEL RAFOLS SARAH DE RIJCKE PAUL WOUTERS Ingenio (CSIC-UPV) CWTS, CWTS, SPRU, Sussex Leiden University Leiden University

SHORT COMMUNICATIONS, ARTICLES COMMUNICATIONS, SHORT SUMMARY A set of guiding principles (a manifesto) on the use of quantitative metrics in research assessment was proposed by Diana Hicks (Georgia Tech) during a panel session on quality standards for S&T indicators at the STI conference in Leiden last week. Various participants in the debate agreed on the responsibility of the scientometric community in better supporting use of scientometrics. Finding the choice of specific indicators too constraining, many voices sup- ported the idea of a joint publication of a set of principles which should guide a responsible use of quantitative metrics. The session also included calls for scientometricians to take a more proactive role as engaged and responsible stakeholders in the development and monitoring of metrics for research assessment, as well as in wider debates on data governance of, such as infrastructure and ownership. In the closure of the conference, the association of scientometric institutes ENID (European Network of Indica- tors Designers) and Ton van Raan as president, offered to play a coordinating role in writing up and publishing a consensus version of the manifesto.

The need to debate these issues has come to its validity. A central aim of the special ses- the forefront in light of reports that uses of sion was to discuss the need for a concerted certain easy-to-use and potentially mislead- response from the scientometric community ing metrics for evaluative purposes have be- to produce more explicit guidelines and ex- come a routine part of academic life, despite pert advice on good scientometric practices. misgivings within the profession itself about The session continued from the 2013 ISSI and

ISSI NEWSLETTER VOL. 10. NR. 3. © International Society for Scientometrics and Informetrics 60 STI conferences in Vienna and Berlin, where As an academic as well as a regulatory disci- full plenary sessions were convened on the pline, scientometrics not only creates reliable need for standards in evaluative bibliomet- knowledge on metrics, but also produces so- rics, and the ethical and policy implications cial technologies for research governance. As of individual-level bibliometrics. such, evaluative metrics attain meaning in a This year’s plenary session started with a certain context, and they also help shape that summary by Ludo Waltman (CWTS) of the context. Though parts of the community pre-conference workshop on technical as- now acknowledge that there is indeed a ‘so- pects of advanced bibliometric indicators. cial’ problem, ethical issues are often either The workshop, co-organised by Ludo, was conveniently bracketed off or ascribed to ‘us- attended by some 25 participants, and top- ers lacking knowledge’. This reveals unease ics that were addressed included 1. Advanced with taking any other-than-technical re- bibliometric indicators (strengths and weak- sponsibility. Sarah plugged the idea of a short nesses of different types of indicators; field joint statement on proper uses of evaluative normalization; country-level and institution- metrics, proposed at the international work- al-level comparisons); 2. Statistical inference shop at OST in Paris (12 May 2014; http://bit. in bibliometric analysis; and 3. Journal impact ly/YsST6Y). She concluded with a plea for a metrics (strenghts and weaknessess of differ- more long-term reconsideration of the field’s ent journal impact metrics; use of the metrics normative position. If the world of research in the assessment of individual researchers). governance is indeed a collective responsi- The workshop discussions were very fruitful bility, then scientometrics should step up and some common ground was found, but and accept its part. This would put the com- that there also remained significant differenc- munity in a much better position to actually es in opinion. Some topics that need further engage productively with stakeholders in the discussion are technical and mathematical process of developing good practices. properties of indicators (e.g., ranking consist- In the ensuing panel discussion, Stephen ency); strong correlations between indicators; Curry (professor of Structural Biology at the need to distinguish between technical is- Imperial College, London, and member of sues and usage issues; purely descriptive ap- HEFCE steering group) expressed a deep con- proaches vs. statistical approaches, and the cern about the seducing power of metrics in importance of user perspectives for technical research assessment and saw a shared, col- aspects of indicator production. There was a lective responsibility for the creation and use clear interest in continuing these discussions of metrics on the side of bibliometricians, re- SHORT COMMUNICATIONS, ARTICLES COMMUNICATIONS, SHORT at a next conference. The slides of the work- searchers and publishers alike. Thus accord- shop are available on request. ing to him technical and usage aspects of in- Ludo’s summary was followed by a short dicators should not be separated artificially. talk by Sarah de Rijcke (CWTS), to set the Lisa Colledge (representing Elsevier as scene for the ensuing panel discussion. Sa- Snowballmetrics project director) talked rah provided an historical explanation for about the Snowballmetrics initiative, and why previous responses by the scientometric presented it as a bottom-up and practical community about misuses of performance approach with the goal to meet the needs of metrics and the need for standards have funding organizations and university senior landed in deaf ears. Evoking Paul Wouters’ level management. According to Lisa, while and Peter Dahler-Larsen’s introductory and it primarily addresses research officers, feed- keynote lectures, she argued that the pre- back from the academic community of bib- ferred normative position of scientometrics liometrics is highly appreciated to contribute (‘We measure, you decide’) and the tenden- to the empowerment of indicator users. cy to provide upstream solutions no longer Stephanie Haustein (University of Mon- serve the double role of the field very well. treal) was not convinced that social me-

ISSI NEWSLETTER VOL. 10. NR. 3. © International Society for Scientometrics and Informetrics 61 dia metrics (a.k.a. altmetrics) lend itself to determine the standards itself. Yet standards standardization due to heterogeneity of data are being developed by database providers sources (tweets, views, downloads) and their and universities, now busy building up new constantly changing nature. She stated that research information systems. Wouters pro- meaning of altmetrics data is highly ambigu- posed that the scientometric community sets ous (attention vs. significance) and a quality as its goal to monitor and analyze evolving control similar to the peer review system in standards. This could help to better under- scientific publications does not yet exist. stand problems and pitfalls and also provide Jonathan Adams (Chief scientist at Digi- technical documentation. tal Science) approved the idea of setting up a The third issue highlighted by Wouters is statement but emphasized that it would have the question of who is responsible. While the to be short, precise and clear to also catch scientometric community cannot assume full the attention of government bodies, funding responsibility for all evaluations in which sci- agencies and senior level university manage- entometric data and indicators play a role, it ment who are uninterested in technical de- can certainly broaden out its agenda. Perhaps tails. Standards will have to live up to the fast- an even more fundamental question is how paced change (data availability, technological public stakeholders can remain in control innovations). He was critical of any fixed set of of the responsibility for publicly funded sci- indicators since this would not accommodate ence when more and more meta-data is being the strategic interests of every organization. privatized. Wouters pleaded for strengthen- Diana Hicks (Georgia Institute of Tech- ing the public nature of the infrastructure of nology) presented a first draft of a set of state- meta-data, including current research infor- ments (the “Leiden Manifesto”), which she mation systems, publication databases and proposed should be published in a top-tier citation indexes. This view does not deny the journal like Nature or Science. The statements important role for for-profit companies who are general principles on how scientometric are often more innovative. Fourth, Wouters indicators should be used, such as for exam- suggested that taking these issues together ple, ‘Metrics properly used support assess- provides an inspiring collective research ments; they do not substitute for judgment’ or agenda for the scientometrics community. ‘Metrics should align with strategic goals’. Diana Hicks’ suggestion of a manifesto In the ensuing debate, many participants or set of principles was followed up on the in the audience proposed initiatives and second day of the STI conference at the an- problems that need to be solved. They were nual meeting of ENID (European Network SHORT COMMUNICATIONS, ARTICLES COMMUNICATIONS, SHORT partially summarized by Paul Wouters who of Indicators Designers). The ENID assem- identified four issues around which the de- bly, and Ton van Raan as president, offered bate evolved. First, he proposed that a cen- to play a coordinating role in writing up the tral issue is the connection between assess- statement. Diana Hicks’ draft will serve as a ment procedures and the primary process basis, and it will also be informed by opinions of knowledge creation. If this connection is from the community, important stakehold- severed, assessments lose part of their useful- ers and intermediary organisations, as well ness for researchers and scholars. as those affected by evaluations. The debate The second question is what kind of on standardization and use will be continued standards are desirable. Who sets them? How in upcoming science policy conferences, with open are they to new developments and dif- a session confirmed for the AAAS (San José, ferent stakeholders? How comprehensive February) and expected sessions in the STI and transparent are or should standards and ISSI conferences in 2015. be? What interests and assumptions are included within them? In the debate it became (Thanks to Sabrina Petersohn clear that scientometricians do not want to for sharing her notes of the debate.)