Vol. 8: 49–60, 2008 ETHICS IN SCIENCE AND ENVIRONMENTAL POLITICS Printed June, 2008 doi: 10.3354/esep00092 Ethics Sci Environ Polit Published online May 27, 2008 Contribution to the Theme Section ‘The use and misuse of OPENPEN bibliometric indices in evaluating scholarly performance’ ACCESSCCESS Challenges for scientometric indicators: data demining, knowledge-flow measurements and diversity issues Michel Zitt1, 2,*, Elise Bassecoulard1 1Lereco U-1134, SAE2 department, INRA, BP 71627, 44316 Nantes Cedex 03, France 2Observatoire des Sciences et des Techniques (OST), 93 rue de Vaugirard, 75006 Paris, France ABSTRACT: Scientometrics and bibliometrics are being forced to respond to a strong increase in demand (e.g. research assessment practices, economics of science & technology, and innovation) and new forms of supply (e.g. availability of publication sources and statistics, Internet developments and online tools). This situation results in contrasting perspectives: on the one hand, it can favour spectac- ular ‘hit-parades’ and some veneration of numbers; on the other hand, it paves the way for more cau- tious and sophisticated evaluation systems, rooted in a better understanding of the dynamics of sci- ence. This paper describes some of the challenges for bibliometric indicators (data ‘demining’, knowledge-flow measurements and diversity issues) underlying, among other applications, reliable evaluation procedures. Responding to these challenges is necessary to promote a better use of scien- tometrics, although there are no guarantees against misuse in decision-making contexts. A few open issues are outlined on the dynamics of science, challenges of the web age, and interactions between scientometrics and scientific communities. KEY WORDS: Evaluation · Scientometrics · Bibliometrics · Indicators · Scientific networks · Multidisciplinarity Resale or republication not permitted without written consent of the publisher INTRODUCTION their own domain, standing at an intersection of disci- plines, evolved as a heterogeneous field, both in topics The aim of scientometrics is to provide quantitative and practices (Schoepflin & Glänzel 2001) and intellec- characterizations of scientific activity. Because of the par- tual repertoire (Peritz & Bar-Ilan 2002). ticular importance of publication in scientific communi- A certain tension has always existed between acade- ties, it largely overlaps with bibliometrics, which is quan- mic/cognitive scientometrics and political/practical titative analysis of media in any written form. In addition scientometrics, the latter of which has been described to disciplines of measurement (infometrics/ data-mining, as ‘a hybrid of social science and bureaucratic exper- statistics and mathematical modelling), scientometrics tise’ (Wouters 2006, p. 21). Often, these aspects can has strong connections with economics and sociology of hardly be disentangled. Scientometrics has to correctly science as well as science policy. The 1970s saw the represent the multiple facets of scientific activity in development of scientometrics as an operational activ- models of use to science policy makers, using quantita- ity — a response to the pressing demand for the ‘measur- tive tools with sound properties. ing of science’, especially in Russia and the USA. As with all decision-support disciplines, scientomet- Amongst the founding fathers of the discipline were de rics must resist the temptation of ‘l’art pour l’art’. A Solla Price (1963), Garfield (1955) and Narin (1976) in the superfluous sophistication is most likely to generate US, Nalimov & Mulczenko (1969) in Russia and Braun & artefacts and black-box effects, but the fascination of Bujdoso (1975) in Hungary. Applying bibliometric meth- ‘magic numbers’, chimerical syntheses in a unique ods to their own field, scientometricians confirm that index of a complex and multidimensional reality, can *Email: [email protected] © Inter-Research 2008 · www.int-res.com 50 Ethics Sci Environ Polit 8: 49–60, 2008 also be misleading. Even if precautions are taken and eation of scientific fields in the subsection ‘Delineation methodologies explicit, end-users (e.g. managers) tend and mapping of scientific areas’. to apply their own rules to bibliometric indicators as There are 2 incentives for the investigation of net- within any decision-making process — scientometri- works. The first is of an academic/cognitive nature. cians having no more control over this. Born from social and information scientists’ interest in understanding communities’ activity and information circulation, the quantitative tools used by scientometri- CONTEXT cians may in turn feed these disciplines. Scientometrics allows varied ways to describe knowledge circulation The bibliometric component of scientometrics is a networks and also the operationalization of hypotheses mirror of science: it uses the published works of scien- about scientific communities, generated by neighbour- tists to answer the questions of policy makers, stake- ing intellectual universes (sociology, economics or holders, scientists themselves, and social scientists tak- even physics). One example is the rising interest of ing research and science as a research object. knowledge economics in scientometric networks, link- Scientific publication is central to the activity of scien- ing academia to economic and social actors and to tific communities and is moreover made available on a political institutions. Data such as coauthorship, cita- large scale by modern databases — Garfield’s Science tions, hyperlinks, and migrations are assumed to Citation index (now Web of Science [WoS]) in the first describe circulation of information, knowledge and place — and the Internet. academic staff. Each publication is both the result and the imprint of The second incentive is the demand for evaluation. scientific networks, primarily social networks among The quest of the scientometrician is 2-fold: promoting scientists or institutions. As different as they may be in the most robust, reliable and acceptable methods and, their theoretical positions, Merton (1942), Bourdieu whenever possible, translating the strategic questions (1975) and Latour & Woolgar (1979) stressed the inter- of policy makers, stakeholders, and scientists into active character of scientific activity. Coauthorship, meaningful measures. citation, and hyperlinks are quite explicit elements of A lasting issue in evaluation is the apparent compe- networks appearing in bibliographic sources. Others tition of bibliometrics with peer-review, a topic cov- are implicit and can be revealed by analysis of textual ered by a huge literature with examples in this Theme elements, from title to full text, or geographical infor- Section (TS). As our focus here is on other aspects, we mation. All of these networks, with further information will not labour this particular point, and will only stress on dates and journals, offer a wealth of material for that scientometrics is not a deus ex machina, coming many types of analysis of scientific activity and knowl- from heaven or hell. It reflects the peer review process, edge circulation. albeit collectively and implicitly, that leads to the writ- Data on external resources, such as human resources ing of a paper, its acceptance by a journal, and its fur- and funding flows, are necessary to complete the land- ther citation. As shown by Wouters (1997), peers are scape. Thus, counting outputs (papers, cites) is the present at every stage of the research and publication very first stage of scientometrics, but assessment of sci- cycle: getting funds, attracting co-workers and co- entific productivity is extremely difficult. We will working, discussing and submitting manuscripts, and describe this issue in more detail in the next section. getting them read and cited. Peers are the hub of this Webs of all kinds, explicitly or implicitly created by wheel and bibliometrics is a mirror of peer-review. scientists, are observed by quantitative methods bor- Looking at this the other way around: peer-review of rowed from informetrics, statistics, data analysis and articles, or of dossiers of scientists applying for funding data-mining, and network theory. The scientometric or tenure, can hardly ignore bibliometric elements, mirror is sometimes distorting: mathematical resources such as records of publication, impact factor of journals are both powerful and rich in artefacts. This arsenal of (quite partial and dangerous when used alone; Seglen methods may be applied to any of the above- 1997), and real impact, especially when these elements mentioned networks. For example, if we wish to map are easily accessible through Thomson databases, Sco- scientific themes by grouping articles, we can design pus or Google Scholar. (1) a topical map, where proximity of articles is mea- Although some convergence is expected (Rinia et al. sured by the use of the same words, (2) a paradigmatic 1998, Aksnes & Taxt 2004, Harnad 2008 [this TS]), bib- map, where the proximity is measured by the use of liometrics and peer-review have their own strengths the same cited references, expressing the intellectual and weaknesses. Scientometrics dilutes many biases base of the article, or (3) an authorship-based map, present in ad hoc ‘peer-review’, such as individual spe- where proximity is measured by the presence of the cialization, personal interests and various pitfalls of same author(s). We will return to mapping and delin- group behaviour that can jeopardize a jury’s efficiency. Zitt & Bassecoulard: