Pitfalls of Bibliometrics: What are the scientometric data good for? Karol Życzkowski (Jagiellonian University and Center for Theoretical Physics, PAS)
Polish Academy of Science Warsaw, April 9, 2013 my scientific interests: - Nonlinear Dynamics also - Mathematical Physics - Statistical Physics - Quantum Chaos - Voting Theory - Quantum Information - Bibliometrics Nobel Prize in Physics 2012 Nobel Prize 2012 Papers Citations Index H Serge Haroche Fr 219 16350 65 David J. Wineland USA 260 21205 71
S. Haroche D. Wineland Experiments on control of photons and atoms, of key importance for foundations of quantum theory Fields Medal 2010 (Mathematics)
E. Lindenstrauss
B.C. Ngo
Fields Medal 2010 Papers Citations Index H Elon Lindenstrausso Israel 34 364 9 Bao Chau Ngo Viet/Fr 9 121 7 Stanislav Smirnov Ru/Sui 27 461 8 Cedric Villani Fr 51 919 14 S. Smirnov
C. Villani
Fields Medal 1998 (Mathematics) R. Borcherds
T. Gowers
Fields Medal 1998 Papers Citations Index H Richard Borcherds UK 20 1302 13 Timothy o Gowers UK 29 744 14 Maxim Kontsevich Ru/Fr 42 2265 18 Curtis McMullen USA 46 790 18
M. Kontsevich
C. McMullen an explanation : Standards in various fields differ! Mathematicians: - publish less papers a year - quote less references in each article - the average number of the authors of a paper is smaller in comparison with physicists !
Conclusion: One should not compare bare data from different fields ! (but rather use data rescaled to the average...) A case example : Consider a (productive) mathematician writing alone 10 articles during last 10 years, and a physicist writing jointly with 4 other colleagues five papers each year, so his 10-year list of publications consists of 50 items. What is a) the amount of work done ? b) estimated increase of their number of citations and e.g. their H-index ? Comparison of different fields I (Data: USA 2006 - 2011)
Authors Articles Citations Citations Citations Articles Scientists per per year per per year per compared Field per year (USA) article per author article scientist to Math. Mathematics 4190 37000 2.0 0.2 15.0 3.5 1 Physics 18227 49000 5.3 2.0 27.8 54.8 15.9 Chemistry 16430 86000 4.3 0.8 31.7 26.0 7.5 Computer Sci. 2188 20000 3.0 0.3 17.4 5.8 1.7 Engineering 14609 144000 3.8 0.4 18.7 7.3 2.1 Space Science 3187 5000 5.9 3.8 41.2 154.9 44.9 Geosciences 11621 21000 4.0 2.2 33.1 73.0 21.2 Agriculture 3469 22000 4.3 0.7 20.8 14.2 4.1 Biological Sci. 49614 193000 5.3 1.4 41.1 55.8 16.2 Medical Science 58664 45000 5.6 7.3 43.2 315,3 91.4 Psychology 9805 114000 3.2 0.3 46.5 13.0 3.8 Social Science 12020 100000 1.9 0.2 26.8 6.2 1.8 Comparison of different fields II Citations of 100-th highly cited scientist (Web of Science, ESI 2002-20012) Citations Relative Citations Relative of 100-th to of 100-th to Field scientist Mathem. Field scientist Mathem. Mathematics 733 1.0 Clinical Physics 14772 20.2 Medicine 17051 23.3 Chemistry 12420 16.9 Economics 960 1.3 Computer Sci. 1247 1.7 Ecology 3013 4.1 Engineering 3165 4.3 Immunology 4169 5.7
Space Science 9700 13.2 Materials Sci. 4750 6.5 Geosciences 3571 4.9 Microbiology 3244 4.4 Agricultural Sci. 1288 1.7 Molecular Biol. 9021 12.3 Biology & Bioch. 6092 8.3 Neuroscience 5781 7.9 Psychiat./Psycho. 3256 4.4 Pharmacology 2081 2.8 Social Sciences 948 1.3 Plant & Animal 3114 4.3 The above numbers are only approximate and should not be treated as convertion coefficients! The reason: differences inside a given field Example: 1.0 Pure
Mathematics 20.2 Particle Physics
Applied Math. Mathematical Physics Entire spectrum of intermediate cases exists…
Statistics of citations in scientific papers What are they good for?
• some people (librarians, publishers, editors) do care about bibliometric indices, e.g. impact factor (IF-2), E. Garfield 1972
IF(i) = ci /(Zi-1 + Zi-2)
ci = the number of citations gained in year i by articles published in i-1 and i-2.
Zi = the number of articles published in i. • can we afford the luxury to neglect it ?? perhaps not... 2008 Impact factor for exemplary journals selected out of 6220 covered by Journal Citatation Report
Cancer J. Clin. 74.6 Phys . Rep. 16.3 New Engl. J. Med 50.0 Phys. Rev. Lett. 7.2 Ann. Rev. Immun. 41.1 Phys. Rev. D 5.1 Rev. Mod. Phys. 34.0 Ann. Math. 3.5 Nature 31.4 Phys. Rev. E 2.5 Cell 31.2 J. Am. Math. Soc 2.4 Lancet 28.4 J. Phys. A 1.5 Science 28.1 Duke. Math. J. 1.4 Ann. Rev. Astr. 24.3 Acta. Phys. Pol.B 0.8 Chem. Rev. 23.6 Rep. Math. Phys. 0.6 Mean Impact factor that a paper will be quoted in 2-years data from Journal of Citation Reports for 1994-2005 Field Size [%] Biology 511 4.76 45.8 20.5 Astronomy 25 4.29 38.3 21.5 Medicine 766 2.89 33.9 18.3 Chemistry 145 2.61 33.1 17.0 Physics 503 1.91 24.0 16.7 Economics 159 0.82 30.4 12.1 Comp. Scien. 124 0.63 17.2 19.3 Mathematics 149 0.56 18.4 8.5 History 23 0.41 81.8 10.1 Althouse, West, Bergstrom, 2008 Data from Journal of Citation Reports for 1994-2005 mean Impact Factor mean number of citations Althouse, West, Bergstrom, 2008 Universality of citation distribution P(cf) Radicchi, Fortunato and Castellano, 2008 P(c) • Study of citation patterns in 19 scientific fields P(c) c • In each discipline one finds the mean citation rescaled citation number cf =c/ … and different publication years cf Rescaled data allow for comparison across different fields and … cf Traps and pitfalls related to citation statistics R. Adler, J. Ewing, P. Taylor, Report of Joint Committee on Quantitative Assessment of Research, November 2008 Distribution of citations for a given paper is given by a power law, P(k) = a k -b, (Garfield 1987) k Example a): 2005 data for Proceedings of AMS imply impact factor IF= 0.43 Example b): 2005 data for Transactions of AMS imply impact factor IF= 0.85 Question: Is an average Transaction paper twice as good as the one published in Proceedings ? • What is the probability that a random Proceedings paper as at least as many citations as the one from Transactions? • ? Example b): 2005 data for Transactions of AMS imply impact factor IF= 0.85 Question: Is an average Transaction paper twice as good as the one published in Proceedings ? • What is the probability that a random Proceedings paper as at least as many citations as the one from Transactions? • the answer is: 62% We are more often wrong than right ! • The 2008 report of Adler, Ewing, Taylor criticise sole reliance on the impact factor, since the ‘objectivity’ of such numbers can be illusory. They compare the judgments of a journal based on its Impact Factor alone to using the weight of a patient to judge his health… • See also: E. Falgas and V. Alexiou: The top-ten in journal impact factor manipulations Arch. Immunol. Theor. Exp. 2008 Why Impact Factor of journals should not be used for evaluating research P. O. Seglen, British Medical J. 1997 IF does not reflect the squewness of the distri- bution (50% of citations are gained by 10% papers). IF depends on the field. Article citations determine the journal IF, but not vice versa! IF depends on the mean number of references in each article. Adding Impact Factor of journals in which all papers of a given author where published is a capital crime against the rules of bibliometry !! (would you sum e.g. the $$$ price of each issue?) • If one considers citations as a quality indicator, one should rather care about the average (true) impact factor of his papers (older than 3 years) defined as the sum of citations his articles published in year i gained in years i+1 and i+2 . = his direct contribution to IF of the journals ! ? ? Are all citations equal ? • Some experts believe that even mediocre articles can make it to the list of often cited papers: just because they are easy to understand and simple to quote... • Is a citation of my work by an expert in the field more valuable than the one by a beginner ? • Perhaps the value of any citation should depend on the scientific output of the citing author.. • Page Rank algorithm of Google ! Various constructions of the graph Each node of the graph represents: a) an individual article Chen, Xie, Maslov, Redner ‘07 Ma, Guan, Zhao ‘08 b) an individual author * K. Życzkowski, ‘Citation graph, weighted impact factors and performance indices’, Scientometrics 2010 * Radicchi et al. Sept. 2009 c) a single scientific journal, Bergstrom 2007 Page Rank algorithm: • One forms a citation graph, the corresponding citation matrix G of size N • One normalises each column of G to obtain a stochastic matrix G’ • To assure that G’ is irreducible (and increase the spectral gap) one defines PageRank matrix H = a G’ +(1-a) W where Wij =1/N, (in Google a =0.85) • Page rank is defined by components of the leading eigenvector of H (Frobenius-Perron !) • Bergstrom C, Eigenfactor: measuring the value and prestige of scholarly journals. C&RL News 68, 314–316 (2007) • Advantages: insensitivity to field differences - insensitivity to insignificant journals free available at: • http://www.eigenfactor.org Mathematical Journals: SCI - JCR 2011 data Eigenfactor Article Influence Score (EF-AIS) characterizes journals better than Impact Factor No. Journal IF-2 IF-5 EF –AIS 1 J AM MATH SOC. 3.8 3.2 5.3 2 FOUND COMPUT MATH 3.6 2.9 3.0 3 ACTA MATH-DJURSHOLM 3.3 4.0 5.9 4 ANN MATH 2.9 3.3 5.0 5 COMMUN PUR APPL MATH 2.6 3.9 4.1 6 INVENT MATH 2.3 2.9 4.5 ... … ….. … … … 10 FIXED POINT THEORY 1.6 1.8 0.7 11 J DIFFER GEOM 1.6 1.5 2.2 12 DUKE MATH J 1.5 1.7 2.8 13 NONLINEAR ANAL-THEOR 1.5 1.6 0.7 14 ANN SCI ECOLE NORM 1.4 1.8 3.2 Impact Factor vs Eigenfactor Article Influence Score journal IF-2 AIS Rev. Mod. Phys. 43.9 28.9 for exemplary Cancer J. Clin. 67.4 24.5 journals : Ann. Rev. Immun. 42.9 23.4 New Engl. J. Med. 50.7 21.3 2011 data Cell 34.7 20.5 Nature 36.2 20.3 ISI Lancet 38.2 13.6 Journal Science 31.2 17.5 Citatation Chem. Rev. 40.1 13.3 Report Econometrica 3.0 8.6 Astronom. Rev. 11.5 8.1 J. Am. Math Soc. 3.8 5.2 Is the number of citations really an objective measure of the scientific quality of an article ?? Several authors formulate various arguments against this statement: e.g. some simple papers are quoted more often than deep articles… while we* analyze data from our own lists of publications to contribute to this discussion ------* B. Kamys, M. Kuś, J. Zakrzewski, K. Życzkowski A humble example I : Number of citations & IF 150 of 100 number 50 citations Total Total 0 0,00 2,00 4,00 6,00 8,00 Impact Factor (IF) of a journal Number of citations of 100 selected papers (in quantum physics) by two authors (KZ and JZ) is weakly correlated with Impact Factor of the journal they appeared A humble example II : Number of citations vs scientific quality 8 6 4 2 0 Scientific Scientific quality 0 50 100 Number of citations of an article Number of citations of selected papers is only weakly correlated with their scientific quality (assesd a posteriori by the authors in scale: [0,10]) A humble example III : Scientific quality vs Impact Factor 8 6 S… 4 2 0 Scientific quality Scientific 0,00 2,00 4,00 6,00 8,00 Impact factor (IF) of a journal Scientific quality of selected papers (assesed a posteriori by the authors) is very weakly correlated with the Impact Factor of the journals they appeared Remarks on tools used: We often work with ISI Web of Science (standard in several fields !), but • Mathematicians prefer MathSciNet • Some physicists like Scopus or • + Publish or Perish is free • Eigenfactor Article Influence is well normalized and much more reliable than IF ISI Web of Science : negligence concerning (some) European journals Example: Reports of Mathematical Physics (RoMP) - best Polish journal in the field, edited by Copernicus University, Toruń: Articles published in 1970-1980 are recorded in ISI database, but do not appear in an author’s citation summary: citations Jamiołkowski A, Rep. Math. Phys. 1972 275 Kijowski J, Rep. Math. Phys. 1974 81 ISI Web of Science : not significant (= redundant) data Calculation of 2-year Impact Factor IF-2 Example: Reproductive Systems Journals Oxford Rev. Reprod. B: IF=1.765 =C/D C= 30 (cites in 1992 to papers from 1990-91 D=17 (articles published in 1990-91) If all articles had jointly one citation more C’=31, IF=1.82 => last digits of IF=1.765 are not significant !! A. Molinié, G. Bodenhausen Bibliometrics as Weapons of Mass Citation Chimia 2010 Since we fear that decision-takers of granting agencies will be too busy to read our humble paper in Chimia, we appeal to scientists of all countries and disciplines to unite against the tyranny of bibliometrics Role of the bibliometric approach i) Bibliometric data have statistical character, so they can be used to compare scientific output in a single country or progress in a given branch of science. ii) Bibliometric data data are less useful to evaluate research record of an individual scientist and they allow for a reasoning in one direction only: - an author with a small number of papers and citations is unlikely to be an influential scientist, - a huge number of papers and citations does not prove that the author is a prominent scientist => peer review is required ! Concluding remarks I I.1) To characterize quality of journals Impact Factor (IF) should be replaced by modern tools, e.g. Eigenfactor Article Score which is: a) normalized (average value is equal to unity) b) insensitive to differencies between the fields c) much more difficult to manipulate than IF d) available free on line ! I.2) Impact Factor should be ruled out from any process of evaluation of scientific record of a single scientist or his grants proposals… I.3) Comparing two scientists from different fields one cannot use bare bibliometric data. Concluding remarks II.a a) for researchers: * Do well your research, write fine papers and publish them in good journals. * Do not care much about impact factors, citations, benchmarks and indices… * Do not use your time and energy to enter the game to inflate your bibilometric indices: ** A good scientist will have healthy numbers according to any bibliometric measure... Concluding remarks II.b b) for reviewers: • During the evaluation procedure treat any bibliometric numbers as auxiliary data only. • bureaucrats love to play with a single numerical indicator (as it often remains as an only tool for them!) • You should take advantage of your deep knowledge of the field: look into the essence of the proposal before looking at any parametric data… Concluding remarks II.c c) for managers of science: • Science has multiple goals – do not try to project the multi-dimensional system into a single axis! • Do not hope to get a single bibliometric parameter universally suitable for evaluation. • Usage of several bibliometric indicators in parallel reduces the risk of data manipulation. • Support versatile usage of bibliometry, in which evaluated researchers take active part. example: benchmarks used in ERC grants: Applicant selects his ten best papers published in the last decade and provides the number each of them was cited. Bibliometrics: Use and Abuse in the Review of Research Performance Stockholm, May 2013 Critique in a Nutshell • Researchers today face an overload of evaluation activities of all kinds, • Time pressure that may jeopardize the quality of assessments, • A heavy bias disqualifying humanities and social sciences which do not fit in the system, • An increasing imbalance between the costs of the evaluation system, the advancement of science, and the profits made by private companies The very last final remark: If necessary, do use bibliometric data, but do it in a reasonable way! ! Thank You !