Critique of Hirsch’s Citation Index: A Combinatorial Fermi Problem Alexander Yong n 2005, physicist J. E. Hirsch [Hi05] pro- desires a quantitative baseline for what “compara- posed the h-index to measure the quality ble,” “very different,” and “similar” actually mean. of a researcher’s output. This metric is the Now, while this would appear to be a matter for largest integer n such that the person has statisticians, we show how textbook combinatorics n papers with at least n citations each, and sheds some light on the relationship between Iall other papers have weakly less than n citations. the h-index and Ncitations. We present a simple Although the original focus of Hirsch’s index was model that raises specific concerns about potential on physicists, the h-index is now widely popular. misuses of the h-index. For example, Google Scholar and the Web of Science To begin, think of the list of a researcher’s highlight the h-index, among other metrics such citations per paper in decreasing order λ = (λ1 ≥ as total citation count, in their profile summaries. λ2 ≥ · · · ≥ λNpapers ) as a partition of size Ncitations. An enticing point made by Hirsch is that the Graphically, λ is identified with its Young diagram. h-index is an easy and useful supplement to a • • For example, λ = (5; 3; 1; 0) $ . citation count (Ncitations), since the latter metric • • may be skewed by a small number of highly cited papers or textbooks. In his words: A combinatorialist will recognize that the h- “I argue that two individuals with similar index of λ is the side-length of the Durfee square h’s are comparable in terms of their overall (marked using •’s above): this is the largest h × h scientific impact, even if their total number square that fits in λ. This simple observation of papers or their total number of citations is nothing new, and appears in both the bib- is very different. Conversely, comparing two liometric and combinatorial literature; see, e.g., individuals (of the same scientific age) with [AnHaKi09, FlSe09]. In particular, since the Young a similar number of total papers or of total diagram of size Ncitations with maximum h-index citation count and very different h values, is roughly a square, we see graphically that the one with the higher h is likely to be the p 0 ≤ h ≤ b Ncitationsc. more accomplished scientist.” Next, consider the following question: It seems to us that users might tend to eyeball Given Ncitations, what is the estimated range of h? differences of h’s and citation counts among in- Taking only Ncitations as input hardly seems dividuals during their assessments. Instead, one like sufficient information to obtain a meaningful Alexander Yong is associate professor of mathematics at answer. It is exactly for this reason that we call the University of Illinois at Urbana-Champaign. His email the question a combinatorial Fermi problem, by address is [email protected]. analogy with usual Fermi problems; see the section DOI: http://dx.doi.org/10.1090/noti1164 “Combinatorial Fermi Problems.” 1040 Notices of the AMS Volume 61, Number 9 Table 1. Confidence intervals for h-index. Ncitations 50 100 200 300 400 500 750 1000 1250 Interval for h [2, 5] [3, 7] [5, 9] [7, 11] [8, 13] [9, 14] [11, 17] [13, 20] [15, 22] 1500 1750 2000 2500 3000 3500 4000 4500 5000 5500 [17, 24] [18, 26] [20, 28] [22, 31] [25, 34] [27, 36] [29, 39] [31, 41] [34, 43] [35, 45] 6000 6500 7000 7500 8000 9000 10000 [36, 47] [37, 49] [39, 51] [40, 52] [42, 54] [44, 57] [47, 60] Table 2. Fields medalists 1998–2010. Medalist Award year Ncitations h Rule of thumb est. Confidence interval T. Gowers 1998 1012 15 17.2 [13, 20] R. Borcherds 1998 1062 14 17.6 [14, 21] C. McMullen 1998 1738 25 22.5 [18, 26] M. Kontsevich 1998 2609 23 27.6 [22, 32] L. Lafforgue 2002 133 5 6.2 [4, 8] V. Voevodsky 2002 1382 20 20.0 [16, 23] G. Perelman 2006 362 8 10.0 [7, 12] W. Werner 2006 1130 19 18.2 [14, 21] A. Okounkov 2006 1677 24 22.1 [18, 25] T. Tao 2006 6730 40 44.3 [38, 51] C. Ngô 2010 228 9 8.2 [5, 10] E. Lindenstrauss 2010 490 12 12.0 [9, 14] S. Smirnov 2010 521 12 12.3 [9, 15] C. Villani 2010 2931 25 29.2 [24, 33] Since we assume no prior knowledge, we con- “The Euler–Gauss Identity and its Application to sider each citation profile in an unbiased manner. the h-index.” That is, each partition of Ncitations is chosen with The asymptotic result we use, due to equal probability. In fact, there is a beautiful the- E. R. Canfield–S. Corteel–C. D. Savage [CaCoSa98], ory concerning the asymptotics of these uniform gives the mode size of the Durfee square when random partitions. This was largely developed Ncitations ! 1. Since their formula is in line with !!Not Supplied!! !!Not Supplied!! Notices of the AMS 1 by A. Vershik and his collaborators; see, e.g., the our computations, even for low Ncitations, we survey [Su10]. reinterpret their work as the rule of thumb for Actually, we are interested in “low” (practi- h-index: cal) values of where not all asymptotic p Ncitations 6 log 2 p p results are exactly relevant. Instead, we use gen- h = Ncitations ≈ 0:54 Ncitations: π erating series and modern desktop computation The focus of this paper is on mathematicians. For to calculate the probability that a random λ has the vast majority of those tested, the actual -index Durfee square size h. More specifically, we ob- h tain Table 1 using the Euler–Gauss identity for computed using MathSciNet or Google Scholar falls partitions: into the confidence intervals. Moreover, we found that the rule of thumb is fairly accurate for pure 1 2 Y 1 X xk mathematicians. For example, Table 2 shows this (1) = 1 + : !!Not1 Supplied!!− xi !!NotQ Supplied!!k − j 2 Noticesfor post-1998 of the Fields AMS medalists.1 1 i=1 k≥1 j=1(1 x ) The proof of (1) via Durfee squares is regularly 1 taught to undergraduate combinatorics students; Citations pre-2000 in MathSciNet are not complete. Google scholar and Thompson Reuters’ Web of Science also have it is recapitulated in the section “The Euler–Gauss sources of error. We decided that MathSciNet was our most Identity and its Application to the h-index.” The complete option for analyzing mathematicians. For rela- pedagogical aims of this note are elaborated upon tively recent Fields medalists, the effect of lost citations is in sections “Combinatorial Fermi Problems” and reduced. October 2014 Notices of the AMS 1041 In [Hi05] it was indicated that the h-index Euler–Gauss Identity ::: ”) asserts concentration has predictive value for winning the Nobel Prize. around the mode Durfee square. Thus, theoretically, However, the relation of the h index to the Fields one expects the rule of thumb to be nearly correct Medal is, in our opinion, unclear. A number of the for large Ncitations. medalists’ h values above are shared (or exceeded) Alas, this is empirically not true, even for pure by noncontenders of similar academic age, or with mathematicians. However, we observe something p those who have similar citation counts. Perhaps related: 0:54 Ncitations is higher than the actual this reflects a cultural difference between the h for almost every very highly cited (Ncitations > mathematics and the scientific communities. 10; 000) scholar in mathematics, physics, computer In the section “Further Comparisons with Em- science and statistics (among others) we considered. pirical Data,” we analyze mathematicians in the On the rare occasion this fails, the estimate is National Academy of Sciences, where we show that only beat by a small percentage (< 5 percent). The the correlation between the rule of thumb and drift in the other direction is often quite large (50 actual h-index is R = 0:94. After removing book percent or more is not unusual in certain areas of citations, R = 0:95. We also discuss Abel Prize engineering or biology). winners and associate professors at three research Near equality occurs among Abel Prize winners. universities. We also considered all prominent physicists high- Ultimately, readers are encouraged to do checks lighted in [Hi05] (except Cohen and Anderson, due of the estimates themselves. to name conflation in Web of Science). The guess is We discuss three implications/possible applica- always an upper bound (on average 14–20 percent tions of our analysis. too high). Near equality is met by D. J. Scalapino (25; 881 citations; 1:00), C. Vafa (22; 902 citations; 0 99), J. N. Bahcall (27 635 citations; 0 98); we have Comparing h’s when Ncitations’s Are Very : ; : given the ratio true h . Different estimated h One reason for highly cited people to have a It is understood that the h-index usually grows lower than expected h-index is that they tend with . However, when are citation counts Ncitations to have highly cited textbooks. Also, famous so different that comparing ’s is uninformative? h academics often run into the “Matthew effect” For example, = 40 (6 730 citations) while hTao ; (e.g., gratuitous citations of their most well-known = 24 (1 677 citations). The model asserts hOkounkov ; articles or books). the probability of hOkounkov ≥ 32 is less than 1 in 10 million. Note the Math Genealogy Project has Anomalous h-Indices fewer than 200; 000 mathematicians. These orders of magnitude predict that no More generally, our estimates give a way to flag an anomalous h-index for active researchers; i.e., mathematician with 1; 677 citations has an h- those that are far outside the confidence interval index of 32, even though technically it can be as or those for which the rule of thumb is especially high as 40.