The Journal Impact Factor Denominator Defining Citable (Counted) Items

Home , Citation impact, JAMA, Journal Citation Reports, Science Citation Index, The BMJ

COMMENTARIES

Marie E. McVeigh, MS The items counted in the denominator of the impact factor are identifiable in the Web of Science database by hav- Stephen J. Mann ing the index field document type set as “Article,” “Re- view,” or “Proceedings Paper” (a specialized subset of the VER ITS 30-YEAR HISTORY, THE JOURNAL IMPACT article document type). These document types identify the factor has been the subject of much discussion scholarly contribution of the journal to the literature and and debate.1 From its first release in 1975, bib- are counted as “citable items” in the denominator of the im- liometricians and library scientists discussed its pact factor. A journal accepted for coverage in the Thom- valueO and its vagaries. In the last decade, discussion has 6 son Reuters citation database is reviewed by experts who shifted to the way in which impact factor data are used. In consider the bibliographic and bibliometric characteristics an environment eager for objective measures of productiv- of all article types published by that journal (eg, items), which ity, relevance, and research value, the impact factor has been are covered by that journal in the context of other materi- applied broadly and indiscriminately.2,3 The impact factor als in the journal, the subject, and the database as a whole. has gone from being a measure of a journal’s citation influ- This journal-specific analysis identifies the journal sec- ence in the broader literature to a surrogate that assesses tions, subsections, or both that contain materials likely to the scholarly value of work published in that journal. These be considered scholarly works, and which therefore have misappropriated metrics have been used to assess indi- the potential to be cited. Among the features considered dur- vidual researchers, institutions, and departments. ing this review process are the following. Such evaluation practices put pressure on scholars to pub- Descriptive article titles: Most primary research and schol- lish in high-impact journals. To continue attracting manu- arly items highlight their content with highly descriptive titles scripts and authors, editors and publishers are pressured to denoting the subject of the study. This is also true of some, ensure a high impact factor for their journal.4 From a dis- but not all, reviews. cussion of uses and merits and improvements and expan- Named author(s) with author address: This information typi- sions of the impact factor, interest has turned to using pub- cally indicates that authors are affiliated with an academic, lishing and editorial practices to influence, alter, and even research, or other institution as opposed to paid contribu- manipulate the impact factor.5 tors (eg, a staff writer or reporter). The impact factor is calculated by considering all cita- Abstract or summary: Abstracts are another common fea- tions in 1 year to a journal’s content published in the prior ture of research and scholarly items and are used to pro- 2 years, divided by the number of substantive, scholarly items vide detailed information about the content and conclu- published in that journal in those same 2 years.1 The im- sions detailed in the body of the item and to guide readers pact factor is a ratio of journal citations to the number of regarding the relevance of the item to their needs. substantive, scholarly articles published by that journal, but Article length: Items less than a half page in length are un- not the mathematical average of the citations to all of the likely to provide significant scholarly contribution. Lengthy journal’s content. Although the calculation treats each jour- works are more suggestive of scholarly effort. nal as a whole rather than as a simple count of all pub- Data content: Whether primary research data or data col- lished items, the numerator and denominator of the im- lected and summarized from 1 or more prior works, the item pact factor ratio have different definitions. The numerator is more likely to be of interest to scholars in the field if a is the number of all citations to the journal from 1 year to figure or table was produced specifically for the current item. the previous 2 years. Any citation to any item represents an Cited references: An item that cites extensively to prior or acknowledgment of the journal. However, the denomina- concurrent works will have the purpose of evidentiary sup- tor is the number of substantive, scholarly articles most likely port of the author’s hypothesis, or of providing a novel syn- to be cited. Author Affiliations: Bibliographic Policy, Journal Citation Reports, Thomson Reuters, Philadelphia, Pennsylvania. See also p 1092. Corresponding Author: Marie E. McVeigh, MS, Thomson Reuters, 3501 Market St, Philadelphia, PA 19104 ([email protected]).

Downloaded From: https://jamanetwork.com/ on 10/01/2021 COMMENTARIES

thesis of previous works. The presence of cited references ing items that are later cited and that contribute to the im- indicates a level of participation in the academic discourse pact factor. As an example of the effectiveness of the cur- and scholarly development of the subject. rent approach used to identify citable items, we identified Cited reference density: This is defined as the number of 1.1 million items published in 2000, and indexed for the cited references per text page of the item. A high density of Science Citation Index-Expanded and/or the Social Sci- references suggests a significant work of scholarship. ences Citation Index (ie, the 2 citation databases with jour- While no one of these characteristics is definitive, taken nals that appear in the Journal Citation Reports6 listings). in concert, these features have proven accurate in identify- For each item, we recorded the number of citations accrued in each year from 2000 through 2007. The 8-year time Table. Distribution of Citable (Countable) Content and Citations for window allowed the capture of a large proportion of total Items Published in 2000 and 2005 lifetime citation of the items. This count of citations in- Total Items Item Count (% of Total) cluded scholarly citations and citations in academic corre- (Citable ϩ spondence, such as a reply to a letter to the editor that cites Not Citable) Citable Not Citable the original letter. A total of 793 395 (72.8%) of the items Indexed in 2000 All in SCIE 1 090 333 793 395 296 938 (27.2) from 2000 were indexed as articles or reviews, and these and SSCIa (72.8) citable items accounted for 97.6% of all citations accrued Annu Rev Med 33 33 (100.0) 0 in the 8 years since their publication (TABLE). BMJ 3335 612 (18.4) 2723 (81.6) Journals with a simple array of published content, such JAMA 1785 377 (21.1) 1408 (78.9) as the Annual Review of Medicine, show 100% of accrued ci- Lancet 3395 821 (24.2) 2574 (75.8) tations are to citable items. Other journals, such as the BMJ, NEJM 1562 379 (24.3) 1183 (75.7) JAMA, Lancet, and the New England Journal of Medicine Total citations by end of 2007 (NEJM), have more complex content, but show a similar pat- All in SCIE 11 624 107 11 347 214 276 893 (2.4) tern of the majority of citations accumulating to the citable and SSCIa (97.6) (counted) items. The diverse content increases the relative Annu Rev Med 1768 1768 (100.0) 0 proportion of citations to items indexed as not citable (not BMJ 32 069 26 302 (82.0) 5767 (18.0) counted). Although these journals have a large number of JAMA 41 468 36 552 (88.1) 4916 (22.9) discrete sections, and vary in presentation and size, the pro- Lancet 63 246 55 255 (87.4) 7991 (12.6) portionate distribution of citations to citable items is con- NEJM 75 984 69 021 (90.8) 75 984 (9.2) sistent. Total citations in 2002b All in SCIE 1 708 953 1 661 819 47 134 (2.8) To focus on the contribution of published items to the and SSCIa (97.2) citation count for the impact factor, only citations to the cit- Annu Rev Med 254 254 (100.0) 0 able items in 2000 (ie, impact factor denominator) from BMJ 4681 3687 (78.8) 994 (21.2) works published in 2002 were considered. This represents JAMA 6214 5333 (85.8) 881 (14.2) one part of the numerator of the 2002 impact factor. Al- Lancet 9878 8400 (85.0) 1478 (15.0) though fewer items contributed citations to the 2002 im- NEJM 11 171 9921 (88.8) 1250 (11.2) pact factor (65.9% of published items) than were cited within Indexed in 2005 8 years after publication (42.1% of published items), the con- All in SCIE 1 277 290 918 215 359 075 (28.1) and SSCIa (71.9) centration of citations to citable items was even more marked; Annu Rev Med 30 30 (100.0) 0 97.2% of citations in 2002 were to items indexed as citable BMJ 2589 522 (20.2) 2067 (79.8) items in 2000 (Table). JAMA 1386 324 (23.4) 1062 (76.6) Because many changes to these journals have taken place Lancet 1730 422 (24.4) 1308 (75.6) in recent years, we considered another example using all NEJM 1798 308 (17.1) 1490 (82.9) items published in 2005. Four of 5 journals decreased both Total citations in 2007c the total number of items and the number of citable items All in SCIE 2 258 888 2 196 063 62 825 (2.8) a published from 2000 to 2005. One journal (NEJM) in- and SSCI (97.2) creased both total items and citable items, but had a net de- Annu Rev Med 426 426 (100.0) 0 crease in the percentage of citable items relative to all con- BMJ 3780 3041 (80.4) 739 (19.6) tent (Table). For all journals, the percentage of citations in JAMA 7514 6723 (89.5) 791 (10.5) 2007 (the impact factor numerator) to the citable items pub- Lancet 10 870 9739 (89.6) 1131 (10.4) lished in 2005 (the impact factor denominator) was con- NEJM 16 218 14 172 (87.4) 2046 (12.6) Abbreviations: Annu Rev Med, Annual Review of Medicine; NEJM, New England Journal of sistent with the values observed in the 2002 impact factor Medicine; SCIE, Science Citation Index-Expanded; SSCI, Social Sciences Citation Index. data. At the macro level of all indexed items and all jour- aThe SCIE and SSCI databases contain journals that have data prepared for listing in the Journal Citation Reports. nals, the percentage of citations associated with citable items bContribution to Journal Citation Reports 2000 impact factor numerator. did not change from 2002 to 2007. For both analysis years, cContribution to Journal Citation Reports 2007 impact factor numerator. citable items collected 97.2% of the total citation activity

Downloaded From: https://jamanetwork.com/ on 10/01/2021 COMMENTARIES

in the second year following their publication. Four of the The impact factor has had success and utility as a jour- 5 journals showed an overall increase in the percentage of nal metric8 due to its concentration on a simple calculation impact factor citations associated with citable items from based on data that are fully visible in the Web of Science.6 2002 to 2007, whereas 1 journal (NEJM) showed a slight These examples based on citable (counted) items indexed decrease (from 88.8% to 87.4%), but increased the total num- in 2000 and 2005 suggest that the current approach for iden- ber of citations (from 9921 to 14 172). tification of citable items in the impact factor denominator The demonstration that a journal’s citation impact is de- is accurate and consistent. termined primarily by citations to items defined as citable Financial Disclosures: Ms McVeigh is an employee of Thomson Reuters, the pub- is consistent with a report by Golubic et al,7 which also found lisher of the Journal Citation Reports. At the time of the preparation of the manuscript, Stephen J. Mann was employed by Thomson Reuters as a research intern. that most of the materials cited in an impact factor numera- Additional Contributions: We thank James Testa, vice president, Editorial Devel- tor were designated as articles or reviews. Golubic et al7 also opment and Publisher Relations, Thomson Reuters, for helpful review and discussion of the manuscript. Mr Testa was not compensated financially for his contri- noted that the majority of items indexed as articles or re- bution. views (ie, counted in the denominator of the impact factor) contained original data, defined as “previously unpub- REFERENCES lished research results presented in numerical or graphical 1. Garfield E. The history and meaning of the journal impact factor. JAMA. 2006; form.” Although Golubic et al7 argued that the varying con- 295(1):90-93. 2. Seglen PO. Why the impact factor of journals should not be used for evaluat- tribution of not citable materials to the impact factor was ing research. BMJ. 1997;314(7079):498-502. problematic, impact as a journal-level measure in the im- 3. Lawrence PA. The mismeasurement of science. Curr Biol. 2007;17(15): R583-R585. pact factor accommodates the notion that not citable items 4. Pringle J. Trends in the use of ISI Citation databases for evaluation. Learn Publ. also contribute to a journal’s influence in the literature be- 2008;21:85-91. cause citations to any item reflect use of the journal. Con- 5. Falagas ME, Alexiou VG. The top-ten in journal impact factor manipulation. Arch Immunol Ther Exp (Warsz). 2008;56(4):223-226. sidering that the impact factor then adjusts this according 6. Thomson Reuters. Web of Science Web site. http://thomsonreuters.com to the number of items most likely to attract citations be- /products_services/science/science_products/scholarly_research_analysis /research_discovery/web_of_science. Accessibility verified August 14, 2009. cause of their scholarly content, it is not a mathematical av- 7. Golubic R, Rudes M, Kovacic N, Marusic M, Marusic A. Calculating impact fac- erage of citations per item,1-3 but rather reflects the jour- tor: how bibliographic classification of journal items affects the impact factor of large and small journals. Sci Eng Ethics. 2008;14(1):41-49. nal’s participation in and its influence on the scholarly 8. Bensman SJ. Garfield and the impact factor. Annu Rev Inform Sci Tech. 2007; literature. 41:93-155.

Electronic Medical Records at a Crossroads Impetus for Change or Missed Opportunity?

Leonard W. D’Avolio, PhD ing to a 2% reduction in 2016 and a 3% reduction in 2017 and beyond. At the same time, leading US health care organizations, HE PRESIDENT’S AMERICAN RECOVERY AND REINVEST- including the Department of Health and Human Services ment Act includes $19 billion in incentives for the (HHS), its Centers for Medicare & Medicaid Services, and 1 adoption of electronic medical records (EMRs) and the Agency for Healthcare Research and Quality, are his health care reforms include an investment of $50 investing heavily in clinical data–dependent quality mea- billionT to promote health information technology. For phy- surement and improvement initiatives such as Pay-for- sicians considering adoption of an EMR system, the legis- Performance2 and Healthy People 2010.3 Few would dis- lation features both the carrot and the stick. Medicare phy- agree that upgrading the information infrastructure of US sicians adopting and making “meaningful use” of EMRs in hospitals and clinics is essential for improving the quality 2011 and 2012 will be eligible for an initial payment of up of US health care. It is therefore understandable that so to $18 000, with reduced subsequent payment amounts in much attention has been given to the promise of wide- 2013 and 2014. Physicians can capitalize on similar incentives available through the Medicaid program (but not both). Author Affiliations: Massachusetts Veterans Epidemiology Research and Infor- mation Center (MAVERIC), Cooperative Studies Program Coordinating Center, Those physicians who do not adopt EMRs by 2015 will face Veterans Affairs Boston Healthcare System, Jamaica Plain, Massachusetts; and Cen- a reduction of 1% in their Medicare fee schedule, increas- ter for Surgery and Public Health, Brigham and Women’s Hospital, Boston, Mas- sachusetts. Corresponding Author: Leonard W. D’Avolio, PhD, Veterans Affairs Boston Health- care System, Massachusetts Veterans Epidemiology Research and Information Cen- See also p 1111. ter (151MAV), 150 S Huntington Ave, Jamaica Plain, MA 02130 (leonard.davolio @va.gov).

Downloaded From: https://jamanetwork.com/ on 10/01/2021