Evaluating Academic Journals Without Impact Factors for Collection Management Decisions
Total Page:16
File Type:pdf, Size:1020Kb
562 College & Research Libraries November 2002 Evaluating Academic Journals without Impact Factors for Collection Management Decisions Juris Dilevko and Esther Atkinson Evaluation of academic journals for collection management decisions is made all the more difficult when some journals do not have impact fac- tors as assigned by the Institute for Scientific Information and its Journal Citation Reports. Focusing on science, technology, and medicine jour- nals, this study presents a method of evaluating such nonranked jour- nals. The method is based on finding a comparator journal to the nonranked journal, distinguishing between original research articles and other article types, tracing citations to these two target journals in citing journals, comparing the quality of the citing journals that cite both target journals, and describing the contextual typology of the citations to the target journals. A case study of two medical science journals, the nonranked Annals of the Royal College of Physicians and Surgeons of Canada and the comparator ranked Canadian Family Physician, illus- trates the method. This method can help in determining the value of a nonranked journal in relation to a ranked journal. here has been a long-standing the United Kingdom, and Tony Stankus debate about the kinds of evi- and Barbara Rice, who concluded that ci- dence that should be used to tation data and use data “correlate well” make informed collection for biochemistry, cell biology, ecology, management decisions about journals. geosciences, and mathematics journals Should actual use statistics be relied on, that are used “at least 25 times per year” or should citation data and analysis be at a major university library in the state used? In a 1998 textbook on serials man- of New York.1 One reason for these dif- agement, Thomas E. Nisonger summa- ferent results is “different library con- rized the opposing viewpoints, noting the texts,” and so many scholars recommend findings of such scholars as Pauline that “citation data should be used in com- Scales, who observed relatively low cor- bination with other evaluative mea- relations between journal citation sures.”2 rankings and actual use of science jour- As is well known, citation analysis is nals at the National Lending Library in based primarily on data produced by the Juris Dilevko is on the Faculty of Information Studies at the University of Toronto; e-mail: [email protected]. Esther Atkinson is Assistant Executive Director of the Health Science Informa- tion Consortium of Toronto, Gerstein Science Information Centre, at the University of Toronto; e-mail: [email protected]. 562 Evaluating Academic Journals without Impact Factors 563 Institute for Scientific Information (ISI), better predicted local journal use (and which offers a number of products and subsequent value of that journal to a par- databases of great help to librarians and ticular institution) than did citation analy- other information professionals. Among sis of the kind performed by national da- these databases are Science Citation In- tabases.11 Keith Swigger and Adeline dex (SCI), Social Sciences Citation Index Wilkes thus recommended that libraries (SSCI), and Arts and Humanities Citation compile local citation indexes to help in Index (AHCI), each of which “provides serials management decisions, something access to current and retrospective biblio- that has become relatively easy to do graphic information, author abstracts, through the type of institution-specific and cited references” in its respective rank commands available in electronic area.3 In addition, ISI produces Journal databases.12,13 Citation Reports (JCR), which is an evalu- Despite these debates, it is safe to say ation and analysis tool consisting of that “an increasing number of academic “quantifiable statistical data that provides and research libraries have begun to re- alize the potential use of citation analysis Theoretically, the fact that a journal as a tool for serials collection manage- is cited frequently indicates that ment.”14 Yet, when libraries do use JCR scholars in the field deem it to be to evaluate journals in their collections, it influential. is often the case that “the journals one wishes to evaluate may not be contained a systematic, objective way to determine in the JCR.”15 For example, of the approxi- the relative importance of journals within mately 126,000 scientific journals pub- their subject categories.”4 JCR contains lished worldwide, the JCR Science Edi- four ISI measures for evaluating journals: tion covers only about 5,700 of them.16,17 impact factor; immediacy index; cited Faced with the fact that many of the jour- half-life; and citing half-life. Arguably, nals in their collections, or journals they impact factor is the most important and may wish to purchase for their collections, most widely used of these measures. It is are not part of JCR and therefore cannot calculated annually by “dividing the easily be compared with ranked journals number of current citations a journal re- in the same field, collection management ceives to articles published in the two librarians may not be properly evaluat- previous years by the number of articles ing such journals for selection and de-se- published in those same years.”5 Theoreti- lection purposes. Starting from the cally, the fact that a journal is cited fre- premise that citation analysis is an impor- quently indicates that scholars in the field tant tool for collection management librar- deem it to be influential. ians, the present study provides a meth- Because JCR citation data “may be odological framework for evaluating a viewed as reflecting journal usage by re- nonranked journal in comparison with a searchers at the national and international ranked journal (collectively referred to as level,” a second debate centers on the target journals) using available citation question of whether such international data and then applies this framework to and national citation data “correspond to a case study in the field of medical sci- the local needs” of individual libraries.6 ence. Among the initiators of this debate were Robert N. Broadus, Elizabeth Pan, and Method Maurice B. Line, with Broadus and Pan The method for comparing a nonranked suggesting that JCR is a valid tool to use journal with a ranked journal involves a when evaluating local collections and number of steps. These steps, together Line arguing the opposite.7–10 Stephen E. with justifications for each step, are out- Wiberley Jr. found that journal rankings lined below according to the order of pro- based on local faculty citation analysis cedure. 564 College & Research Libraries November 2002 Finding a Comparator Journal if letters are included as source items only Finding a comparator journal involves when they meet a certain “point” level examining a significant number of rel- that qualifies them as “substantive re- evant journals in the same field that ful- search,” that journal’s impact factor may fill a similar function for the same gen- decline because this new and tighter cri- eral readership. Ideally, both the ranked terion not only eliminates many letters as journal and the nonranked journal should source items (denominators), but also the have the same number of issues per year, numerous citations to those letters (nu- or as close to the same number of issues merators in the impact factor calculation). per year as possible, with approximately Brian D. Scanlan argued that original the same number of source items (defined research articles are most typically “rep- below) and articles. Why is this impor- resentative of advances in science.”23 In tant? Roger Taylor was among the first to addition, they are the most frequent type suggest that the number of and length of of article published, thus providing a criti- articles published has a great deal of bear- cal mass of data on which to base conclu- ing on a journal’s impact factor. A journal sions. Thus, the method described here publishing a small number of longer re- looks only at original research articles. view papers will have an advantage, in Nevertheless, if an academic library terms of the calculation of impact factor, wishes to evaluate a nonranked journal over one that publishes a greater number that typically publishes many review ar- of shorter articles, notes, and brief com- ticles, it would be appropriate to compare munications.18 The reason is that when review articles in the nonranked journal doing literature reviews for their own with review articles in the comparator articles, scholars gravitate to one perti- ranked journal. The main point here is nent inclusive review article in their field that when comparing two or more jour- because it provides a convenient sum- nals, similar articles should be compared mary and relevant overview of recent re- with similar articles; more precisely, the search rather than using far-flung and citations received by one type of article possibly difficult-to-find shorter original should only be compared with those re- research articles.19 Some journals, in fact, ceived by the same type of article. Fur- have moved toward increasing their im- thermore, as noted by M. Amin and M. pact factor through the publication of spe- Mabe, when “citations to only selected cific types of articles.20 Thus, to get a good article types are divided by the number sense of the quality of the source items of those selected article types, consider- (including articles) in a nonranked jour- able differences can emerge from the pub- nal, it should be compared with a journal lished impact factors.”24 that is similar in terms of the number and In this regard, the best approach is, length of source items and articles, as well first, to scan the table of contents (TOC) as in scope and content.21 of each issue of the journal under exami- nation.