COMPARISON BETWEEN THE IMPACT FACTOR VALUES AND THE SCOPUS BIBLIOMETRIC INDICATORS CITESCORE, SJR AND SNIP, FOR A SET OF 1800 JOURNALS DISTRIBUTED IN SIX RESEARCH AREAS

Jean Rancourt Librarian Objectives of the Study:

• Determine the correlation level between the journal Impact Factor (IF) and 3 alternatives indicators available through the Scopus database (SNIP, SJR & CiteScore) in 6 different research areas (business & management, mathematics, computer science, psychology, biological sciences & sociology)

• More specifically, generate quantitative data to support the use of the Scopus bibliometric indicators by our faculties as possible alternatives to the well known journal IF (no longer available through JCR and WoS at our university)

1 Presentation Outlook:

• Definitions of the 4 bibliometric indicators considered in our study

• Litterature review on journal performance bibliometric indicators comparisons

• Methodology used

• Results by research fields & globally

• Conclusion

2 Impact Factor (IF) (Garfield, 1963):

Index based on the Journal of Citation Report (JCR) (Web of Science)

Included: editorials, comments, letters to the editor, errata, etc.

number of citations received for all document types during year T for all items published during the previous 2 years (T-1 & T-2) IF (year T)= total citable items published by the journal in the 2 previous years (T-1 & T-2)

articles, conference papers & reviews

3 CiteScore (Elsevier, 2016): Index based on the SCOPUS database

number of citations received for all document types during year T for all items published during the previous 3 years (T-1, T-2 & T-3) CiteScore= (year T) all documents published by the journal in the 3 previous years (T-1, T-2 & T-3)

Since all documents are considered in the denominator of the CiteScore, journals such as Nature and Science, that contain a higher proportion of editorials, comments, letters to the editor, etc, obtain a lower CiteScore in comparison with their IF.

(Sugimoto & Larivière, 2018)

4 SCImago Journal Rank (SJR) (Moya & Guerrero, 2007):

Index based on the SCOPUS database

“It expresses the average number of weighted citations received in the selected year by the documents published in the selected journal in the three previous years, – i.e. weighted citations received in year T by documents published in the journal in years T-1, T-2 and T-3”

Complex algorithm based on Google’s PageRank

(González-Pereira et al., 2010; Renjith & Shihab, 2018) 5 SCImago Journal rank (SJR) (Moya & Guerrero, 2007):

• Citations don’t have the same weight – Citations from highly cited journals weigh more than those from low-cited ones. In other words, a citation from a source with a relatively high SJR is worth more than a citation from a source with a relatively low SJR

– The weight of a citation is increased when the citing journal is topically closely related, and decreases when the citation is coming from a peripheral journal (assumption: journals in the periphery are less able to speak with authority)

• Citations afforded to editorials, letters to the editor, commentaries, perspectives, news, etc, are not taken into account. Only peer-reviewed documents from Scopus covered sources are considered

• Journal self-citations are partially excluded (maximum of 33%)

(Colledge et al., 2010)

6 Source Normalized Impact per Paper (SNIP) (Moed, 2009): Index based on the SCOPUS database

IPP IPP: Impact per Paper SNIP= DCP DCP: Database Citation Potential

IPP= average number of citations to the publications of a given source

total number of citations received during year T for JA, CP and R published during the previous 3 years (T-1, T-2 and T-3) IPP (year T)= total number of JA, CP & R published during the previous 3 years (T-1, T-2 and T-3)

JA: Journal Article; CP: Conference Paper; R: Reviews Peer review materials

7 Source Normalized Impact per Paper (SNIP) (Moed, 2009):

IPP IPP: Impact per Paper SNIP= DCP DCP: Database Citation Potential

n . DCP= ⅓ ⅓ ensures that the average 1 1 1 SNIP-value for all journals + + … + in the database is close to 1 p1r1 p2r2 pnrn

Subject field: collection of documents citing a given journal n: number of publications in the journal’s subject field (in the year of analysis) r: number of references in the publication considered that appeared in the 3 preceding years p: proportion of these publications that have at least one active reference (JA, CP or R) “The DCP is the citation potential in a journal’s subject field, a measure of the citation characteristics of the field the journal sits in, determined by how often and how rapidly authors cite other works, and how well their field is covered by the database (in this case, Scopus).”

(Colledge et al., 2010; Rousseau et al., 2018; Sugimoto & Larivière, 2018) 8 Source Normalized Impact per Paper (SNIP) (Moed, 2009):

“The SNIP attemps to correct for varying sizes and citation rates across each discipline. […] The underlying idea is that it is better to receive 1 citation among 10, than to receive 1 citation among 70.”

Exemple: Citation SNIP SJR Journal IPP Potential (DCP) (2008) (2008) Inventiones Mathematicae 1,5 0,4 3,8 0,075 Molecular Cell 13,0 3,2 4,0 6,073

(Colledge et al., 2010; Rousseau et al., 2018)

9 Impact Factor CiteScore SJR SNIP

Data source WoS Scopus Scopus Scopus

2 years Window 3 years 3 years 3 years (5 years) Scopus Scopus Availibility JCR (WoS) Scopus scimagojr.com journalindicators.com

Type Popularity Popularity Prestige Popularity

Complex Simple Simple Complex Calculation (based on PageRank; (num. ≠ denom.) (num. = denom.) (uses citation potential) weighted citations)

Autocit. Included Included Included, up to ⅓ Included

Corrects for differences in Simple; 3 year Useful to classify journals within citation habits; noncitable Simple; best known items excluded; allows journal Strenght window; the same discipline; data freely and most utilized comparisons across num.=denom. available disciplines; data freely available Short window; Penalizes journals that receive Autocitations; does not autocitations; ease of citations from less widely read consider citation prestige; Weakness Autocitations; manipulation; same ones; difficult to compare complex and difficult to weight for all citations numbers across disciplines check

(Ahmad et al., 2017b; Ferrand et al., 2019; Mingers & Yang, 2017; Setti, 2013). BIBLIOMETRIC INDICATORS SUMMARY 10 Correlations Between the IF and SJR, CiteScore or SNIP: Literature Review Selected Examples 27 articles that reported correlations between the journal metrics considered in our study were identified (see annex for details).

Study Nb of Articles Topic Correlation r1 Elkins (2010) 5503 Multidisciplinary IF ↔ SJR 0,890 IF ↔ SJR 0,839 S.-Olivan (2015) 9657 Multidisciplinary IF ↔ SNIP 0,645 Brown (2018) 14 Occupational therapy IF ↔ CiteScore 0,838 Villasenor (2019) 122 Nuclear medecine IF ↔ CiteScore 0,878 Supply Chain Yuan (2019) 64 IF ↔ CiteScore 0,922 Management & Logistics Generally speaking, very strong to strong correlations2 were observed between the IF and the indicators available in Scopus (SJR, SNIP and CiteScore), although some disciplinary differences were observed. 1r: Pearson’s correlation coefficient or Spearman’s rho. 2 Based on Cozby (2008) criteria for r (see also Elkins et al., 2010) 11 Methodology Used in Our Study:

In order to generate quantitative data that will support (or not) the use of the journal metrics available in Scopus, in place of the Journal Impact Factor, we carried out the following steps:

• Get Scopus Author Ids for our faculties (6 research fields considered)  Business & management, Mathematics, Computer Science, Psychology, Biological Sciences and Sociology • Extract publications from Scopus

• Export results into EndNote (independently for each department)

• Extract journal titles using EndNote (Journals Term List tool)

• Export journal titles to Excel

• Extract the 2016 Journal Impact Factor manually from journals’ home page (or from other online ressources)

12 • Extract the 2016 SJR, CiteScore and SNIP data from Scopus and merge them automatically with existing data using the Tibco Spotfire software (trial version)

• Discard journals that don’t have complete data

• Rank journals according to each 4 metrics and generate corresponding new columns in Excel

• Generate graphs and statistics (Spearman’s rho and probability p) using theTibco Spotfire software (trial version) and SPSS (for Pearson’s correlation)  For our data, the Spearman’s rho and the the Pearson’s correlations were identical. Thus, only the Spearman’s rho will be presented.

13 Note regarding research field assignation:

With this approach, the research field assigned to a publication correspond to the author department affiliation.

Thus, an article published by a professor of mathematics in a journal dedicated to medicine, will be assigned to mathematics, regardless of its content.

14 Our methodology is mirrored on Elkins’ work who compared the ranking of over 5000 journals based on different metrics (Elkins’ data shown on the side for IF ↔ SJR). Spearman’s rho Correlation level 0,80-1,00 Very strong 0,60-0,79 Strong 0,40-0,59 Moderate 0,20-0,39 Weak 0,00-0,19 Very weak

Probabilities p of less than 0.05 were considered significant.

(Elkins et al., 2010) 15 Research Fields and Number of Journals considered:

Business & Management: 389 Mathematics: 269 Computer Science: 252 Psychology: 500 Biological Sciences: 563 Sociology: 45

All fields combined: 1819*

* There are some overlaps between the 6 fields. A given journal may be part of 2 or more categories (Ex.: Nature). 16 Business & Management (r= 0,92) Mathematics (r= 0,95) Computer Science (r=0,93)

Psychology (r= 0,90) Biological Sciences (r= 0,96) Sociology (r= 0,84)

Abscissa (x): Journal Impact Factor Rank; ordinate (y): CiteScore Rank. Probabilities p < 0,001 in all cases.

IF ↔ CiteScore: Very high correlation; r (Spearman’s rho): 0,84 – 0,96

CITESCORE DATA 17 Business & Management (r= 0,69) Mathematics (r= 0,66) Computer Science (r= 0,78)

Psychology (r= 0,89) Biological Sciences (r= 0,93) Sociology (r= 0,77)

Abscissa (x): Journal Impact Factor Rank; ordinate (y): SJR Rank. Probabilities p < 0,001 in all cases.

IF ↔ SJR: Strong to very strong correlation; r (Spearman’s rho): 0,66 – 0,93

SJR DATA 18 Business & Management (r= 0,78) Mathematics (r= 0,65) Computer Science (r= 0,64)

Psychology (r= 0,72) Biological Sciences (r= 0,84) Sociology (r= 0,65)

Abscissa (x): Journal Impact Factor Rank; ordinate (y): SNIP Rank. Probabilities p < 0,001 in all cases.

IF ↔ SNIP: Strong to very strong correlation; r (Spearman’s rho): 0,64 – 0,84

SNIP DATA 19 Global data (6 research fields combined):

CiteScore (r= 0,94) SJR (r= 0,77) SNIP (r= 0,63)

Abscissa (x): Journal Impact Factor Rank; ordinate (y): Other Rank. Probabilities p < 0,001 in all cases.

The strongest correlation is observed for IF ↔ CiteScore (r= 0,94)

GLOBAL DATA 20 Summary: Spearman’s rho correlations between the Journal Impact Factor (IF) and 3 Scopus metrics (CiteScore, SJR and SNIP) for 6 field areas; correlations between SJR and SNIP also highlighted. Field Nb of Journals CiteScore SJR SNIP SJR vs SNIP All (6 fields) 1819 0,94 0,77 0,63 0,74 Business & 389 0,92 0,69 0,78 0,85 Management Mathematics 269 0,95 0,66 0,65 0,79 Computer Science 252 0,93 0,78 0,64 0,63 Psychology 500 0,90 0,89 0,72 0,81 Biological Sc. 563 0,96 0,93 0,84 0,81 Sociology 45 0,84 0,77 0,65 0,85

Probabilities p < 0,001 in all cases.

Bold: Very strong correlations (r > 0,80) SUMMARY 21 Conclusion:

• The strongest correlation observed is for IF ↔ CiteScore (r > 0,90 for 5/6 fields) - Exception: Sociology (r= 0,84)

• Correlations for IF ↔ SJR are, in general, much smaller. However, in the case of psychology & biological sciences, very strong correlations were observed (r= 0,89 & 0,93, respectivement)

• Generally speaking, SNIP corrrelates better with SJR than with IF. However, in the case of computer science & biological sciences, the level of correlation is similar

• In general, our results are in line with previous studies

The CiteScore indicator showed the highest correlation with the traditional journal Impact and offers a good alternative to it.

CONCLUSION 22 Study Nb of Articles Topic Correlation r* Leydesdorff (2009) 6158 Multidisciplinary IF ↔ SJR 0,796 Rousseau (2009) 77 Multidisciplinary IF ↔ SJR 0,915 Schöpfel (2009) 143 Multidisciplinary (French J.) IF ↔ SJR 0,76 Elkins (2010) 5503 Multidisciplinary IF ↔ SJR 0,89 801 Engineering IF ↔ SJR 0,920 IF ↔ SNIP 0,523 1940 Medicine IF ↔ SJR 0,834 IF ↔ SNIP 0,866 563 Physics & Astronomy IF ↔ SJR 0,921 IF ↔ SNIP 0,849 T.-Salinas (2010) 1017 Social Sciences IF ↔ SJR 0,680 IF ↔ SNIP 0,612 161 Arts & Humanities IF ↔ SJR 0,959 IF ↔ SNIP 0,772 224 Economics IF ↔ SJR 0,017 IF ↔ SNIP 0,745

*r: Pearson’s correlation coefficient or Spearman’s rho.

ANNEX: PREVIOUS STUDIES (1) 23 Study Nb of Articles Topic Correlation r* Jacso (2010) 50 Information & Library Sc. IF ↔ SJR 0,88 Rocha (2010) 99 Multidisciplinary (Brazilian J.) IF ↔ SJR 0,884 Siebelt (2010) 18 (10+8) Orthopaedics IF ↔ SJR 0,98; 0,93 Sicilia (2011) 259 Computer Science IF ↔ SJR 0,93 Ramin (2012) 13 Nuclear Medecine IF ↔ SJR 0,919 Kianifar (2014) 11 Pediatric Neurology IF ↔ SJR 0,736 Cantin (2015) 20 Anatomy IF ↔ SJR -0,037 IF ↔ SJR 0,839 S.-Olivan (2015) 9657 Multidisciplinary IF ↔ SNIP 0,645 Mahmood (2016) 88 Dentistry IF ↔ SJR 0,865 Ahmad (2017a) 50 Environ. Engineering IF ↔ SJR 0,862 Ahmad (2017b) 131 Mechanical Engineering IF ↔ SJR 0,874 Ahmad (2017c) 85 Water Ressources IF ↔ SJR 0,806 Ahmad (2017d) 75 Analytical Chemistry IF ↔ SJR 0,946 *r: Pearson’s correlation coefficient or Spearman’s rho. ANNEX: PREVIOUS STUDIES (2) 24 Study Nb of Articles Topic Correlation r* IF ↔ SJR 0,806 Mingers (2017) 426 Business & Management IF ↔ SNIP 0,853 Ahmad (2018a) 50 Biochemistry/Mol. Biology IF ↔ SJR 0,916 Ahmad (2018b) 61 Construction & Build. Tech. IF ↔ SJR 0,848 IF ↔ SJR 0.920 Brown (2018) 14 Occupational therapy IF ↔ CiteScore 0,838 IF ↔ SNIP 0,898 Perera (2018) 519 Multidisciplinary IF ↔ SJR 0,796 Waris (2018) 81 Sports Science IF ↔ SJR 0,905 Yuen (2018) 54 Neurosurgery FI ↔ SJR 0,951 IF ↔ SJR 0,884 Villasenor (2019) 122 Nuclear medecine IF ↔ CiteScore 0,878 IF ↔ SNIP 0,788 IF ↔ SJR 0,708 Supply Chain Management Yuan (2019) 64 IF ↔ CiteScore 0,922 & Logistics IF ↔ SNIP 0,888 *r: Pearson’s correlation coefficient or Spearman’s rho. ANNEX: PREVIOUS STUDIES (3) 25 Ahmad, S., Abdel-Magid, H. I., Waris, A. & Abdel-Magid, I. M. (2018a). Correlation between journal citation indices for Biochemistry and Molecular Biology Journals. Library Philosophy & Practice, (December), 1-13.

Ahmad, S., Sohail, M., Waris, A., Elginaid, A. & Mohammed, I. (2018b). SCImago, Eigenfactor Score, and H5 Index Journal Rank Indicator: A Study of Journals in the area of Construction and Building Technologies. DESIDOC Journal of Library & Information Technology, 38(4), 278-285.

Ahmad, S., Abdel-Magid, I. M. & Hussain, A. (2017a). Comparison among journal impact factor, SCimago journal rank indicator, eigenfactor score and h5-index of environmental engineering journals. COLLNET Journal of Scientometrics and Information Management, 11(1), 133-151. doi: 10.1080/09737766.2016.1266807

Ahmad, S., Magid, E. T. I. M. A., Magid, C. I. M. A. & Waris, A. (2017b). Comparison among Selected Journal Quality Indicators of Mechanical Engineering Journals. Journal of Scientometric Research, 6(3), 151-158.

Ahmad, S., Sohail, M. & Abdel-Magid, I. M. (2017c). SCImago, Eigenfactor Score and H5 Index Journal Rank Indicator: Alternatives to the Journal Impact Factor for Water Resources Journals. LIBRES: Library & Information Science Research Electronic Journal, 27(2), 97-111.

Ahmad, A., Ahmad, S., Waris, A. & Abdel-Magid, I. M. (2017d). Comparison of Selected Journal Quality Indicators of Analytical Chemistry Journals. SRELS Journal of Information Management, 54(4), 175-182. doi: 10.17821/srels/2017/v54i4/113833

Brown, T. & Gutman, S. A. (2018). Impact factor, eigenfactor, article influence, Scopus SNIP, and SCImage journal rank of occupational therapy journals. Scandinavian Journal of Occupational Therapy, 1-9. doi: 10.1080/11038128.2018.1473489

Cantín, M., Muñoz, M. & Roa, I. (2015). Comparison between Impact Factor, Eigenfactor Score, and SCImago Journal Rank Indicator in Anatomy and Morphology Journals. Comparación entre Factor de Impacto, Eigenfactor Score e Indicador SCImago Journal Rank en Revistas de Anatomía y Morfología., 33(3), 1183-1188. doi: 10.4067/S0717-95022015000300060

Colledge, L., De Moya-Anegón, F., Guerrero-Bote, V., López-Illescas, C., El Aisati, M. & Moed, H. F. (2010). SJR and SNIP: two new journal metrics in Elsevier's Scopus. Serials, 23(3), 215-221. doi: 10.1629/23215

Cozby, P. C. (2008). Methods in behavioral research. New York : McGraw-Hill.

Elkins, M. R., Maher, C. G., Herbert, R. D., Moseley, A. M. & Sherrington, C. (2010). Correlation between the journal impact factor and three other journal citation indices. Scientometrics, 85(1), 81-93.

REFERENCES (1) 26 Ferrand, É., Larivière, V., Lebel, D. & Bussières, J. F. (2019). Indicateurs de notoriété des revues scientifiques, des chercheurs et des articles publiés en santé : perspective pharmaceutique. Annales Pharmaceutiques Françaises, 77(1), 1-14. doi: 10.1016/j.pharma.2018.09.001

González-Pereira, B., Guerrero-Bote, V. P. & Moya-Anegón, F. (2010). A new approach to the metric of journals’ scientific prestige: The SJR indicator. Journal of informetrics, 4(3), 379-391. doi: 10.1016/j.joi.2010.03.002

Jacsó, P. (2010). Comparison of journal impact rankings in the SCImago Journal & Country Rank and the Journal Citation Reports databases. Online Information Review, 34(4), 642-657. doi: 10.1108/14684521011073034

Kianifar, H., Sadeghi, R. & Zarifmahmoudi, L. (2014). Comparison between impact factor, eigenfactor metrics, and SCimago journal rank indicator of pediatric neurology journals. Acta Informatica Medica, 22(2), 103-106. doi: 10.5455/aim.2014.22.103-106

Leydesdorff, L. (2009). How are new citation-based journal indicators adding to the bibliometric toolbox? Journal of the American Society for Information Science and Technology, 60(7), 1327-1336. doi: 10.1002/asi.21024

Mahmood, K. & Almas, K. (2016). SCImago journal rank indicator: A viable alternative to journal impact factor for dental journals. Libres, 26(2), 144- 151.

Mingers, J. & Yang, L. (2017). Evaluating journal quality: A review of journal citation indicators and ranking in business and management. European Journal of Operational Research, 257(1), 323-337. doi: 10.1016/j.ejor.2016.07.058

Ramin, S. & Shirazi, A. S. (2012). Comparison between impact factor, SCImago journal rank indicator and eigenfactor score of nuclear medicine journals. Nuclear Medicine Review, 15(2), 132-136.

Renjith, V. R. & Shihab, I. (2018). h-index of geology journals: A statistical analysis based on scimago journal and country ranking. COLLNET Journal of Scientometrics and Information Management, 12(1), 97-107. doi: 10.1080/09737766.2017.1400753

Rocha-e-Silva, M. (2010). Impact factor, Scimago Indexes and the Brazilian journal rating system: where do we go from here? Clinics (Sao Paulo, Brazil), 65(4), 351-355. doi: 10.1590/S1807-59322010000400001

Rousseau, R., Egghe, L. & Guns, R. (2018). Becoming Metric-Wise: A Bibliometric Guide for Researchers. Cambridge, MA : Chandos Publishing.

Rousseau, R. & STIMULATE 8 Group. (2009). On the relation between the WoS impact factor, the Eigenfactor, the SCImago Journal Rank, the Article Influence Score and the journal h-index. Preprint. Retrieved from http://eprints.rclis.org/13304/

REFERENCES (2) 27 Salvador-Oliván, J.-A. & Agustín-Lacruz, M.-D.-C. (2015). Correlación entre indicadores bibliométricos en revistas de Web of Science y Scopus. Revista General de Información y Documentación, 25(2), 341-359. doi: 10.5209/rev_RGID.2015.v25.n2.51241

Schöpfel, J. & Prost, H. (2009). Comparison of SCImago Journal Rank Indicator (SJR) with JCR journal impact factor (IF) for French journals. Psychologie Francaise, 54(4), 287-305. doi: 10.1016/j.psfr.2009.07.002

Setti, G. (2013). Bibliometric Indicators: Why Do We Need More Than One? IEEE Access, 1, 232-246. doi: 10.1109/ACCESS.2013.2261115

Sicilia, M. A., Sánchez-Alonso, S. & García-Barriocanal, E. (2011). Comparing impact factors from two different citation databases: The case of Computer Science. Journal of Informetrics, 5(4), 698-704. doi: 10.1016/j.joi.2011.01.007

Siebelt, M., Siebelt, T., Pilot, P., Bloem, R. M., Bhandari, M. & Poolman, R. W. (2010). Citation analysis of orthopaedic literature; 18 major orthopaedic journals compared for Impact Factor and SCImago. BMC Musculoskeletal Disorders, 11(1), 4. doi: 10.1186/1471-2474-11-4

Sugimoto, C. R. & Larivière, V. (2018). Measuring Research : What Everyone Needs To Know. New York, NY : Oxford University Press.

Torres-Salinas, D. & Jiménez-Contreras, E. (2010). Introduction and comparative study of the new scientific journals citation indicators in Journal Citation Reports and Scopus. El profesional de la información, 19(2), 201-207. doi: 10.3145/epi.2010.mar.12

Villaseñor-Almaraz, M., Islas-Serrano, J., Murata, C. & Roldan-Valadez, E. (2019). Impact factor correlations with Scimago Journal Rank, Source Normalized Impact per Paper, Eigenfactor Score, and the CiteScore in Radiology, Nuclear Medicine & Medical Imaging journals. Radiologia Medica. doi: 10.1007/s11547-019-00996-z

Waris, A., Ahmad, S., Isam, C., Abdel-Magid, M. & Hussain, A. (2017). Comparison among Journal Quality Indicators of Sports Science Journals. Library Herald, 55(3), 339-351.

Yuan, L., Li, J., Li, R., Lu, X. & Wu, D. (2019). Mapping the evaluation results between quantitative metrics and meta-synthesis from experts’ judgements: evidence from the Supply Chain Management and Logistics journals ranking. Soft Computing. doi: 10.1007/s00500-019-03837-3

Yuen, J. (2018). Comparison of Impact Factor, Eigenfactor Metrics, and SCImago Journal Rank Indicator and h-index for Neurosurgical and Spinal Surgical Journals. World Neurosurgery, 119, e328-e337. doi: https://doi.org/10.1016/j.wneu.2018.07.144

REFERENCES (3) 28