Assessment of Research Fields in Scopus and Web of Science from the Viewpoint of National Research Evaluation in Slovenia
Total Page：16
File Type：pdf, Size：1020Kb
Scientometrics (2014) Volume 98, Issue 2: 1491–1504 Assessment of research fields in Scopus and Web of Science from the viewpoint of national research evaluation in Slovenia Tomaz Bartol 1 a, Gordana Budimir 2, Doris Dekleva-Smrekar 3, Miro Pusnik 3, Primoz Juznic 4 1 Agronomy Department, Biotechnical Faculty, University of Ljubljana, Ljubljana, Slovenia 2 Institute of Information Science, Maribor, Slovenia 3 Central Technological Library at the University of Ljubljana, Ljubljana, Slovenia 4 Department of Library and Information Science and Book Studies, Faculty of Arts, University of Ljubljana, Ljubljana, Slovenia Abstract Web of Science ( WOS ) and SCOPUS have often been compared with regard to user interface, countries, institutions, author sets, etc., but rarely employing a more systematic assessment of major research fields and national production. The aim of this study was to appraise the differences among major research fields in SCOPUS and WOS based on a standardized classification of fields and assessed for the case of an entire country (Slovenia). We analyzed all documents and citations received by authors who were actively engaged in research in Slovenia between 1996 and 2011 (50,000 unique documents by 10,000 researchers). Documents were tracked and linked to SCOPUS and WOS using complex algorithms in the Slovenian COBISS bibliographic system and SICRIS research system where the subject areas or research fields of all documents are harmonized by the Frascati/ OECD classification, thus offsetting some major differences between WOS and SCOPUS in database-specific subject schemes as well as limitations of deriving data directly from databases. SCOPUS leads over WOS in indexed documents as well as citations in all research fields. This is especially evident in social sciences, humanities, and engineering & technology. The least citations per document were received in humanities and most citations in medical and natural sciences, which exhibit similar counts. Engineering & technology reveals only half the citations per document compared to the previous two fields. Agriculture is found in the middle. The established differences between databases and research fields provide the Slovenian research funding agency with additional criteria for a more balanced evaluation of research. Keywords: bibliometrics; citation analysis; research performance; research evaluation; research fields; research information systems; Slovenia Introduction a Author's version. The final paper is available at link. springer.com: http://link.springer.com/article/10.1007/s11192-013-1148-8 Bartol, T., Budimir, G., Dekleva-Smrekar, D., Pusnik, M., & Juznic, P. (2014). Assessment of research fields in Scopus and Web of Science in the view of national research evaluation in Slovenia. Scientometrics , 98 (2), 1491– 1504. doi:10.1007/s11192-013-1148-8 Scientometrics (2014) Volume 98, Issue 2: 1491–1504 Similarities and differences between WOS and SCOPUS have frequently been compared with regard to various criteria. Most studies have focused on comparing the user interface (retrieval using different fields) and differences in coverage according to various issues. Less research has been published that attempts a more comprehensive and balanced evaluation of the role of the two databases in the assessment of a wider national production of publications, especially with regard to possible differences among different scientific fields. It is known that the transparency of research is an important element of scientific activity. This involves the use of different quantitative methods such as the employment of bibliometric indicators for the purposes of research evaluation. If the two citation databases are used concurrently for such evaluation, it is important to know the principal characteristics of both databases in tracking publications and citations in different scientific fields. The general purpose of our research is to compare SCOPUS and WOS by assessing, more specifically, differences among different fields of research. The case of Slovenian documents and citations appearing in both databases will be studied. We assume that differences exist among databases in coverage and citability that have frequently been addressed in the literature. More particularly, however, we also wish to establish if there exist some particular differences among different major research fields, not only between databases but also within each database. It is important to assess such differences more systematically. Not only do the databases differ in coverage, but different research fields may also exhibit different publishing patterns and different levels of citability. Such information may serve as a basis for a more balanced appraisal of research fields for the purposes of evaluating researchers and research activities in Slovenia. Specifically, if the differences are not taken into account, some researchers, for example in the fields of social sciences, humanities, and possibly some other fields, may not compete for very limited national research funds on a level playing field with other researchers since certain fields may exhibit much higher citability. The study covers some 50,000 unique documents published by almost 10,000 authors and indexed by WOS and SCOPUS between 1996 and 2011. The study employs a systematic authority-control managed by the COBISS system (the techniques are presented in detail in the next section), which uses an internationally harmonized classification of scientific fields for both databases to offset many well-known differences between the two databases, for example, in classification, missing data, errors, or problems in capturing consistent sets in the data derived directly from databases. The study thus not only provides systematically collected and analyzed information on scientific production in Slovenia for the purposes of evaluation, but also offers a more balanced general overview of the two databases and field-specific characteristics based on a large set of harmonized data over a longer period of time. In preparation, we reviewed some selected previous papers that emphasized the issues tackled in our research. Comparison of the two databases is usually based on selected country data, institutions, selected journals, publication types, subject categories (subject areas), etc. As opposed to our research, however, most authors derived data directly from the databases, which presents substantial limitations in an analysis. The assessment of database coverage by countries, institutions, or journals seems to be the most frequent object of research. Benoit and Marsh (2009) assessed universities with a political studies program in Ireland (North and South) and compared Irish departments without any kind of national level research ranking and review with the United Kingdom, where the Research Assessment Exercise (RAE) permits some identification of centers of research excellence. Scientific publishing of the health-sciences-related departments of Navarra Scientometrics (2014) Volume 98, Issue 2: 1491–1504 University was assessed by Torres-Salinas et al. (2009) on a sample of the 50 most cited researchers, with SCOPUS providing 14.7% more citations. A study by Haddow and Genoni (2009), which was based on Australian education journals, also found that the coverage in SCOPUS outperformed WOS . SCOPUS also performed better in some areas of the social sciences (Lasda-Bergman 2012). In general, however, the advantage of SCOPUS over WOS lies in the better coverage of journals in languages other than English (Leydesdorff et al. 2010). For example, 240 Russian journal titles are indexed by SCOPUS , and 160 by WOS (Zibareva and Soloshenko 2011). On the other hand, such national journals play a more peripheral role in the international journal communication system (Lopez-Illescas et al. 2009). Despite the differences, both databases are comparable in terms of rankings (Bar-Ilan et al. 2007) with the important exception of certain authors who publish their research in some major series that are only indexed by SCOPUS (Bar-Ilan 2008). In particular, the top countries have similar ranks in both databases (Archambault et al. 2009). Here, however, it needs to be pointed out that analyses based on smaller entities (publications, institutions) produce considerably different results for the two databases than an analysis based on larger entities such as research domains and previously mentioned countries (Meho and Sugimoto 2009). Researchers frequently report possible errors and limitations involving analyses that derive data directly from databases. Vieira and Gomes (2009) used such data in an assessment of two universities and reported possible inaccuracies in the retrieval of bibliographic units due to input errors, for example, the country in the authors’ addresses. In a study covering WOS and several other databases, it was observed that sometimes a city may be present in the affiliation field while the country is missing (Bartol and Hocevar 2005). Even though the identification of an author may be good, the missing data can also be attributed to journal publishers who have not always had consistent policies for including the country affiliation of the authors (Jacso 2009). Many Russian-language references are frequently not taken into account in “citation reports” because of problems involving transliteration (Zibareva and Soloshenko 2011). Furthermore, the policy of assigning particular document types may change over time, which creates unclear differentiating