Scientometrics (2014) Volume 98, Issue 2: 1491–1504

Assessment of research fields in and from the viewpoint of national research evaluation in Slovenia

Tomaz Bartol 1 a, Gordana Budimir 2, Doris Dekleva-Smrekar 3, Miro Pusnik 3, Primoz Juznic 4

1 Agronomy Department, Biotechnical Faculty, University of Ljubljana, Ljubljana, Slovenia 2 Institute of Information Science, Maribor, Slovenia 3 Central Technological Library at the University of Ljubljana, Ljubljana, Slovenia 4 Department of Library and Information Science and Book Studies, Faculty of Arts, University of Ljubljana, Ljubljana, Slovenia

Abstract

Web of Science ( WOS ) and SCOPUS have often been compared with regard to user interface, countries, institutions, author sets, etc., but rarely employing a more systematic assessment of major research fields and national production. The aim of this study was to appraise the differences among major research fields in SCOPUS and WOS based on a standardized classification of fields and assessed for the case of an entire country (Slovenia). We analyzed all documents and citations received by authors who were actively engaged in research in Slovenia between 1996 and 2011 (50,000 unique documents by 10,000 researchers). Documents were tracked and linked to SCOPUS and WOS using complex algorithms in the Slovenian COBISS bibliographic system and SICRIS research system where the subject areas or research fields of all documents are harmonized by the Frascati/ OECD classification, thus offsetting some major differences between WOS and SCOPUS in database-specific subject schemes as well as limitations of deriving data directly from databases. SCOPUS leads over WOS in indexed documents as well as citations in all research fields. This is especially evident in social sciences, humanities, and engineering & technology. The least citations per document were received in humanities and most citations in medical and natural sciences, which exhibit similar counts. Engineering & technology reveals only half the citations per document compared to the previous two fields. Agriculture is found in the middle. The established differences between databases and research fields provide the Slovenian research funding agency with additional criteria for a more balanced evaluation of research.

Keywords: bibliometrics; citation analysis; research performance; research evaluation; research fields; research information systems; Slovenia

Introduction a Author's version. The final paper is available at link. springer.com: http://link.springer.com/article/10.1007/s11192-013-1148-8

Bartol, T., Budimir, G., Dekleva-Smrekar, D., Pusnik, M., & Juznic, P. (2014). Assessment of research fields in Scopus and Web of Science in the view of national research evaluation in Slovenia. Scientometrics , 98 (2), 1491– 1504. doi:10.1007/s11192-013-1148-8 Scientometrics (2014) Volume 98, Issue 2: 1491–1504

Similarities and differences between WOS and SCOPUS have frequently been compared with regard to various criteria. Most studies have focused on comparing the user interface (retrieval using different fields) and differences in coverage according to various issues. Less research has been published that attempts a more comprehensive and balanced evaluation of the role of the two databases in the assessment of a wider national production of publications, especially with regard to possible differences among different scientific fields. It is known that the transparency of research is an important element of scientific activity. This involves the use of different quantitative methods such as the employment of bibliometric indicators for the purposes of research evaluation. If the two citation databases are used concurrently for such evaluation, it is important to know the principal characteristics of both databases in tracking publications and citations in different scientific fields. The general purpose of our research is to compare SCOPUS and WOS by assessing, more specifically, differences among different fields of research. The case of Slovenian documents and citations appearing in both databases will be studied. We assume that differences exist among databases in coverage and citability that have frequently been addressed in the literature. More particularly, however, we also wish to establish if there exist some particular differences among different major research fields, not only between databases but also within each database. It is important to assess such differences more systematically. Not only do the databases differ in coverage, but different research fields may also exhibit different publishing patterns and different levels of citability. Such information may serve as a basis for a more balanced appraisal of research fields for the purposes of evaluating researchers and research activities in Slovenia. Specifically, if the differences are not taken into account, some researchers, for example in the fields of social sciences, humanities, and possibly some other fields, may not compete for very limited national research funds on a level playing field with other researchers since certain fields may exhibit much higher citability. The study covers some 50,000 unique documents published by almost 10,000 authors and indexed by WOS and SCOPUS between 1996 and 2011. The study employs a systematic authority-control managed by the COBISS system (the techniques are presented in detail in the next section), which uses an internationally harmonized classification of scientific fields for both databases to offset many well-known differences between the two databases, for example, in classification, missing data, errors, or problems in capturing consistent sets in the data derived directly from databases. The study thus not only provides systematically collected and analyzed information on scientific production in Slovenia for the purposes of evaluation, but also offers a more balanced general overview of the two databases and field-specific characteristics based on a large set of harmonized data over a longer period of time. In preparation, we reviewed some selected previous papers that emphasized the issues tackled in our research. Comparison of the two databases is usually based on selected country data, institutions, selected journals, publication types, subject categories (subject areas), etc. As opposed to our research, however, most authors derived data directly from the databases, which presents substantial limitations in an analysis. The assessment of database coverage by countries, institutions, or journals seems to be the most frequent object of research. Benoit and Marsh (2009) assessed universities with a political studies program in Ireland (North and South) and compared Irish departments without any kind of national level research ranking and review with the United Kingdom, where the Research Assessment Exercise (RAE) permits some identification of centers of research excellence. Scientific publishing of the health-sciences-related departments of Navarra Scientometrics (2014) Volume 98, Issue 2: 1491–1504

University was assessed by Torres-Salinas et al. (2009) on a sample of the 50 most cited researchers, with SCOPUS providing 14.7% more citations. A study by Haddow and Genoni (2009), which was based on Australian education journals, also found that the coverage in SCOPUS outperformed WOS . SCOPUS also performed better in some areas of the social sciences (Lasda-Bergman 2012). In general, however, the advantage of SCOPUS over WOS lies in the better coverage of journals in languages other than English (Leydesdorff et al. 2010). For example, 240 Russian journal titles are indexed by SCOPUS , and 160 by WOS (Zibareva and Soloshenko 2011). On the other hand, such national journals play a more peripheral role in the international journal communication system (Lopez-Illescas et al. 2009). Despite the differences, both databases are comparable in terms of rankings (Bar-Ilan et al. 2007) with the important exception of certain authors who publish their research in some major series that are only indexed by SCOPUS (Bar-Ilan 2008). In particular, the top countries have similar ranks in both databases (Archambault et al. 2009). Here, however, it needs to be pointed out that analyses based on smaller entities (publications, institutions) produce considerably different results for the two databases than an analysis based on larger entities such as research domains and previously mentioned countries (Meho and Sugimoto 2009). Researchers frequently report possible errors and limitations involving analyses that derive data directly from databases. Vieira and Gomes (2009) used such data in an assessment of two universities and reported possible inaccuracies in the retrieval of bibliographic units due to input errors, for example, the country in the authors’ addresses. In a study covering WOS and several other databases, it was observed that sometimes a city may be present in the affiliation field while the country is missing (Bartol and Hocevar 2005). Even though the identification of an author may be good, the missing data can also be attributed to journal publishers who have not always had consistent policies for including the country affiliation of the authors (Jacso 2009). Many Russian-language references are frequently not taken into account in “citation reports” because of problems involving transliteration (Zibareva and Soloshenko 2011). Furthermore, the policy of assigning particular document types may change over time, which creates unclear differentiating criteria among different subject disciplines (Harzing 2013). Subject categories or subject areas are database-specific, so differences exist in the representation of research fields in each database. Important differences in categorization exist even among the top journals within a particular category (Abrizah et al. 2013). Such journal classification systems may hold at the macro-level but will be less applicable at more specialized levels (Tijssen 2010). Some “marginal” fields may be covered by several different subject categories (Chirici 2012). Researchers therefore sometimes employ a harmonization based on a common classification. The European classification of the EFS (European Foundation for Science) was used for the field of social sciences and humanities by Pumain et al. (2010) in order to offset some differences between WOS and SCOPUS in subject classification. Archambault et al. (2009) used the categories of the United States’ National Science Foundation (NSF) to delineate the fields of natural sciences and engineering. Bornmann and Leydesdorff (2013) used Frascati (OECD) subject categories to compare the outputs of six highly productive countries. However, in spite of the common guidelines in the Frascati manual, differences will remain in the way R&D statistics are compiled nationally (Wendt et al. 2012). Finally, it needs to be underlined that the coverage, scope, and search functionalities in both databases are constantly evolving; therefore, the information in the literature can only refer to the coverage that exists at the time of the analysis (Lopez-Illescas et al. 2009). The constant expansion of coverage towards more local or national relevance or additional Scientometrics (2014) Volume 98, Issue 2: 1491–1504 publication types thus represents a serious challenge for citation based methods and indicators (Glanzel and Moed 2012). We may at this point add that Slovenia’s current research information system, SICRIS , which was used as the main tool for our research, mitigates many of the above challenges and detects changes in both databases very promptly through constant updating, thus providing a very good tool for a thorough scientometric assessment of national production in relation to both SCOPUS and WOS . Its functionalities are presented more systematically in the next section.

Materials and Methods

The functionalities of both citation databases have been presented on many occasions so in this paper we will not again address their major features but will place more emphasis on the evaluation of data as derived from the authority-controlled SICRIS and COBISS systems and subsequent links to both SCOPUS and WOS . Funded by the Slovenian Research Agency ( SRA ), the Slovenian current research information system (SICRIS ) contains data on research organizations and groups, research projects and programs, and individual researchers. Bibliographic evaluation of researchers' publishing activity is provided through the COBISS system and its union bibliographic database. Software development is managed by the Institute of Information Science ( IZUM ). Researcher bibliographies became obligatory in 1997, involving inputs by professional librarians (Demsar and Juznic 2013). One of the more important features of the system is the functionality of a scientometric assessment of researchers’ bibliographies, whereby all bibliographic units that are covered by WOS are provided with links to the corresponding WOS record (Curk et al. 2006). The system has facilitated several analyses of scientific output, for example, of collaboration among scientists and co-authorship patterns (Perc 2010; Peclin et al. 2012) or collaboration within selected research disciplines such as biotechnology, mathematics, physics, and sociology (Kronegger et al. 2011). In order to complement the WOS -related data in COBISS , in 2012 the SRA decided to provide similar links to the bibliographic output in SCOPUS . The feature now permits tracing published documents as well as counting citations to documents according to both WOS - and SCOPUS -derived indicators. COBISS links bibliographic records to the general SCOPUS database (SciVerse/Elsevier) and WOS Citation Databases (Science Citation Index Expanded, Social Sciences Citation Index, Arts & Humanities Citation Index; Thomson Reuters) for all bibliographic units (co-)authored by scientists who are subject to the systematic evaluation of the published research in SICRIS . The input data is checked and verified both by the authors and the respective bibliographic services, usually at an author's department responsible for the accuracy of data. During this process, complex record matching algorithms are used to link the corresponding records in COBISS , SCOPUS , and WOS . The links for records that remain unlinked during the automated process are established manually. When the links are identified and established, they are saved in a special database so it is no longer necessary each time to determine which bibliographic units in WOS and SCOPUS belong to a researcher. In order to enable this analysis, all relevant records were arranged in an experimental archive to facilitate a more thorough exploration. The citations were counted on the principle of all citations. Some limitations existed due to different ways of recording authors' names in COBISS (full names) and in WOS or SCOPUS (surnames and initials). Some differences were also found in the bibliographic data. Such differences represent a challenge for the automatic identification and coupling of corresponding records. The largest variances between the two Scientometrics (2014) Volume 98, Issue 2: 1491–1504 databases occurred in the titles containing chemical formulae or other symbols and also, to a lesser extent, in the years of publication, ISSN numbers, DOI identifiers, volumes, issues, and number of pages. The authority control in COBISS compensates for a significant portion of such differences. Our analysis involved bibliographic units (co-)authored by 9,828 registered researchers who were active between 1996 and 2011. Researchers are closely monitored on the basis of a unique author identifier, i.e. the SRA code (Slovenian Research Agency code). The analysis covered all researchers who are registered in the SICRIS system and evaluated for the purposes of the SRA . Besides all the Slovenian authors who are active in public research organizations, the system includes also Slovenian researchers working abroad and some foreign researchers who are active in Slovenia and need to be evaluated for the purposes of the SRA . We therefore included all documents that were co-authored by at least one such registered researcher. According to the principles of research policy in Slovenia, in order to better track some possible differences among different research fields, all researchers also need to associate their particular research area with one of the six major research fields (Field of Research Classification) and their respective narrower fields, for example, biology, mathematics, or physics in the natural sciences or civil engineering, computer science, or geodesy in the engineering sciences and technologies. This classification, which is used by the SRA , has been roughly harmonized with the Field of Science and Technology Classification in the Frascati Manual (OECD), with some modifications related to the specifics of research groups in Slovenia. Table 1 presents fields of research in Slovenia matched with the major fields of science and technology according to Frascati and the number of different researchers classified in the respective fields in the 1996–2011 period.

Table 1 Fields of Science and Technology as used for research classification by the SRA (Slovenian Research Agency) and Frascati/ OECD ; researchers active in the field between 1996 and 2011.

Field of Research Classification Major Field of Science Research (SRA ) and ers Technology (Frascati/ OECD ) Biotechnical Sciences Agricultural Sciences 906 (agriculture) Engineering Sciences and Engineering and 3,000 Technologies Technology Humanities Humanities 718 Medical Sciences Social and Health 1,838 Sciences Natural Sciences and Natural Sciences 2,281 Mathematics Social Sciences Social Sciences 1,085 Total 9,828

If a document was co-authored by several authors all pertaining to the field of natural sciences, it was counted only once in this field. However, we also needed to take into account Scientometrics (2014) Volume 98, Issue 2: 1491–1504 those selected “multidisciplinary” documents that had been co-authored by authors from different fields. For example, if a document was co-authored by authors pertaining to the natural sciences and authors from the medical sciences, it was counted once in each field. Our method involves the principles of whole counting, which is applied in most bibliometric analyses. Even though some authors make a case for fractionalized counting (Aksnes et al. 2012), this was not practical for our analysis since the whole counting method is also employed by the source data in SICRIS . The same method was used on both databases on the same controlled source of data, so both databases were compared on the same principles for all fields of science.

Results

Our more detailed analysis was conducted on 9,828 researchers who classified their research activity into one of the six major fields of science and technology presented in Table 1. Unclassified researchers were not assessed. The classified researchers (co-)authored 44,226 documents in SCOPUS and 38,339 documents in WOS , according to the links in COBISS . These documents were cited 500,094 and 432,311 times, respectively (Fig. 1). The analysis covered those more “authoritative” document types that are used for assessment of research activity in SICRIS such as articles, proceedings papers, and chapters. Some miscellaneous document types such as “abstracts-only,” corrections, and letters were not included.

Documents Citations 50000 500000

45000 450000

40000 400000

35000 350000

30000 300000

25000 250000

20000 200000

15000 150000

10000 100000

5000 50000

0 0 Scopus WOS (a) Scopus WOS (b)

Fig. 1 The total number of documents (a) and citations (b) in the 1996–2011 period in SCOPUS and WOS databases

Scientometrics (2014) Volume 98, Issue 2: 1491–1504

Figure 1 serves as a general reference for more detailed analyses in the continuation of the paper. At this point it needs to be reiterated that SCOPUS was introduced only recently so it is possible that some bibliographic records in COBISS may not have been linked to SCOPUS as yet. Figures 2 and 3 present the yearly growth of published documents in the observed period by major fields of science according to the subject matter of an author's research activities. The four associated life and applied sciences (agricultural sciences, medical (and health) sciences, natural sciences, and engineering & technology) are presented together (Fig. 2), and separately from the social sciences and humanities (Fig. 3). Namely, the total numbers of documents for the social sciences and humanities are much lower on account of smaller research groups and different publishing and citation patterns in these two fields. Figure 2 reveals a continual growth in all four major life and applied sciences fields in both databases. These numbers must be perceived in relation to the registered researchers in their respective fields as shown in Table 1. The growth is fairly steady although in some isolated phases there is some irregularity in natural sciences and engineering & technology, which can be inferred from the 2004 and 2006 data. The total numbers are the highest in the natural sciences, indicating specific publishing patterns in this scientific field. Agricultural sciences return the lowest numbers, with only 906 active researchers in this field (Table 1).

2000

1800

Agric. sci. Scopus 1600 Agric. sci. WOS 1400 Medic. sci. Scopus

1200 Medic. sci. WOS

Natural sci. Scopus 1000 Natural sci. WOS

800 Engineer./Tech. Scopus

600 Engineer./Tech. WOS

400

200

0 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011

Fig. 2 Growth of documents in agricultural sciences, medical sciences, natural sciences, and engineering & technology in the 1996–2011 period in SCOPUS and WOS databases

Scientometrics (2014) Volume 98, Issue 2: 1491–1504

500

450

400

350 Social sci. Scopus

300 Social sci. WOS

250 Humanities Scopus 200 Humanities WOS 150

100

50

0 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011

Fig. 3 Growth of documents in social sciences and the humanities in the 1996–2011 period in SCOPUS and WOS databases

Some other outcomes are more interesting. Even though SCOPUS reveals higher numbers than WOS in all groups, we can observe some differences in database-specific results between the natural sciences on the one hand and engineering & technology on the other. Although SCOPUS shows some lead over WOS in the natural sciences, this lead is much more substantial in engineering & technology. The differences are not strongly influenced by the better coverage of Slovenian national journals in SCOPUS . While 34 Slovenian journals are indexed by SCOPUS compared to 26 by WOS , the research evaluation policy in Slovenia encourages presentation of research in the international context, thus giving less incentive to national publishing with fewer articles appearing in Slovenian publications. Figure 3 illustrates the trends in social sciences and the humanities. Social sciences display an accelerated growth after 2006 in both databases. Some stagnation can be observed in the humanities in 2006 (Fig. 3). This development, however, is more than effectively offset in the next year. While there is a brief converging between SCOPUS and WOS in the humanities in 2009, publications in WOS then seem to “lose some steam.” On average, however, SCOPUS exhibits better inclusion of documents in social sciences and humanities in comparison to the fields in life and applied sciences.

Scientometrics (2014) Volume 98, Issue 2: 1491–1504

20000

18000 250000

16000

14000 200000

12000 150000 10000 Citations Documents 8000 100000 6000 Scopus Documents WOS Documents 4000 Scopus Citations 50000 WOS Citations 2000

0 0 Agric. sci. Medic. sci. Natural sci. Engineer./Tech.

Fig. 4 Documents (left y-axis scale) and citations (right y-axis scale) in agricultural sciences, medical sciences, natural sciences, and engineering & technology in the 1996–2011 period in SCOPUS and WOS databases

The final analysis brings the assessment of published documents in relation to citations received in all respective major fields in both databases (Figures 4 and 5). The data are presented in two-axis charts to better illustrate the differences among the fields. The left-hand axes show the number of total documents for each respective field; the right-hand axes present citations. SCOPUS yields more results in all disciplines but the figures vary substantially among the disciplines. The difference in database coverage in the life and applied sciences group (Figure 4) is most noticeable in engineering & technology. Some 2,000 more documents can be found in SCOPUS . The citations (right-hand axis) reveal a similar pattern. The difference in percentages between the databases is higher in the social sciences and humanities (Figure 5). Both fields display more documents and citations in SCOPUS than in WOS . To offer some additional perspective, we also provide information on the Slovenian journals included in SCOPUS and not WOS , and vice versa, by database-specific general classifications. SCOPUS includes 15 Slovenian journals not indexed by WOS , and WOS seven journals not indexed by SCOPUS (Table 2). One of the SCOPUS journals is listed under both Agricultural and Biological Sciences and Environmental Science. Altogether, 34 Slovenian journals are indexed by SCOPUS and 26 by WOS . Some of these journals, however, though recorded on indexed-journals lists in databases, do not return any results (or just a few dozen). Also, many journals are very rarely or never cited, not even as self-citations.

Scientometrics (2014) Volume 98, Issue 2: 1491–1504

3000 16000

14000 2500

12000

2000 10000

Scopus Documents 1500 8000 WOS Documents

Citations Scopus Citations Documents WOS Citations 6000 1000

4000

500 2000

0 0 Social sci. Humanities

Fig. 5 Documents (left y-axis scale) and citations (right y-axis scale) in social sciences and humanities in the 1996–2011 period in SCOPUS and WOS databases

Table 2 Slovenian journals listed only in SCOPUS or only in WOS

SCOPUS Subject Area Journals Agricultural and Biological Sciences 2 Arts and Humanities 1 Computer Science 2 Environmental Science 1 Medicine 1 Pharmacology, Toxicology and 1 Pharmaceutics Physics and Astronomy 1 Social Sciences 7 WOS (general groups) Journals Arts & Humanities (AHCI) 2 Sciences (SCIE) 3 Social Sciences (SSCI) 2

With regard to citations, further differences among the disciplines are evident in both databases. The average number of citations per published document is much lower in engineering & technology (7.4 cit/doc – WOS ) than in the natural sciences (14.4 cit/doc – WOS ) or medical sciences (13.6 cit/doc – WOS ). In WOS , 17,678 and 11,134 respective documents have been attributed to natural sciences and engineering & technology. However, natural sciences received 253,752 citations in this database, and engineering & technology only 82,590 Scientometrics (2014) Volume 98, Issue 2: 1491–1504

citations. SCOPUS shows similar patterns but returns more documents and citations. In the agricultural sciences, SCOPUS returns some 10% more documents and citations, with citations per document producing similar results: 10.8 cit/doc in SCOPUS and 10.6 cit/doc in WOS . However, the medical sciences show 13.6 cit/doc in WOS and 15.0 cit/doc in SCOPUS . The difference between the two databases is more evident in the social sciences and humanities (Figure 5). In SCOPUS , social sciences display 2,884 documents with 13,423 citations (4.7 cit/doc), and humanities 1,877 documents with 2,505 citations (1.3 cit/doc). Citations per document in WOS are similar with 4.2 and 1.2 cit/doc, respectively. In WOS , however, the total citation count for humanities is only 1,656, lower by more than a third compared to SCOPUS (2,505 citations). In general, the humanities yield the lowest count of citations per document in both databases.

Discussion

Our study employed the two global citation databases SCOPUS and WOS to assess certain differences between the databases and to ascertain how such differences are reflected in the major research fields and the possible applicability of such information in the evaluation of published research in Slovenia. We employed the Slovenian national system for the evaluation of research ( SICRIS ), which is linked to the unique bibliographic functionality of the COBISS system, wherein all registered researchers are systematically assessed for inclusion of their documents in the two databases along with pertaining citation counts. Most similar analyses have focused on a limited set of data and experimental materials are almost invariably derived directly from the databases. Our analysis, however, systematically reviewed all scientific fields based on a harmonized classification of fields and employed data of a vast majority of researchers in Slovenia. This analysis thus involved almost the entire national production over an extended period of time (1996–2011). To avoid many possible errors in the identification of authors, incorrect country, missing author affiliation or changing patterns in the classification of publication types in some disciplines (Vieira and Gomes 2009; Jacso 2009; Harzing 2013), our data were subjected to a rigorous authority control. Our assessment of differences in publishing and citing patterns among different major fields of science was based on an international classification, thus offsetting some major problems with subject schemes based on journals (Chirici 2012). It is also known that database differences exist in the coverage of certain document types which may even be assigned incorrectly (Bar-Ilan 2008; Meho and Sugimoto 2009, Harzing 2013). However, since the records in COBISS are supplied with permanent document-type indicators, our harmonized input data represented an important factor for the assessment of databases on equivalent basis. The results of our study show that SCOPUS , generally speaking, provides more results, both in the number of records as well as in the number of related citations. This is in agreement with the findings by, for example, Torres-Salinas et al. (2009) and Haddow and Genoni (2009). There is no substantial difference between the two databases in the coverage of Slovenian national journals as opposed to the considerable advantage of SCOPUS over WOS in the coverage of the national journals of some other countries (Torres-Salinas et al. 2009; Zibareva and Soloshenko 2011; Leydesdorff et al. 2010). Also, many Slovenian journals generate very low citation counts in both databases. On the other hand, some more established Slovenian journals publish works by non-Slovenian authors. A “national bias” thus plays a relatively minor role in our study. Scientometrics (2014) Volume 98, Issue 2: 1491–1504

The analysis, more specifically, addressed differences between databases on the basis of detecting some possible special characteristics in different research fields. Some other authors have also tackled some specific differences, for example, among specific subject areas particularly in a more national-language oriented publishing landscape (Lopez-Illescas et al. 2009) or in the coverage of specific publications (Bar-Ilan 2008). Our analysis, however, systematically employed a harmonized subject classification of research fields and involved all scientific fields. The model (case) is thus very comprehensive. We found some more noticeable differences between the databases in the fields of social sciences and humanities, with SCOPUS exhibiting noticeably higher counts than WOS . Perhaps more interesting are the figures in the so-called life-and-applied-sciences group comprising agriculture, engineering & technology, medical, and natural sciences. While SCOPUS shows better coverage in all four major fields, this difference is much more evident in engineering & technology with SCOPUS covering substantially more documents than WOS . The differences in agriculture, medical, and natural sciences are much smaller. The final part of the analysis tackled the possible differences in citations received by Slovenian authors in relation to published documents, again for both databases. SCOPUS shows more citations per document than WOS in all fields. We detected the highest citations-per- document counts in the natural and medical sciences. Among all the fields, citations per document are the highest in medical sciences in SCOPUS . In the life-and-applied-sciences group, the citation counts are the lowest in engineering & technology. Agriculture is placed approximately in the middle. Citation counts are much lower in the social sciences, and the lowest in humanities, clearly indicating more particular publishing and citation patterns in these two fields. Some other document types such as books may yield hundreds of additional citations in other bibliographic services; however, such citations are not monitored on the same principles by the two databases. Consequently, this study in no way sought to judge the overall quality of coverage of publications by Slovenian authors in the respective fields of science. More complex assessment of such patterns is beyond the scope of the present analysis but may be carried out in our future research on a smaller sample. Our results nevertheless clearly indicate the existence of substantial differences in the “citability” of research fields established on a very comprehensive national collection of data over a longer period of time. The identification of these differences is important because research grants need to be approved and funds allocated through a fair evaluation system. For example, SCOPUS seems to provide more extensive coverage and more citations in social sciences and humanities so it could be preferred for a more balanced evaluation of these fields. Also, since engineering & technology scientists receive “weaker” citation counts on average than natural scientists, the former can compete fairly for funds only within their particular peer group. This study thus provides some important information for policy makers who can use the findings for the purposes of a more impartial appraisal of research outcomes.

Conclusions

Our study represents a systematic attempt to evaluate the differences in published documents and citations across all major research fields in the SCOPUS and Web of Science databases based on a standardized classification of fields. These differences were assessed for the case of an entire country (Slovenia). This was possible by employing an in-depth analysis of SCOPUS - and WOS -linked data in the COBISS system. This national information system systematically tracks such data on the principles of rigorous authority-control and the harmonized international Scientometrics (2014) Volume 98, Issue 2: 1491–1504 subject classification of all documents, thus offsetting many well-known problems of deriving data directly from these databases. The results corroborate some previous studies that found SCOPUS to return more results than WOS . However, our study also brings a detailed analysis of more particular differences among different scientific fields. The differences are most obvious in the fields of social sciences, humanities, and engineering & technology, with SCOPUS returning noticeably more results than WOS in the number of total documents and citations. Some further characteristics can also be observed within different scientific fields. The least citations per document are received in the humanities. Social sciences perform better. However, more interesting figures emerge in life and applied sciences: the field of engineering & technology exhibits much lower citation counts per document than comparable agricultural and especially medical and natural sciences in both databases. The detection and systematic assessment of these differences provide Slovenia’s funding agency with additional criteria for a more transparent and balanced evaluation of research. The results are based on comprehensive data, established on a very large and inclusive set of documents and citations over a longer period of time, which also offers some more widely applicable information on both databases.

References

Abrizah, A., Zainab, A. N., Kiran, K., & Raj, R. G. (2013). LIS journals scientific impact and subject categorization: a comparison between Web of Science and Scopus. Scientometrics , 94 (2), 721–740. Aksnes, D. W., Schneider, J. W., & Gunnarsson, M. (2012). Ranking national research systems by citation indicators. A comparative analysis using whole and fractionalised counting methods. Journal of Informetrics , 6(1), 36–43. Archambault, E., Campbell, D., Gingras, Y., & Lariviere, V. (2009). Comparing Bibliometric Statistics Obtained from the Web of Science and Scopus. arXiv:0903.5254 , doi:10.1002/asi.21062. Bar-Ilan, J. (2008). Which h-index? A comparison of WoS, Scopus and Google Scholar. Scientometrics , 74(2), 257–271. Bar-Ilan, J., Levene, M., & Lin, A. (2007). Some measures for comparing citation databases. Journal of Informetrics , 1(1), 26–34. Bartol, T., & Hocevar, M. (2005). The capital cities of the ten new European Union countries in selected bibliographic databases. Scientometrics, 65 (2), 173–187. Benoit, K., & Marsh, M. (2009). A Relative Impact Ranking of Political Studies in Ireland. The Economic and Social Review , 40 (3), 269–298. Bornmann, L., & Leydesdorff, L. (2013). Macro-Indicators of Citation Impacts of Six Prolific Countries: InCites Data and the Statistical Significance of Trends. PLoS ONE , 8(2), e56768. Chirici, G. (2012). Assessing the scientific productivity of Italian forest researchers using the Web of Science, SCOPUS and SCIMAGO databases. iForest - Biogeosciences and Forestry , 5(3), 101–107. Curk, L., Budimir, G., Seljak, T., & Gerkes, M. (2006). [Linking the SICRIS – COBISS.SI – Web of Science systems]. Organizacija znanja , 11 (4), 230–235. Demsar, F., & Juznic, P. (2013). Transparency of research policy and the role of librarian. Journal of Librarianship and Information Science . doi:10.1177/0961000613503002. Glanzel, W., & Moed, H. F. (2012). Opinion paper: Thoughts and facts on bibliometric indicators. Scientometrics . doi:10.1007/s11192-012-0898-z. Haddow, G., & Genoni, P. (2009). Australian Education Journals: Quantitative and Qualitative Indicators. Australian Academic & Research Libraries , 40 (2), 88–104. Harzing, A.-W. (2013). Document categories in the ISI web of knowledge: Misunderstanding the social sciences? Scientometrics , 94(1), 23–34. Scientometrics (2014) Volume 98, Issue 2: 1491–1504

Jacsó, P. (2009). Errors of omission and their implications for computing scientometric measures in evaluating the publishing productivity and impact of countries. Online Information Review , 33 (2), 376–385. Kronegger, L., Ferligoj, A., & Doreian, P. (2011). On the dynamics of national scientific systems. Quality & Quantity , 45 (5), 989–1015. Lasda Bergman, E. M. (2012). Finding Citations to Social Work Literature: The Relative Benefits of Using Web of Science, Scopus, or Google Scholar. The Journal of Academic Librarianship , 38 (6), 370–379. Leydesdorff, L., Moya-Anegón (de ), F., & Guerrero-Bote, V. P. (2010). Journal maps on the basis of Scopus data: A comparison with the of the ISI. Journal of the American Society for Information Science and Technology , 61 (2), 352–369. Lopez-Illescas, C., Moya Anegon (de), F., & Moed, H. F. (2009). Comparing Bibliometric Country-by-Country Rankings Derived from the Web of Science and Scopus: The Effect of Poorly Cited Journals in Oncology. Journal of Information Science , 35 (2), 244–256. Meho, L. I., & Sugimoto, C. R. (2009). Assessing the scholarly impact of information studies: A tale of two citation databases—Scopus and Web of Science. Journal of the American Society for Information Science and Technology , 60 (12), 2499–2508. Peclin, S., Juznic, P., Blagus, R., Sajko Cizek, M., & Stare, J. (2012). Effects of international collaboration and status of journal on impact of papers. Scientometrics , 93 (3), 937–948. Perc, M. (2010). Growth and structure of Slovenia’s scientific collaboration network. Journal of Informetrics , 4(4), 475–482. Pumain, D., Kosmopoulos, C., & Dassa, M. (2010). JournalBase - A Comparative International Study of Databases in the Social Sciences and the Humanities (SSH). Cybergeo: European Journal of Geography , (article 484), doi:10.4000/cybergeo.22862 Tijssen, R. J. W. (2010). Discarding the “basic science/applied science” dichotomy: A knowledge utilization triangle classification system of research journals. Journal of the American Society for Information Science and Technology , 61 (9), 1842–1852. Torres-Salinas, D., Lopez-Cózar, E., & Jiménez-Contreras, E. (2009). Ranking of departments and researchers within a university using two different databases: Web of Science versus Scopus. Scientometrics , 80 (3), 761– 774. Vieira, E. S., & Gomes, J. A. N. F. (2009). A comparison of Scopus and Web of Science for a typical university. Scientometrics , 81 (2), 587–600. Wendt, K., Aksnes, D. W., Sivertsen, G., & Karlsson, S. (2012). Challenges in Cross-National Comparisons of R&D Expenditure and Publication Output. In Proceedings of 17th International Conference on Science and Technology Indicators (Vol. 2, pp. 826–834). Presented at the STI 2012, Montreal, Canada, September 5-8. Zibareva, I., & Soloshenko, N. (2011). Russian scientific publications 2005–2009 in the science citation index, scopus, and chemical abstracts databases. Scientific and Technical Information Processing , 38 (3), 212–223.