View metadata, citation and similar papers at core.ac.uk brought to you by CORE InfoTrend 64 (2009)4 provided by Göteborgs universitets publikationer - e-publicering och e-arkiv

Håkan Carlsson has been Allocation of working as a bibliometrici- an at Gothenburg Univer- sity Library since 2008. Research Funds He is the chairman of the SUHF working group Using Bibliometric for bibliometrics and has held a number of positions in national committees. Indicators – Asset Before coming to Goth- enburg he was Director and Challenge to of Scientific Communi- cation at Lund University Libraries. Håkan holds a doctorate degree in chemistry and has also worked for Swedish Higher a number of years in the USA as a chemistry professor. Education Sector (Photo: Johan Wingborg)

Håkan Carlsson

factor in this development is the recent intro- Since 2009, the Swedish government has allo- duction of a bibliometrics-based national al- cated a part of its direct funding to the Swed- location system for the direct research funding ish universities based on a bibliometric indica- of the Swedish universities and university col- tor, which stems from the number of publica- leges. As of the 2009 financial year, a part of tions and citations found in the Web of Science the funding from the national government to bibliographic database. This paper discusses the universities depends on prior performance the creation and detailed structure of the new of the university researchers in terms of publi- indicator, which has met criticism from both cations, citations and external funding. The researchers and the Swedish Research Council, structure, origin and consequences of this new set to maintain the new system. The current system are investigated here. focus on evaluation and quality assurance in the area of higher education, together with the Bibliometrics for the Allocation introduction of the new indicator, has created of National Research Funding an increased demand on the university admin- The use of incentive structures has become istration and libraries for publication databases more and more important in the desire to and bibliometric services form a better and more productive society. This has also been true for researchers apply- ing for research funding, which has been scru- The area of bibliometrics in Sweden has expe- tinized by peers for decades. An area where rienced strong growth for a number of years. this has not been implemented until recently The interest in assessments and quality revi- is the direct financing of basic university re- sion has reached the higher education sector search services, which have been funded by and publication analysis has been found to be the public largely based on a percent increase specifically valuable in assessing general re- from the previous year’s levels. This has creat- search performance. A new and contributing ed uncertainty as to the validity of the system

82 InfoTrend 64 (2009)4

and slowly eroded it in favor of the peer-re- built on a number of indicators, which had view-based research councils, which have been found to have a high acceptance in the grown in size. research community. The major part of the Several countries have made attempts to system was to be based on panel evaluation, find solutions to this challenge. The United much like the RAE system. In addition nor- Kingdom unveiled its Research Assessment malized citations in the international citation Exercise1 (RAE) as far back as 1986 and set database Web of Science, external funding, out to assess all institutions using peer-review number of teaching staff with a Ph.D. and panels. This cumbersome exercise has been re- number of female professors would make up peated roughly every five years to give input to the indicator. The choice of citation-based the British research councils in their work to metrics was to some extent made because of fund research. Norway, on the contrary, chose its relative ease of startup as the system was to a bibliometric system and allocates a part of be functioning already in 2010 as stipulated its direct funding on a system built on publi- by the Ministry. A system based on self-re- cation counts and publication channel quality ported data, like the one in Norway, would for publications reported by the Norwegian require several years of preparations. researchers.2 In the consultation following the inquiry, Sweden has been complacent to these de- the universities and other stake holders ex- velopments for a long time. With a histori- pressed their opinions on the proposed sys- cally strong R&D sector, the political initia- tems. Many issues were raised, but a compara- tive was lacking. In 2004, the EU took a clear tively large number of respondents were rela- stance through its Lisbon Strategy to focus on tively positive to the idea of citation reassuring the infrastructure of the knowl- bibliometrics as a part of a new allocation sys- edge-based community.3 This, in part, was the tem. This was also the case for the external reason that the Swedish Government in 2006 funding indicator. During the same period, decided to widen an inquiry into improving the consultants responsible for the idea of ci- the education funding model to also include tation analysis, headed by Dr. Ulf Sandström, the construction of a model for the research in cooperation with the Swedish National area.4 Agency for Higher Education (HSV), pub- In November 2007, the inquiry (Resursu- lished a more in-depth study of some con- tredningen) published its findings.5 It suggest- cerns aired in the consultation regarding the ed an extensive system for evaluation-based bibliometric indicator.6 funding for both education and research. The In October 2008, the final system was re- new system would be governed by an interme- vealed through the government bill “Ett lyft diary agency, which would separate the uni- för forskning och innovation”.7 Only the cita- versities from the funder, and determine allo- tion metrics and the external funding indica- cation keys each year. The research system was tors remained. The idea of an intermediary

1 http://www.rae.ac.uk 2 http://dbh.nsd.uib.no/pub/hjelp.jsp?rapport=antall& 3 http://ec.europa.eu/growthandjobs/pdf/kok_report_en.pdf 4 http://www.sou.gov.se/kommittedirektiv/2006/Dir2006_29.pdf 5 http://www.regeringen.se/sb/d/8439/a/91339 6 http://www.hsv.se/download/18.8f0e4c9119e2b4a60c80006308/0818R.pdf 7 http://www.regeringen.se/sb/d/10003/a/113957

83 InfoTrend 64 (2009)4

had been removed and the Swedish Research university hospitals must include the corre- Council (VR) had been put in charge of main- sponding university name to be counted. On- taining and calculating the indicator. The bill ly first authors and corresponding authors are also introduced extensive funding to specific considered and the publications are split if named strategic research areas, in which recip- these researchers come from different institu- ients were to be selected by the Ministry of Ed- tions. It should be noted that address infor- ucation. The suggested bill was signed into law mation is not always easy to interpret and a by a decision of the Riksdag in January 2009. considerable effort is therefore put into this work. The number of publications, and cita- The Nuts and Bolts tions to these publications, are counted. As Because of its relatively quick processing, the the base for the 2010 allocation, data from new system was taken into production already 2005-2008 is used. for the 2009 financial year. As a part of the in- Methodologically, the raw number of creased focus on research, and because of the publications and citations are somewhat mis- new added reliability in the funding model, leading, mainly due to differences in publica- new fresh funding was made available. This tion rate, citation rate and database coverage new funding was allocated in its entirety with for different areas of research. This leads up to the new system. From 2010 a part of the old a need for normalization. Several years back, a funding will also be reallocated. This is done method for normalizing citations, creating by the removal of 10 percentage of the fund- what is called field-normalized citation ing of each university and redistribution is counts, was proposed and has won some ac- based on the new system of the withheld ceptance.8 The system for normalizing publi- amount. This mode of action will ensure a cations is new with this proposal. gentle transition to a more and more indica- Citations are normalized based on three tor-based allocation at the same time as the different conditions: the field of research, the new funding limits the chance of anyone los- document type (article, review or letter) and ing out in the system. A base funding of SEK publication year. It is done by collecting all pa- 8 000 per student also insured the level of pers worldwide in the WoS database that share funding for the small colleges. the same field, document type and publication The exact construction of the bibliometric year with the analyzed article. Field is deter- indicator is of interest for purely bibliometric mined by the journal and can be one of several reasons, but has also been discussed by re- of the 255 journal fields in WoS. From the ref- searchers trying to understand the system. erence body of papers, an average number of Therefore I will make an attempt to present it citations per publication can be calculated and in some detail. The data for the indicator is the field-normalized citation is constructed by collected from Web of Science (WoS), which dividing the number of citations for the ana- is a database indexing over 10 000 (recently 6 lyzed article by the average citation of the ref- 500) scientific journals in all areas of science. erence publications. The resulting indicator is Based on publication addresses, the individual practical in that it is 1 for publications with contributions are contributed to the different world-average citation numbers. A score of Swedish universities. Publications that are au- 1.22 would indicate that the article or a group thored by researchers with affiliation to the of articles are cited 22 percentage more than

8 Moed, H. F.; De Bruin, E.; Van Leeuwen, T. N. Scientometrics, 1995, 33, 381-422

84 InfoTrend 64 (2009)4

the world average. It should be mentioned that cation in the HS fields count nearly 15 times the citation distribution per publication is higher than a publication in a more proliferate skewed, which has led to a debate questioning WoS area such as chemistry. the value of the average as an indicator. The final step in the calculation of the bib- Publications were also in need of normali- liometric indicator is to determine the prod- zation. The main reason for this is the fact that uct of the field-normalized citation and the a large part of research is conducted in areas, number of average productive researchers. where the quality research is presented in non- This score is then used together with the indi- journal publication channels (books, anthol- cator for external funding. The system was ogy chapters or conferences). Since these quickly found to favor the specialized univer- forms of publication are outside the WoS cov- sities and especially the external funding gave erage, an apparent lower publication rate can the Humanities and Social Sciences a disad- be found in large parts of the humanities and vantage. To weaken this effect a factor was in- the social and applied sciences. In the new troduced where institutions with large Hu- bibliometric system, the publications are nor- manities and Social Sciences (HS) activities malized by comparing the publication rate of were given a higher indicator. Based on the other Nordic researchers in a number of rath- old funding scheme, institutions were given er broad fields of science.9The number of twice the points equivalent to the ratio of old fields was first 23 in the inquiry report, though funding for HS research, 1.5 times for the Sci- in the HSV report and the bill they had in- ence section and 1.0 for Medicine and Engi- creased to 34. To generate the normalization neering. The other areas were denoted 1.1. data, the average number of publications is The final collected indicator was made up of calculated. The database itself shows all publi- 50-50 of bibliometrics (publications and cita- cations with authors that have published at tions) and external funding multiplied by the least one publication. By the use of the War- correction factor. ing distribution, the number of authors which has not published anything can be approxi- Criticism and Fine-tuning mated and together the average production During the preparatory work of the new sys- per researcher can be determined.9 Using the tem and especially after its final form, criti- average production the publications of the an- cism has been heard from several directions. alyzed university can be converted to the Independent researchers have aired concerns number of average productive researchers, to whether research can be evaluated with bib- which is used as the normalized value for pub- liometric tools or whether it improves the lications in the evaluation. Because less than quality of research in Sweden.10 The limited 10 percentage of the publications from the coverage of the humanities and social sciences Humanities and Social Sciences are visible in has led to a debate whether these areas are WoS, an average researcher only publishes a overseen by the Ministry.11 small fraction of a WoS publication each four- The Swedish Research Council was in the year period. This leads to the fact that a publi- bill given the responsibility for maintenance

9 Sandström, U.; Sandström E. Res. Eval. 2009, 18, 243-250 10 http://www.fekis.se/debatt/2008/11/konstgjord-konkurrens-fungerar-inte-i-hogskolan-heller http://www.sulf.se/templates/CopyrightPage.aspx?id=9902 11 http://www.dn.se/kultur-noje/debatt-essa/sverker-lenas-utraknad-humaniora-1.888122

85 InfoTrend 64 (2009)4

and development of the bibliometric indi­- gathered a number of problems and tried to cator. They were also given the task to pro­- find solutions for these. The list was submitted duce the yearly statistical material for the for consultation to all Swedish universities and ­Ministry.12 To allow for improvements already colleges. These are summarized in Table 1. the first year, the Department of Research Pol- In May 2009, the Research Council pre- icy Analysis of the Research Council em- sented a first report in their new task to over- barked on an in-depth study of the new sys- see the new system.14 The suggestion included tem. The sector was invited to act as a refer- a suggestion for substantial revision of the sys- ence group for the process. The work came to tem. The Ministry was urged to suspend the focus on a number of obstacles. The Waring current system for a year to allow for the sys- method for publication normalization was tem to be completely revised. The HS section studied in depth and determined to be too was to be removed from the system perma- unstable. The coverage of the data for the Hu- nently until more reliable data was made manities and Social Sciences was also a major available. The Waring method was suggested point of weakness. to be replaced by a system based on incoming To aid the process, SUHF concurrently funding and the normalized productive re-

Table 1. Problems of the bibliometric indicator in the new Swedish allocation system for direct funding (freely after SUHF)13

Problem Possible solution

Humanities and Social Science researchers have Other sources than WoS, mainly self-reported data, limited possibility to affect the system. must be employed. The system is geared towards and encourages Other sources than WoS, mainly self-reported data, journal publishing. must be employed. The system is not transparent enough for the Data must be made available for the universities, researchers. so they can understand their outcome. External funding yields more publications, Only the part of publications that counts for the ratio so it is counted double. of direct funding should be used for the indicator. The applied correction factor is arbitrary Also the indicator for external funding should be normalized prior to use. The borderline to the university hospitals is unclear This needs to be considered in a further study. The technique of identifying all Nordic researchers Other sources than WoS, mainly self-reported data, to give data for the Waring process is too uncertain must be employed. The Waring system is too uncertain Other sources than WoS, mainly self-reported data, must be employed. The use of only first and corresponding author Also the other researchers should be allowed to is unfair. share the publication.

12 http://www.regeringen.se/sb/d/10692/a/120853 13 http://www.suhf.se/web/Problemstallningar_kring_den_nya_nationella_fordelningsindikatorn_for_direkta_ medel_till_forskning_-_skrivelse_till_VR_april_2009.aspx 14 http://www.vr.se/download/18.72e6b52e1211cd0bba8800010145/bibliometrisk_indikator.pdf

86 InfoTrend 64 (2009)4

searcher was to be replaced by a normalized tutions, where the university libraries often cost of each publication. This would also al- were the settings because of their knowledge low for better comparisons with institutions in related areas. with large propositions external funding. The The professional work was intensified as university hospitals were to be included in the the schools sensed an increased need for an data in their entirety. internal understanding of their activities and Upon submission of the report, it was as the Ministry’s reporting requirements had quickly clear that the Ministry had not ex- already increased prior to the new financing pected to be handed a suggestion with such system. The reporting requirements also put a longed-going revisions in a first report. It was focus on the need for an internal publication made public through a renewed task in the reporting system at the universities and publi- summer, when the Ministry requested VR to cation databases were implemented first at calculate the indicator for the 2010 financial Uppsala university and then at more and more year as it was constructed before VR’s report. universities. A study made a few years ago This task was later completed by the Research identified publication databases at 26 of 41 Council.15 Swedish universities and colleges; more or less all were maintained or coordinated by the Need for Institutional Support university libraries.17 When it comes to the consequences of the The surge in interest also yielded a need new system for Swedish university researchers, for further cooperation in Sweden. In 2007 the change has only just begun. Even if criti- SUHF created a working group for bibliomet- cism is still strong, several universities are rics under its subsidiary for library directors.18 looking for ways to encourage their faculty to The group’s major responsibilities have been publish in ways that would benefit their fund- in networking, staff development and knowl- ing. This has especially been clear at smaller edge exchanged. In 2007 a list of 10 persons institutions, where each publication has a with library connections were indentified as larger weight for the overall outcome for the bibliometric staff. This list has now grown to university. over 80. In 2008, the group commissioned an An area where the developments have al- inventory study of the area and identified 13 ready had a major impact is in the area of bib- universities with organized bibliometric activ- liometrics itself. Bibliometric research has ities.19 A year later this group has grown to 19 been present at the Swedish universities for a institutions and now covers all large universi- considerable time. It was pioneered on a larger ties and half of all Swedish institutions of scale by Olle Persson at Umeå University al- higher education.20 ready in the 1970’s.16 It was also earlier used as Most of the bibliometric activities are still a more professional tool at a number of insti- concentrated in the libraries. The presence of

15 http://www.vr.se/download/18.2dc104631220c7154e180002895/Missiv_Redovisning+Bibliometriuppdrag. pdf 16 http://www.issi-society.info/ollepersson60/ 17 http://tinyurl.com/epubse 18 http://www.suhf.se/web/Arbetsgrupper_inom_Forum_for_bibliotekschefer.aspx 19 http://hdl.handle.net/2077/18182 20 http://publications.lib.chalmers.se/cpl/record/index.xsql?pubid=101049

87 InfoTrend 64 (2009)4

high-quality data sources such as the publica- Databases for Self-reporting and tion databases paired with the bibliometric the Next Steps competences have together defined a new li- What can be said about the immediate future brary service. This is also apparent in the in- of this still quickly developing area? SUHF has crease of bibliometrics-related diploma theses recently published recommendations for qua- from the Swedish library schools. The biblio- lity improvements to the publication databa- metric staff today aids in the identification of ses. The lacking or uneven quality of these da- researchers for upcoming research funding tabases forced the Ministry in 2007 to base the opportunities, finding suitable internal and national system on the Web of Science. The external collaborators and later in enriching quality has since then become better and with the contents of the application itself. the development this fall of a national aggrega- Another common task for the local biblio- ting service, Swepub, accessibility will also in- metric staff is the production of indicators for crease. This may make it possible to shift data the local allocation of funds for research. The sources to the local publication databases. university and faculty administrations have The bill and the resent annual budget have discovered the use of bibliometric indicators indicated a new government inquiry related to from the national system and would like to the area. The prediction is that this inquiry will develop similar systems locally. Unfortunately, be given the task of looking into the area of this is a complicated undertaking. It first in- panels as an evaluative means for financial al- cludes identifying the local organizational locations, but it is also possible that the task of structure in the data. This can often be done incorporating the current system with panel by coupling external data to the publication data will still need to be addressed. This may database data. Next field-normalized data shed some new light on the problems of the must be acquired, if that type of data is de- current system addressed in the VR report. sired in the indicator. This data is not readily available since it requires reference data from Conclusions the whole world and can not easily be calcu- The introduction of a performance-based eva- lated from the normal library resouces. A final luation and allocation systems is not problem obstacle is to engineer the indicator. The na- free in any organization and especially not so tional indicators are very broad in their scope in the academic realm. The process takes ma- and the statistical downside of bibliometrics ny years and affects the academic sector in the becomes clear as the aggregations of publica- process. It also leads to a healthy discussion. tions get smaller and the statistical uncertain- The new opportunities for university libraries ty increases. These and other issues often give are also inspiring and utilize classical library rise to heated discussions at the biannual met- knowledge in a new way. I personally look for- rics seminars for bibliometric professionals or- ward to the direct funding to education as the ganized by the SUHF working group. next area of reform and to a whole new set of inspiring controversies.

21 http://www.suhf.se/web/REK2009-3.aspx 22 http://www.swepub.se

88